In the nineties I had my first Project Management related training, where I was introduced to the Standish Group‘s Chaos Study. This study collected, through question forms, information on IT related projects and their success. The study asked for projects to be classified as “Failed”, “Challenged”, or “Successful”, and asked respondents what factors influenced this outcome. The score for successful projects in 2004 was abysmal: only 29%, with 53% “Challenged”. (Over budget, late, or incomplete) For the training, this study was used to emphasize (among other things) the need for good project management, by referring to the list of factors influencing the result.

At that time I had already seen several project become challenged due to bad (“less then good”?) project management. However, if you read recent articles referring to the Chaos study, you’ll find that many are questioning the project success classification. They challenge the study’s findings by asking if a project’s success was measured correctly. As an all-round software guy, I know that project success is hard to grasp, but I still thought the study’s results very useful due to that list of factors. In my eyes, that was the most significant finding, not the bare success scores. Then again, I accepted the premise that a lot of projects don’t qualify as the success they’re presented as. Some people just seem to prefer denial.

So what is a Success anyway?

I guess this is the basic question: how do we define a success? As an IT pro, you tend to overemphasize the quality of the delivered product as a success factor, but getting good value for your money is not to be underestimated either. Both depend on the project actually delivering something to the customer, which may sound obvious, but I have actually worked on projects that did not deliver, and still didn’t affect our standing with the customer. Something else happened during the project that kept the customer happy: we showed we knew our stuff and did it well, but along the way we came to the conclusion that the project should be scrapped.

Let’s take a look at some of the things I ran into over the years, starting with the engineering view: it’s a success if, as the software engineer, you can look at the software and feel good about it.

Welcome to the the Real World™!

My first project after  university life concerned a system for the job development center of a large IT company. The project was billed as revolutionary, because it used Object Oriented database called Matisse. (it is mentioned by name only in Wikipedia) The idea was to model all pages using an O-O design, and generate HTML from it. This was 1995, so that last bit was also a large contributor to the “revolutionary” bit. It was also the main reason I was asked to work on it, because I was co-founder of a non-profit ISP with some friends a few years earlier, so I was clearly an expert in Internet technology. (It also proved I was no expert in business; had we gone commercial, I would probably be retired by now) Due to the time it took us to get the database to do what we wanted, combined with nervous managers who wanted to see results, I wrote some small C programs that would read a blob of text from the database and do some variable substitution. The result was not only good for demos, but it became the basis for the finished product. Boss happy, customer happy, bills (and thus salaries) paid, score one for the good guys… right?

Somehow it didn’t give me the warm fuzzy feeling of a “success”. We didn’t do what we set out to do, which was building a O-O modeled GUI and fully using the database for what it could do to support the paradigm. What we did deliver was a hack using ‘old-school’ CGI. What I learned from this is that sometimes you just realize your magnificent world-changing plans won’t deliver in time, and you fall back on just doing what you know will. No need to feel bad about it, just learn about the limits of bleeding edge technology, and move on. Don’t forget about it though, the technology might turn out usable in the future and you have an edge with your knowledge about it.

So how about software quality? Hey! I’ve got an app for that!

Automagic Metrics

Over the years, apart from a steady increase in high-quality IDE’s and automated build servers, software quality metrics have improved a lot. Whereas modular design, fan-in/fan-out, cohesion, and other aspects of well-wrought code have been known to us for a long time, nowadays we can put the data with it. Modern Java development uses automated build tools which can be used to perform hands-free daily builds, yea even on-change builds. Using tools such as Sonar we get amazing views on our code and how well it stacks up to a huge set of metrics. Think of a metric that you read about as an indication of good (or bad) code, and you have a pretty good odds there’s a plugin for it to pull a graph out of Sonar for it. Test Driven Development is also hot, and we’ve got plugins to measure test coverage.

So I know this program with great test coverage… (those dots give it away, don’t they. :-)) Well, let’s skip to the observations: just having tests is not the way to ensuring quality. The better you specify what a method does, the better you can test if it actually delivers on that promise, but don’t trust on it catching all possible fault scenarios. Even worse, if you’re going to judge the quality only on test coverage, you are likely to get loads of light-weight tests that don’t help. Take accessor methods: do I really need to test that “self.xyz = xyz;” actually performs the assignment? But if I don’t, Sonar is going to call me out and put my name on the public shaming-board for not providing tests when I should. If your company uses test coverage of developers’ code for performance reviews, you can bet you’re going to get great coverage, but don’t bet on the quality of the tests.

Code comments are also known to be great indicators of code maintainability, right? However, if you read the general opinion of highly regarded thinkers on the subject, they’ll tell you to only put in comments where they actually provide benefit. How about the kind of comments that end up on the Daily WTF? Should we then forego the usage of such tools? Of course not! Lots of useful programming standards can be enforced using them, and a standardized way to do things enhances maintainability. For the rest, use them to get an indication that something may be amiss.

So how about the customer perspective? Having a happy customer is about the best advertising you can get, as any sales rep can tell you. However, if you listen to the complaints of developers, customers are an unreasonable bunch who don’t know what they want. If they do know what they want, they often don’t understand the implications of what they want. Ok, I think we have a answer to that too.

The Happy Customer

Ok, it sounds cheesy, but they exist. Even better, they’re yours to make! The main point is to treat them with respect and be open about your work. Sure, it has to be a two-way thing; as a customer, you should not expect miracles when you keep a wall between yourself and the development team. Let me put a few observations up front:

• Generally speaking, if you want something you’re not using at the moment, it will change you and the way you work, or you find out you didn’t really need it that much. This is true while going shopping for a new gadget, but also when you order a new application. Take a look at what your first mobile phone caused and I think you will agree it changed you more than you expected. With an fixed-line phone, you tend to start a conversation with “How are you doing,” while the first thing said into a mobile phone tends to be “Where are you?”
• If you haven’t thoroughly investigated what your new toy is going to do to/for you, you cannot predict how it will fit in your changing life. We loved Short Message Services when they were first offered, but what we learned is that we really wanted email on the damned thing. Now that we finally have email, the SMS-cashcow is slowly receding into the background. (Apart from the Japanese of course, who immediately went for email. But then again, they also are keeping FAXes alive, so I don’t know if they’re a good predictor. Anyway, I digress.)
• If it is going to take a long time to get your final product, you’ll find some things should have been done differently. If you have to wait half a year for your new car, you’ll secretly envy the people ordering their cars when you take delivery of yours.

If you’re willing to accept the above, then your guide to happiness lies in having previews as development goes along, being able to provide feedback on what looks good and what, now that you see it, doesn’t, and being allowed to change your mind. Given that the economy is not showing itself from its best side, you might also prefer to be able to break up the work in pieces and use your growing insights to decide what needs to come next and what can wait or even be dropped. Rings a bell perhaps? What you want is your development team to be agile. You want to be able to decide along the way that, even though web pages can be modeled using an O-O paradigm, for a small system you’ll settle for something that just works. You want your code to be maintainable, but it shouldn’t take months to get it to pass some set of strict rules.

Back to Chaos

So how about that study? It asked for a qualitative judgement on scope, schedule and resources. The top influences that respondents pointed at for what made a project “Successful”, were: end-user involvement, management support, clear requirements, good planning, and small steps. Not a single thing about technology or software quality in there. I don’t care if the mark should have been a bit more to the left or right on the graph, I care about these insights. These are all things Agile methodologies have taught us to treasure. I’m not saying you couldn’t do well without Agile, but at least use a methodology that takes these lessons to heart. If you do it well, you’ll see your chances of success increase rapidly.