Software Quality is a nice subject to write about. So much to say with plenty of viewpoints to choose from. Wikipedia tells me it is about functional quality (how well it complies  or conforms with design or requirements), or structural quality. (the non-functional aspects) There’s an ISO standard on Software Quality Requirements and Evaluation, and another one on Process Improvement and Capability Determination. Via that route we can then continue into the process for making a quality product and invoke CMMI; the integrated model for maturity capabilities.

Actually, there’s so much on the subject you’d wonder why it even is an issue. I personally think there is a simple reason for all this: for software, quality is an inherently vague quality! [sic]

The Quest for Quality Software

Given that software development is a process, Quality Assurance or Quality Management are obvious subjects to address. From management, the directive to “do something about” the quality of the software produced is something you can expect, and management reporting should include metrics on the subject. Budget is allocated to procure software, either bought or built, so to ensure the budget is well-spent, we need to address this. However, there is an obvious difference in the details; management simply wants “Quality Improvement”, whereas the delivery teams will need to know exactly which aspects of quality to spend time on. If you have a modern Java development setup, you’ll likely find a build server producing daily (if not more often) builds of the sources, paired with several pages of metrics on the source code itself. Programming standards will have been defined, source code formatters tweaked, and teams know they need to fix any finding up to a certain level to meet the Definition of Done.

So how do we go about choosing those threshold levels? Even more important, when will “enhancing quality” become “gold plating”? If you’re not an experienced developer yourself, you might be surprised how heated the discussions can become on those subjects. Whereas a traditional shop with high documentation standards may very well be able to fill up a large piece of shelf with what they see as the minimum required set, modern Agile approaches focus more on the code as the primary documentation, preferably even generating the documentation from specially marked comments in the code. And speaking about that code itself; a consistently applied structure may render what a newcomer views as unreadable spaghetti, into easily recognisable and maintainable code. At the same time, a single bad experience may easily result in raising the threshold beyond levels thought acceptable by a professional auditor.

Quality Software as in Quality Cars

I’m an Architect, usually of the Software kind, so I like analogues: let’s look at cars. If you want to talk about a high-quality car, I suppose brands like Mercedes, BMW and Lexus come to mind. If you ask people who are in search of a new car, they will naturally place quality in the list of aspects they base their choice on. So why do so many of them buy cars that are not on the top quality list? Don’t they want high quality? Sure they do, but for most of us mortals quality loses to value. The Oxford Dictionaries tell us that “quality” can mean “the degree of excellence of something”, which Wikipedia puts as the “business” use of the word. (the non-inferiority or superiority of something) However, this comes at a price, so we need to look at the word “value” as well, which is shown with primary meaning “the regard that something is held to deserve,” acknowledging that it is essentially a subjective metric. A bit further down we get the meaning of the word most appropriate here, which is “the worth of something compared to the price paid or asked for it.”

So is management wrong in asking for “Quality Software”? I dare say not, because, within their frame of mind, it is a valid request. What we, as Software Engineers, need to do is give them what they want, which is a metric for the value of software that best matches their concept of software quality. We need to use that second meaning of quality shown in Wikipedia, termed the use in philosophy, where it simply means “attribute or property”.

Now here’s a Chance for Captain Faramir…

I tried to restrain myself, but failed utterly. Please forgive this avid fan of the Lord of the Rings. As for off-topic remarks, I’ll restrict myself to just mentioning that you should ignore the ridiculous subplot Peter Jackson foisted upon us, of Faramir recognising the dangers of the ring to himself, but at the same time thinking his father might be able to survive its temptations. Both book and film however agree that the ring presented Faramir with a chance “to show his quality.” Peter Jackson lets Sam assure us that it was “of the highest.”

So there it is; the quality of a man. And my problem with it (mind you, not with Faramir, especially not with the original from the book) is that it doesn’t tell me anything I can put a value on; it is not a true metric. Even worse: his father would (at that moment in the story at least) utterly disagree, because he would base the measure of quality of his son on a different viewpoint. When asked if he would prefer Boromir, the elder son, to be alive rather than Faramir, he agreed; The value of Faramir was, according to him, less.

So do we print out a list of all the metrics our SONAR builds produce and ask management to take their pick? Let’s save them the trouble of trying to understand those algorithms: can you choose which of those numbers you want to be reported on? Remember, the value must accurately reflect the state of the software. If the number goes up, we should be able to correlate that with other reports provided, such as happy business managers, service levels in the green, low cost of maintenance, and so on. Sounds to me like a difficult choice. Anyone with a solid understanding of the programming language involved can easily punch holes in any suggestion you would make, and, honestly speaking, would probably be right.

The Pitfalls of Relying on Technical Software Metrics

Take everyone’s favourite darling in these Test Driven Development days, “Test Coverage.”  This metric exists for a good purpose: code not covered by a test is code where we only have the author’s pledge that it works. I consider myself a senior level developer, but I’m human, so I know I slip up sometimes. Pair programming is not an admission of defeat; it is a valuable way to add quality control (couldn’t resist, sorry) and spreading knowledge, if not just simply an acknowledgement that two minds know more than one. But given that software maintenance is often seen as a valuable tool for on-the-job training of new employees, I have to accept that somebody else will “mess with” my code. Having an automated test in the build to verify specs are still met is an extremely useful tool.

So introduce a target, say 80%, and require all code to have an 80% test coverage. Your team is the best in the business, right? Top-flight programmers, all of them. So when they write a piece of code which performs a task which cannot be verified other than in a fully integrated environment, they will add an integration test, not a unit test. Since integration tests tend to be more coarse-grained than basic unit tests, your coverage, which is implemented with a simple counter, will start to drop. Also, some pieces of code are so simple and obvious, adding a test would be like verifying that 1+1 equals 2 on a daily basis, just to make sure mathematics still works as you learned it. And then I haven’t even mentioned the scenario’s where planning pressures make you prioritise which tests get worked out first. (Don’t shoot the messenger, it happens often enough.)

Did I mention that management reporting tends to show up in yearly performance reviews? So what are the chances of developers starting to pay lip-service only to the intent of unit testing in such an environment? Dummy tests might be a simple and effective means to boost your rating, costing little time to produce while increasing your test coverage. Naturally, the real value of the metric is gone.

Towards a Useful Solution

Gee, can I really solve this one? Frederick P. Brooks, in his ever relevant The Mythical Man-month, stated that there is no silver bullet. There’s no single thing that can help you out and magically solve you problem. Software Quality standards do not really allow us to easily define a metric either. Rather, they help to structure and discuss what metrics are available. If we look at the maturity of the software development methodology, CMMI tells us the highest level we can achieve is “Optimising,” and I think that is what we should be looking for. We should have metrics on the process and show they are improving.

On the Management side we need insightful metrics about the value of what is delivered. There are some obvious things to look at, but mostly we’ll need to ask them how they view this. What do they think of as the manifestation of high quality software? Most likely this will concern financials, as that is what forms the bottom line for any corporation, but customer satisfaction should pop-up in there as well. Service Levels and their metrics will play a key part, as Service Management is actually the most direct driving force for software quality, bearing the brunt of what happens when badly written code is pushed out.

Technical metrics, such as test coverage, complexity indexes, or even maintainability scores, are a valuable reporting tool, but they should never form the core of a management report, as their value is indicative, not normative. They are a part of the development team’s own quality management report, allowing them to strive for that “optimising” aspect of their methodology. And while planning for improvements, remember that, just as with any other goods and services, “bang-for-buck” is plotted with an exponential curve. Beyond a certain point you will find that you will not be able to afford further improvement.