Unit Tests

This time I will briefly describe the many benefits of Units Tests then later look at their limitations and challenges and how to overcome them.

What is a Unit Test?




The term UNIT TEST was once a synonym for module tests, referring to any test of an individual unit or module. Originally done manually, such as using some test rig to enter input and check output, then later by stepping through code in a debugger. This was typically only performed once a module had been implemented or an enhancement completed.



Nowadays, the term Unit Test has evolved to the stricter definition of a comprehensive set of automated tests of a module that can be run at any time to check that it works.

I'm sure you noticed that I have talked about Unit Tests in past posts, when relevant. That has meant I have mentioned them in almost every post since they have benefits in many areas of software development, particularly Agile software development. It's now time to tie everything together (and even put a nice red bow on it for Christmas :).



The idea for what are now called Unit Tests has been around for many years. (I myself independently discovered them idea in 1985, giving it the possibly more accurate but duller name of "automated module regression tests".) However, the use of Unit Tests has only really started gaining momentum in the last decade or so since the rise of XP (Extreme Programming), and TDD (Test Driven Development).



Better Design



I talked a little about how Unit Tests help with software maintenance last week, and I will summarize again below, but first I will explain how Unit Tests can even assist with the original design and implementation before we even consider changing it.



There is a lot of evidence that creating Unit Tests, at the same time as you write the code to be tested, results in a much better initial design. It helps the programmer to think about things (boundary conditions, error conditions, etc) that are often ignored in the rush to get something completed.

creating

Unit Tests ...

results in a

better

initial design

TDD (which is, in a way, an extension of Unit Testing) further improves the software quality, for example in verifiability (see ). It also assists in ensuring that the Unit Tests themselves are correct.



Another advantage is that creating Units Tests, while writing the code (eg, using TDD) means that bugs are found much earlier. This allows the design to be suitably modified before it becomes ossified.



Finally, the main advantage to the design is that Unit Tests make it much easier to resist the urge to add unnecessary code. This is the idea behind YAGNI in XP.



I could write a whole post on YAGNI (and probably will) and have also briefly talked about it in . In brief, it is the idea that you do the absolute minimum to implement something.



Why are extra things done anyway? There are lots of reasons:



* a future extension is thought to be certain

* adding it now avoids later costs of change (see below)

* something is seen as a nice addition for the user and is trivial to add

* the developer wants to try something more challenging or interesting

* the designer wants to try a new technique that would look good for their CV

* a more general problem is solved in the name of reusability, rather than the specific one required

* anticipated code degradation means it will be hard/impossible to add later

* distrust of code maintainers to later add a feature properly

* as an attempt to avoid repeated regression testing when things are added

* finally (and ironically) changes are made to make it more maintainable



The consequences are:



* a more complex design, which is less easily understood

* undocumented features which are not debugged and tested properly

* when software is rewritten or just refactored, undocumented features are forgotten or inadvertently disabled - this can be very annoying for users

* reluctance to refactor from fear of breaking undocumented features

* can often constrain the design making future changes more difficult

* usually makes the code less maintainable



Note that there is one consequence that I have not touted here. I have seen it said (for example see the ) that it is faster and requires less effort to create a simple design. This may sometimes be true but in my experience it is often harder and more time-consuming to create the simpler design than a more complex one.

Unit Tests mean that most of the anticipated problems with YAGNI disappear. You don't need to add the feature now because Unit Tests allow you to add it easily later. That is they reduce the costs of change. Moreover, the code does not degrade and maintainers are more likely to make modifications in a way that does not subvert the original design, because they don't have to fret over introducing new bugs, and good Unit Tests can even guide them how to make the changes,



Further, I have found that the use of Units Tests with a simple design makes it easier to maintain than a design which is explicitly built to be maintainable.



Finally, Unit Tests free developers to concentrate on the task at hand, rather than being distracted by tasks they are not really good at (like predicting future user requirements).

Handling Change



In brief the problem is that the costs of changing software are traditionally very high, primarily due to the risk of introducing new bugs (and the consequent problems) and the need for regression testing. These costs cause two effects:



* resisting change, which is bad for the long-term viability of software

* emphasis on avoiding mistakes causing an unnecessarily large up-front effort



Resisting change can result in lost business opportunities simply by avoiding adding new features. Further, the code is not refactored to improve maintainability or take advantage of a better design. Further still, improvements due to external technical advances (eg, improved hardware or software libraries), are often missed.



Worst of all, even when changes are made they are not done properly for various reasons like:



1. It may be very hard to make the change in a way that is consistent with the original design. A good example of this is given at the end of Item 43 in Scott Meyers book EFFECTIVE C++, where multiple inheritance is used to avoid making the correct changes to the class hierarchy.

when changes are made

they are not

done properly

2. The changes may affect other people. For example, a library or DLL from another section may need to be changed. There can be a reluctance to ask others to make changes that might appear to be for the sake of one's own convenience. I gave an example of how this happened to me in a previous post - .



3. Often making changes properly may increase the chance of new bugs appearing (or old bugs reappearing). An example, is the common practice of cloning a function, or module, to handle a change, and making slight modifications for the new circumstance. This preserves the original function or module so that there is no chance of bugs being introduced in existing behavior; but the consequence is that there will be duplicate code, which violates the DRY principle, and causes a maintenance problem.



4. The possibility of introducing new bugs may require large amounts of manual regression testing. This can have a large cost and is usually very tedious for those that do the testing.



Finally, the problem of spending an enormous effort up-front to get the specification RIGHT FIRST TIME has many well-known, undesirable consequences. First, it makes managers very nervous when there appears to be a large cost (wages) with very little tangible evidence that anything has been accomplished. The analysts also soon realize that they don't really know what is required and/or what they are doing and that there is no chance of getting the analysis and design right first time. There is a large amount of effort which could be better spent in another way - the Agile way.



Agile methodologies generally help greatly to reduce the cost of change by catching bugs (and other defects) much earlier. But Units Tests especially enhance this aspect of Agile methodologies by:



* detecting bugs earlier (reducing the cost of delay)

* make code more maintainable (reducing cost of rework)

* allow changes to be made properly

* making refactoring easier and less risky

* work as a form of "living documentation" making changes easier (see below)

* even guiding developers to ensure that future code changes are done properly



Documentation



Technical documentation has always been one of the biggest problems in software development. Studies have shown (at least in Waterfall-style projects) that poor technical specifications are the major reason for project failure, followed by poor estimations (which are actually due to incomplete specs).



Before I talk about the relationship between Unit Tests and documentation, let's just look at the problems and their causes. There are a lot of problems with technical documentation which I group into three rough categories:



1. ACCURACY. One problem is that documents are simply incorrect, and the reviewing/proof-reading required to remove all errors would be onerous. Of course, another major problem is that they are incomplete, often with gaping holes, and no amount of reviewing is going to find a gap that nobody has thought of.



One way to attempt to overcome this has been to require people with authority to sign off on a document (presumably after having read it). That way, at least you have somebody to blame when things go wrong! Having more people read and sign-off increases that the chance of spotting mistakes but then, of course, the blame-ability for each individual is diluted.



2. UP TO DATE. One reason documents are incorrect is because things change, usually a lot. Documents, even if correct initially, are invariably never in accord with the software at any particular point in time. One of the main reasons, is finding time to update them. You don't want to keep modifying something when you suspect it will change again in the near future. The end result is you keep postponing updating the document until it becomes irrelevant and everyone has forgotten about it, or the task is so big that you can never find time.



One reason documents are not updated is that they are hard to verify. It can be very difficult to check if there is a discrepancy between the actual software and the documentation. Even though the document and the code are very closely related there is no direct connection between, except via the brains of the developers. This is another good example of DIRE (Don't Isolate Related Entities).



3. UNDERSTANDABILITY. Finally, documentation is difficult to read. There are many reasons, like being too vague, including irrelevant information, bad grammar and incorrect terminology, etc. A common problem is assumed knowledge on the part of the reader - a good introduction/summary at the start of a document is almost never done but can be a great time-saver and avoid a lot of confusion.



The lack of a summary is symptomatic of the basic problem - the author is entirely focussed on getting all the facts down. This is like what happens with a newbie programmer - they are so focussed on getting the software working (ie, correct) they give no regard to other important attributes (like understandability). Unfortunately, document writers rarely get past the "newbie" stage, being mainly concerned with correctness not with the understandability of what they write.



FAVOUR WORKING SOFTWARE OVER COMPREHENSIVE DOCUMENTATION



It is no mistake that this, the second of the four points of the , deals with documentation. And there is no better example of favouring code over documentation than Unit Tests, since they are working software which actually obviates the need for much documentation. On top of their other advantages Unit Tests work as a form of living documentation which records not only how the code is supposed to work but also shows others how to use it.



Most documentation is poor, but even with the best you are never certain you understood it correctly. Most programmers create a little test program to check their understanding. With Units Test that little test program is already done for you.



There are many other ways that Unit Tests work as an improved documentation. By now, you probably get the gist so I will just create a list:



* correct - unlike documentation mistakes are immediately obvious

* verifiable - you can easily check if Unit Tests are correct by running them

* understandable - working code is much easier to read/test than documentation

* modifiable - Unit Tests allow code to be more readily modified

* up to date - Unit Tests (if run and maintained) are never out of date



Of course, like documentation Unit Tests can be incomplete (and often are). This is something I will talk about in the next blog, but for now I will simply say that code coverage analysis can help to ensure that tests are reasonably complete and up to date.



Organization



Finally, Unit Tests are good for organizing your work. You develop a rhythm when using Units Tests (and particularly when using TDD), of getting into a cycle of coding/testing/fixing. Somehow it becomes more obvious what you need to do next. For example, after implementing a feature you just "fix the next red light" until all Unit Tests pass.



Summary



Once you start using Unit Tests you keep finding more things to like about them. Your design is more likely to be correct and more verifiable. Further the initial design can be simpler because you don't have to try to guess what will happen in the future. And when things do change, enhancements are made more quickly and reliably and bugs are found faster. Best of all changes can be made properly and the code refactored without fear of introducing bugs.



However, the main point I was trying to get across is that without Unit Tests I don't believe an Agile methodology can work. (This is one of my criticisms of Scrum - that it does not prescribe Unit Tests as essential to the methodology.) Unit Tests allow the software to start out simple and evolve into what it needs to be. They allow us to resist trying to predict the future.



So why aren't they universally used? First, there is the cost - I don't think the cost/benefit ratio is truly appreciated. Also there are several challenges with Unit Tests that I will explain how to overcome in my next post...
Full Post