Wednesday, January 20, 2010

Bugs free software development

This is the holy grail of the software industry, how can we write bugs free software.
Surely there is not such thing, what we can do is improve the development process toward this goal.
My assumption is that programmers will always create bugs as they develop, so I will focus on the safety harness - testing.

We can divide the test into 2 categories:
  • Automatic tests,
  • Human tests.
First let's look at automatic tests; there are 2 kinds of them:

System tests -- tests that check the whole system by performing user stories as defined in the spec.
  • Pros
    • System tests are important because they tests/use the system like real users do .For example system tests run on top of setup installation using the real database and other 3rd party components.
    • It is very easy to design good set of system tests, you just have to follow the user stories that the system should support.
    • One big advantage of system tests are that they are not tied to implementation of specific module, for example you can change the underline algorithm since the user story stay the same the test remain correct.  
  • Cons
    • System tests takes time to run and hard to understand where things go wrong when failed, so in most of the cases the tests infrastructure should be accompanied by report infrastructure.
    • It can be technically very hard to prepare the infrastructure required for for system tests because of the dependencies of 3rd party components.
And unit tests -- tests that written by programmer to test a piece of code before or after writing this piece of code.


  • Pros

    • Unit test are run fast and can be part of the compilation process on the developer machine, so bugs are caught and fix on the spot.
    • Since it run fast it is possible to cover in some case all the input and the output for a given piece of code and this is very nice.
    • Unit test are very easy to fix when fails.
    • Unit tests in general create better code because it force the programmer to think of interface and use its piece of code on the spot.
  • Cons
    • Unit tests are highly dependent on the code they tests, you can not change algorithm without rewrite the tests.
    • Good set of unit tests are very hard to write, it is an art.
A good software development process should rely both on unit tests and on system tests as part of the development process.
The first line of defense should be the unit tests, while the system test should be used to verify that the old user stories still working.

What about human tests?
First it is clear that there should be phase of human tests in or after the development process
this is because automatic tests will never give you 100% cover in nontrivial software.

I myself think that it is better to push the human tests as early as possible in the development process because:
  • That way you always know where you stand.
  • The more early you find bug, the easier it to fix it.
The real difficulty in this case where the team contains programmers and testers is how to make the testers citizens with equal rights.

5 comments: