Agile is Ruthless Automated Testing

Test automation is one of these components whose importance is difficult to understand at the beginning. It's not obvious and by consequence it is often underestimated. Probably, it should have been stated in the Agile Manifesto in a more obvious way (now you need to know what 'working software' means for the signers). What's worse, it is difficult to do and it requires a great deal of work!!  For what??!! Let's see.

Why Automated Tests are important in an Agile Context

Automated tests are important both in projects that use traditional and agile methodologies because they allow to regress all the functionality without effort. However, in agile projects they are more important. Here are the reasons:

1. It supports a sustainable process.

Agile states the importance of a sustainable process, that is a process that allows to deliver features continually in a long period of time (i.e. deliver features each iteration, while keeping the existing ones working). In other words, a process that maintains the "cost of change" stable (not growing exponentially) is needed.

Getting to this point is really hard. In a short period of time (an iteration), new features need to be constructed and old ones need to be tested. As more and more features exist, there is more and more regression work, which becomes very time consuming, tedious and error prone. Automating this work becomes the key point to keep the process sustainable.

Agile suggests there should be an extensive suite of automated tests that cover most of the functionality. This "test harness" should be running continually to maximize the feedback and it should be up to date at all times. This means that for every new functionality added to the project, there should be tests that are added to the test suite.

There is another important point on sustainability. Part of being able to have a sustainable process is minimizing the introduction of bugs during the iterations. Computer systems usually have a large entropy. That means that changing one part of the system may impact other areas which are very difficult to predict just by analyzing static code. Good automated tests provide this information very quickly allowing developers to make the changes and see the results immediately. This continual feedback prevents lots of bugs from appearing, amplifies the knowledge of the developer and also gives the courage to introduce changes in the design or make complicated refactorings.

2. The quality of the code is increased

This is like a turn around of things. The main objective of tests is to regress functionality, but it has been proven that just the fact of having to write them improves the quality of the code under test. Interfaces are better understood and designed and the code is cleaner. It also limits the code to do exactly what is supposed to do, as programming is done to make the tests pass.

TDD takes writing tests to the extreme. Tests are written before the code. The code is then written to make the tests pass. So, the tests drive the development. This technique has many advantages: We can be certain that the software is working at all times (which improves the morale) and there is a very high code coverage which gives the possibility of making bigger changes in the code.

3. It allows the team to be agile

This reason is very tied with the first, but I think it's worth mentioning it in a separate point. The design in agile emerges iteration after iteration. Every iteration, the design of the application could be changed to accommodate the new requirements. This changes should be done in the cleanest way (It is not acceptable under the agile principles to accommodate changes with patches). The cleanest design for the new requirements should be thought out and all changes and refactorings should be performed (otherwise, it wouldn't be sustainable)

There is also the possibility that features that have been completed need to change (possibly in reaction to market changes or because the implementation of a new user story so requires). Agile is prepared for very dynamic environments and therefore it should be prepared for these changes. It's nothing but the test harness what allows to make "big" changes or turns of direction.


4. It improves speed

With all the work involved in creating and maintaining tests, it can be easily derived that a project with automated tests would take longer to finish. Well, this is not often the case. In many occasions the speed is improved. The main reason for this is fewer bugs are introduced along the way. The flow is stable and, at all times, the code is known to be working. Even more, if there are bugs, these can be fixed with a fewer probability of introducing a new one (minimizing the effect that Brooks called: one step forward, two backwards)

Dean Wampler's post explains this in more detail:

Basically, what the graph represents is a comparison between a traditional project without automated tests (orange line) and a project that uses TDD. When the project is reaching the end (up until this moment the speed seems to be greater) QA starts working and discovering errors leading to a big increase of work. Not having integration tests leaves the possibility of discovering integration problems as well. After developers start fixing bugs, new ones can be introduced more easily because there is not test suite that provides feedback on the changes made. Finally, the project ends with all the stress derived from having "discovered" all the problems at the very end. On the contrary, a project that uses TDD has more work at the beginning, but then it maintains a more stable flow of work. At all times, the code base is known to be working and therefore the expectation is to have a smooth ending.

5. It lowers Maintenance Costs

Most of the systems have a much greater cost in their maintenance phase than in the construction phase. A lot of systems that take only a few months to be built are years in production. The cost of maintaining projects without automated tests is considerably higher than maintaining one with a good test automated suite.  Fixing a bug many months after it has been introduced is extremely difficult. Even more, the changes necessary to fix the bugs may have unforeseen consequences. In traditional projects, Brooks reckons that fixing a bug during the maintenance phase has a 20% to 50% probability of introducing a new bug. He mentions that the reason for this high probability is again the high entropy of computer systems and also the possibility that these bugs are solved by people who was not involved in the construction of the system.

An automated test harness is invaluable in these circumstances.The tests allow the developer who will fix the issue to understand the logic of what needs to be solved. After introducing the fix, it allows him/her to be more confident that no new bugs have been introduced.


6. It is the best documentation

Automated tests represent the best possible documentation. Conversations with the Product Owner and the different stakeholders live into GUI tests and design discussion live into integration and unit tests. The key aspect is they are executable documentation. They are much more valuable than static documents like models and written documentation because they provide continual feedback. Besides, they are written in a language that provides no ambiguity.

7. It supports other practices

It is important to point that test automation provides a lot of advantages per se, but it also supports other practices allowing to reach the synergy Beck talked about in his first XP book. Automated tests can be a great help in requirement specifications (with the latest BDD approaches, the Product Owner is able to write the requirements as executable specifications). It supports coding: Agiles strives to have clean code that works at all times and automated tests are a great help in this quest. It supports collective ownership, as automated tests provide great help to understand certain aspects of the functionality. It supports refactoring. In fact, it would be impossible to refactor continually if there isn't a test harness that prevents the introduction of bugs. In conclusion, if it is performed correctly, it makes all the other practices work better. If it is not done at all, other practices are debilitated. The whole approach relies on having automated tests to drive the project.

Why don't we automate them then?

Which are the ropes that prevent us from doing it, even if we understand all the advantages?.

1. Tight schedules

Whenever we feel the pressure of the schedule, we stop writing tests (or don't fix the tests that are broken). After all, tests don't constitute the core of our system. Of course, this has direct consequences in the near future.

Ward Cunningham coined the "Debt Metaphor" to explain the refactorings they were doing at WyCash product. He explained that the knowledge gained during the project need to be reflected in the code. In other words, the code should look as if it was written with all the knowledge in mind, while at the beginning fewer things were known. To justify the design changes and refactorings, Cunningham drew the analogy to financial debt. The needed changes represent interest, which if it was not paid would inevitably increase and would inevitably have to be paid later at greater costs. This analogy was expanded to all tasks in software development that might be postponed (tests, TODOs, etc.). Postponing them generates an immediate increase in speed at the cost of creating debt. If this debt is not paid soon, greater interests would have to be paid later.

2. Technical Difficulties

Writing tests is really difficult. It takes time to understand how unit or integration tests should be written (and a bad written test is useless or even worse: it creates a false sense of security). Besides this, there is also the need of investing time and effort in creating an infrastructure for testing (if there isn't anything that could be used).

A Project's Strategy towards Automation

Crispin/Gregory talk about defining an strategy towards automation. An automation strategy defines the plan to deal with automation in a particular project. These are just one of the few things that need to be defined:

- Layers that are going to be tested
- Tools that are going to be used
- The process that is going to be followed to commit code, create the tests,  what will be done if the build is broken, etc.

It is important to have everything in mind when thinking about the strategy

Technical factors, such as:
- Available frameworks and tools
- Familiarity of the developers with the frameworks and tools chosen
- Initial necessary investment (in developing the tools or training)

and factors more related with the business, such as:
- Importance of the project for the enterprise
- How much business involvement will be during the construction
- The life of the product post implementation

Each project is different, so each project will have its own strategy (i.e. there isn't a best strategy). Green field projects will have a different strategy than projects with legacy code. The technologies chosen for the project are a decisive factor as well (not the same to do an outlook plugin, an iphone application, a CRM, a mesh, etc).  All the team should adhere to this strategy, otherwise it is doom to fail.

Conclusion

Test automation is hard, time consuming and doesn't add value directly. However, not doing cripples many of the Agile benefits. An Agile project with no test automation is not sustainable, its quality decreases and has many chances of taking longer. In a project without test automation, is hard to see where you are standing (Is everything working?). And it definitely requires more repetitive, boring and error prone work.

Every project is different. The needed product is different, under different circumstances and with different groups. Everything needs to be taken into account to define a strategy towards automation. The strategy should be one that everyone believes and commits to.

References

1. Lisa Crispin, Janet Gregory - Agile Testing: A Practical Guide for Testers and Agile Teams.
2. http://www.scrumalliance.org/articles/106-definition-of-done-a-reference
3. http://www.agile-software-development.com/2007/07/definition-of-done-10-point-checklist.html
4. http://blog.objectmentor.com/articles/2007/09/30/why-you-have-time-for-tdd-but-may-not-know-it-yet
5. http://gamesfromwithin.com/?p=50
6. Kent Beck - Test Driven Development - By Example
7. http://www.youtube.com/watch?v=pqeJFYwnkjE
8. http://en.wikipedia.org/wiki/Technical_debt