Agile is Ruthless Automated Testing

Test automation is one of these components whose importance is difficult to understand at the beginning. It's not obvious and by consequence it is often underestimated. Probably, it should have been stated in the Agile Manifesto in a more obvious way (now you need to know what 'working software' means for the signers). What's worse, it is difficult to do and it requires a great deal of work!!  For what??!! Let's see.

Why Automated Tests are important in an Agile Context

Automated tests are important both in projects that use traditional and agile methodologies because they allow to regress all the functionality without effort. However, in agile projects they are more important. Here are the reasons:

1. It supports a sustainable process.

Agile states the importance of a sustainable process, that is a process that allows to deliver features continually in a long period of time (i.e. deliver features each iteration, while keeping the existing ones working). In other words, a process that maintains the "cost of change" stable (not growing exponentially) is needed.

Getting to this point is really hard. In a short period of time (an iteration), new features need to be constructed and old ones need to be tested. As more and more features exist, there is more and more regression work, which becomes very time consuming, tedious and error prone. Automating this work becomes the key point to keep the process sustainable.

Agile suggests there should be an extensive suite of automated tests that cover most of the functionality. This "test harness" should be running continually to maximize the feedback and it should be up to date at all times. This means that for every new functionality added to the project, there should be tests that are added to the test suite.

There is another important point on sustainability. Part of being able to have a sustainable process is minimizing the introduction of bugs during the iterations. Computer systems usually have a large entropy. That means that changing one part of the system may impact other areas which are very difficult to predict just by analyzing static code. Good automated tests provide this information very quickly allowing developers to make the changes and see the results immediately. This continual feedback prevents lots of bugs from appearing, amplifies the knowledge of the developer and also gives the courage to introduce changes in the design or make complicated refactorings.

2. The quality of the code is increased

This is like a turn around of things. The main objective of tests is to regress functionality, but it has been proven that just the fact of having to write them improves the quality of the code under test. Interfaces are better understood and designed and the code is cleaner. It also limits the code to do exactly what is supposed to do, as programming is done to make the tests pass.

TDD takes writing tests to the extreme. Tests are written before the code. The code is then written to make the tests pass. So, the tests drive the development. This technique has many advantages: We can be certain that the software is working at all times (which improves the morale) and there is a very high code coverage which gives the possibility of making bigger changes in the code.

3. It allows the team to be agile

This reason is very tied with the first, but I think it's worth mentioning it in a separate point. The design in agile emerges iteration after iteration. Every iteration, the design of the application could be changed to accommodate the new requirements. This changes should be done in the cleanest way (It is not acceptable under the agile principles to accommodate changes with patches). The cleanest design for the new requirements should be thought out and all changes and refactorings should be performed (otherwise, it wouldn't be sustainable)

There is also the possibility that features that have been completed need to change (possibly in reaction to market changes or because the implementation of a new user story so requires). Agile is prepared for very dynamic environments and therefore it should be prepared for these changes. It's nothing but the test harness what allows to make "big" changes or turns of direction.


4. It improves speed

With all the work involved in creating and maintaining tests, it can be easily derived that a project with automated tests would take longer to finish. Well, this is not often the case. In many occasions the speed is improved. The main reason for this is fewer bugs are introduced along the way. The flow is stable and, at all times, the code is known to be working. Even more, if there are bugs, these can be fixed with a fewer probability of introducing a new one (minimizing the effect that Brooks called: one step forward, two backwards)

Dean Wampler's post explains this in more detail:

Basically, what the graph represents is a comparison between a traditional project without automated tests (orange line) and a project that uses TDD. When the project is reaching the end (up until this moment the speed seems to be greater) QA starts working and discovering errors leading to a big increase of work. Not having integration tests leaves the possibility of discovering integration problems as well. After developers start fixing bugs, new ones can be introduced more easily because there is not test suite that provides feedback on the changes made. Finally, the project ends with all the stress derived from having "discovered" all the problems at the very end. On the contrary, a project that uses TDD has more work at the beginning, but then it maintains a more stable flow of work. At all times, the code base is known to be working and therefore the expectation is to have a smooth ending.

5. It lowers Maintenance Costs

Most of the systems have a much greater cost in their maintenance phase than in the construction phase. A lot of systems that take only a few months to be built are years in production. The cost of maintaining projects without automated tests is considerably higher than maintaining one with a good test automated suite.  Fixing a bug many months after it has been introduced is extremely difficult. Even more, the changes necessary to fix the bugs may have unforeseen consequences. In traditional projects, Brooks reckons that fixing a bug during the maintenance phase has a 20% to 50% probability of introducing a new bug. He mentions that the reason for this high probability is again the high entropy of computer systems and also the possibility that these bugs are solved by people who was not involved in the construction of the system.

An automated test harness is invaluable in these circumstances.The tests allow the developer who will fix the issue to understand the logic of what needs to be solved. After introducing the fix, it allows him/her to be more confident that no new bugs have been introduced.


6. It is the best documentation

Automated tests represent the best possible documentation. Conversations with the Product Owner and the different stakeholders live into GUI tests and design discussion live into integration and unit tests. The key aspect is they are executable documentation. They are much more valuable than static documents like models and written documentation because they provide continual feedback. Besides, they are written in a language that provides no ambiguity.

7. It supports other practices

It is important to point that test automation provides a lot of advantages per se, but it also supports other practices allowing to reach the synergy Beck talked about in his first XP book. Automated tests can be a great help in requirement specifications (with the latest BDD approaches, the Product Owner is able to write the requirements as executable specifications). It supports coding: Agiles strives to have clean code that works at all times and automated tests are a great help in this quest. It supports collective ownership, as automated tests provide great help to understand certain aspects of the functionality. It supports refactoring. In fact, it would be impossible to refactor continually if there isn't a test harness that prevents the introduction of bugs. In conclusion, if it is performed correctly, it makes all the other practices work better. If it is not done at all, other practices are debilitated. The whole approach relies on having automated tests to drive the project.

Why don't we automate them then?

Which are the ropes that prevent us from doing it, even if we understand all the advantages?.

1. Tight schedules

Whenever we feel the pressure of the schedule, we stop writing tests (or don't fix the tests that are broken). After all, tests don't constitute the core of our system. Of course, this has direct consequences in the near future.

Ward Cunningham coined the "Debt Metaphor" to explain the refactorings they were doing at WyCash product. He explained that the knowledge gained during the project need to be reflected in the code. In other words, the code should look as if it was written with all the knowledge in mind, while at the beginning fewer things were known. To justify the design changes and refactorings, Cunningham drew the analogy to financial debt. The needed changes represent interest, which if it was not paid would inevitably increase and would inevitably have to be paid later at greater costs. This analogy was expanded to all tasks in software development that might be postponed (tests, TODOs, etc.). Postponing them generates an immediate increase in speed at the cost of creating debt. If this debt is not paid soon, greater interests would have to be paid later.

2. Technical Difficulties

Writing tests is really difficult. It takes time to understand how unit or integration tests should be written (and a bad written test is useless or even worse: it creates a false sense of security). Besides this, there is also the need of investing time and effort in creating an infrastructure for testing (if there isn't anything that could be used).

A Project's Strategy towards Automation

Crispin/Gregory talk about defining an strategy towards automation. An automation strategy defines the plan to deal with automation in a particular project. These are just one of the few things that need to be defined:

- Layers that are going to be tested
- Tools that are going to be used
- The process that is going to be followed to commit code, create the tests,  what will be done if the build is broken, etc.

It is important to have everything in mind when thinking about the strategy

Technical factors, such as:
- Available frameworks and tools
- Familiarity of the developers with the frameworks and tools chosen
- Initial necessary investment (in developing the tools or training)

and factors more related with the business, such as:
- Importance of the project for the enterprise
- How much business involvement will be during the construction
- The life of the product post implementation

Each project is different, so each project will have its own strategy (i.e. there isn't a best strategy). Green field projects will have a different strategy than projects with legacy code. The technologies chosen for the project are a decisive factor as well (not the same to do an outlook plugin, an iphone application, a CRM, a mesh, etc).  All the team should adhere to this strategy, otherwise it is doom to fail.

Conclusion

Test automation is hard, time consuming and doesn't add value directly. However, not doing cripples many of the Agile benefits. An Agile project with no test automation is not sustainable, its quality decreases and has many chances of taking longer. In a project without test automation, is hard to see where you are standing (Is everything working?). And it definitely requires more repetitive, boring and error prone work.

Every project is different. The needed product is different, under different circumstances and with different groups. Everything needs to be taken into account to define a strategy towards automation. The strategy should be one that everyone believes and commits to.

References

1. Lisa Crispin, Janet Gregory - Agile Testing: A Practical Guide for Testers and Agile Teams.
2. http://www.scrumalliance.org/articles/106-definition-of-done-a-reference
3. http://www.agile-software-development.com/2007/07/definition-of-done-10-point-checklist.html
4. http://blog.objectmentor.com/articles/2007/09/30/why-you-have-time-for-tdd-but-may-not-know-it-yet
5. http://gamesfromwithin.com/?p=50
6. Kent Beck - Test Driven Development - By Example
7. http://www.youtube.com/watch?v=pqeJFYwnkjE
8. http://en.wikipedia.org/wiki/Technical_debt

Agile is Clean Code

But ... Isn't this a requirement for all approaches? Why do I have the impression that in Agile, the technical aspect is more important? When I was at University, I remember vividly having heard (and unfortunately later repeated) that the programmer had one of the less important tasks in the chain of producing software. The most important, more senior and smarter people are the architects/designers that analyse and design the application, giving the programmers some diagrams that they 'just' have to translate into code. That seemed like an easy task!! 
   

What means Clean Code?

Ron Jeffries says that we should write "clean code that works". Uncle Bob then made an entire book of Clean Code. What is clean code? Following uncle Bob's explanation, at the high level, it is code that is well structured and which follows SOLID design principles. At a lower level, it's code that has meaningful names, good and useful comments, elegant functions and formatted code.  It's code that expresses the intent of the developer in a clear way. It's code that contains no duplication. And is code that is easy to understand.


And how do you know it works? You don't know because you code it and you are a great programmer :-). You know it because there are automated tests that certify it. Agile development relies on test automation to know that features work and to know thay they do all the time. Beck says "Software features that can't be demonstrated by automated tests simply don't exist". Test automation is one of the pillars of clean code because it allows to continually refactor, it allows to redesign and ultimately it allows to deliver a continuous flow of value.

Why is so important in Agile?

These are a few reasons to justify the 'pain' of writing better code:

1. Cost of Change is lowered: Agile welcomes change, even late in the development process. However, to be able to change, it should be cheap to change. Code that is well designed, easy to read and thoroughfully tested is easy to change and can be changed without fear of introducing new bugs. On the contrary, code that is hard to understand, is not well modularized and has no automated tests is difficult to change and the risks of introducing bugs while modifying it are many.
2. Iterative & Incremental Development means more time coding: In sequential development, there are big phases of analysis and design before the coding phase starts. When the code phase starts, most of the design decisions should have been taken, so development proceeds fast. In Agile, development starts earlier so this phase takes more time. As the code base needs to be manipulated for a longer time, it pays to have it clean. Reading Clean Code, one of the a-ha moments that I had was in the chapter where Martin explains that in the life of a system, the majority of the time is spent reading code, not writing it. I started thinking about my experience, and that was entirely true. I have worked in many systems that already have a lot of features in it. In those systems, each time that I needed to add a new feature, I spent most of my time finding where to put the feature and how to accomodate it to the existing features than actually coding it.
3. Iterative & Incremental Development means the code is changed frequently: Design is performed incrementally in Agile. Every iteration, new features are included in the code base. The design of the system should be one that cares about all features included until that moment. Therefore, this means that a lot of times the existing design needs to be modified. If the code is not maintained clean, understanding and modifying it may become difficult in a short period of time. 
4. Working software is more important than Comprehensive Documentation: This is what the Agile Manifesto says. But having the code base in good state, with good comments, good names, well modularized and with a comprehensive set of automated tests is the best documentation that a developer may have. Been able to understand a piece of code easily because it's simple and it expresses its purpose with clarity is much better than going through UML design documents. Being able to run and debug a set of unit tests for certain functionality provides the best tool to understand that functionality. After all, what really happens is what is in the code, not what is in the documentation. Code is always up to date while documentation is hard to maintain. 

Why don't we do it then?

Lack of skill? Sometimes... This is a craft. A difficult one and we need to spend a lot of time getting better on it. Commitments.. Yes. Our own commitments and the commitments of our managers. Being under pressure and with the deadlines on sight, make it 'work' takes another meaning (I mean, you just want to throw some code that at least compiles and shows something on the screen). I won't follow. There are always plenty of reasons to do things worse!!

Humm.... but Scrum doesn't include a topic on technical practices

I've seen great debates last year over the lack of technical practices in Scrum. Some people believe it is a shortcoming of Scrum while others say Scrum is a product development framework and therefore it is out of scope to include a technical section. Without entering into this debate, something that I've seen a number of times is software development teams starting 'just' with Scrum, leaving the introduction of some XP practices like TDD for the future (if you take a look the popularity of XP and Scrum, I guess many people observed the same!). This may be justified by an approach where changes are introduced gradually (after all, Scrum per se represents a major change at the organizational and process level).

But what happens if you start with 'just' Scrum?. Basically, I believe Scrum alone is not sustainable. At least, in software development it is not. As iterations go by and the code base grows, there is more and more work to finish each feature. By consequence, teams cannot keep their commitments, the business people get mad and you hit the Scrum Wall (ouch, hit it many times)

This is still a topic I think about often. Scrum, per se, is not sustainable in software development. However, it is the most succesful process in the Agile world. It seems that leaving technical aspects out of it is both something that has help Scrum become more popular  and the reason of many of its failures in the development world. 

Conclusion

Clean Code is one of the pillars of Agile. Being able to deliver clean code that works greatly enhances the chances of deliver value sustainably and it makes our life as programmers better. Clean code provides the best ROI and the best documentation. Of course it is difficult to do it and it certainly takes a lot more time than writing crappy code. However, by not doing it, we lose a lot of the advantages that Agile claims.

Real Men of Agile Genius

Enter the @VersionOne #AgileGenius sweepstakes. Prizes include an Apple iPad. Takes about 1 minute. Visit http://bit.ly/agile-genius