Agile is Iterative and Incremental Development

Definitions

Agile is iterative and incremental development. Iterative and incremental development was borne in response to the weakness of the Waterfall model.

In Waterfall, the product is developed sequentially, delivering all the product at the end of the last phase:

from wikipedia: http://en.wikipedia.org/wiki/Iterative_and_incremental_development

The problems with this approach are:

- Specifying all the requirements at the beginning is very difficult. When the customer is able to see a working product, is very expensive to make changes.

- It's very risky. Developers starts facing the real problems after 2 entire phases, which could be in the middle of the project. A lot of money has already been expended at this moment. 

The Agile approach to overcome these problems is to deliver the software in increments. In each increment, activities from all these phases are included.

from wikipedia: http://en.wikipedia.org/wiki/Iterative_and_incremental_development

It is important to understand the different between "incremental" and "iterative": 

In incremental development, work is break up into smaller pieces which are scheduled, developed and integrated when completed. An increment is an increment in business functionality. In other words, an increment in Agile delivers something new to the customer which s/he can see and experiment with. To be able to accomplish this, work is break up based on functionality. A feature is developed in its entirety, from the UI to the back end in an iteration. 

Iterative development means the product is built in iterations, where each iteration expands the product until the project is completed. As the process is also incremental, each iteration includes parts of all phases (i.e. in each iteration, there is some analysis, some design, some code and some testing). 

When researching to write this post, I came accross this article (http://www.stsc.hill.af.mil/crosstalk/2008/05/0805Cockburn.html) written by Cockburn. Something that caught my attention about this article is he states that if after an iteration is completed and the feedback from the customer is received, there isn't time to incorporate these changes, then the process isn't really iterative. There should be scheduled time to rework, to apply the changes the customer requested, to revise the performance, to redesign or the "iterative" part disappears and we are following into the same Waterfall mistakes. To be honest, I don't understand very well why it would stop being iterative. The iterations are still there and would expand the product in each iteration but not where the customer is expecting. To be even more honest, I always saw them as one entire concept. If the delivery is incremental, is there any other way to do it other than iteratively (recursively?).

Pros & Cons

Basically the advantages are the disadvantages of the waterfall model:

+ The development team and the product team learn from what has already been constructed. The development teams face the difficulties early in the project life. The product team starts looking at the product early as well and starts materializing their ideas and suggesting the necessary changes needed to obtain the best possible product. 

+ Value can be delivered early. Doing incremental development is the basis for being able to delivered a partial release.

+ Risk is reduced. Developers "attack" the riskiest features early in the project's life.

+ It's the approach to take when exploring, when working on a not known domain. It's the way scientists work. They make an hypothesis and try to prove it via experiments. Spending a lot of time architecting and designing does not make a lot of sense in these types of projects. 

What are the cons then?

- The only disadvantage I can think of is it's difficult. Squishing all phases into a short iteration presents many difficulties to all members of the team. Developers need to accommodate the feature to the existing design. Testers need to start thinking about the test cases in the air, when the feature doesn't exist yet. All members of the team need to work concurrently on the same feature, which requires the team to coordinate the work very well. Add to this that development needs to be sustainable and therefore completing a unit of work means not only delivering the functionality. It means delivering it with the best possible design, with clean and refactored code and an automated suite that can regress this functionality in the future.

Architecting and designing in an iteration....?

This is perhaps the most controversial topic in incremental development. Detractors of Agile usually ask where (the heck) does the architecture and design fit into an iterative and incremental approach!!!

At the beginning of the project, there is a period where the vision is defined and the basic requirements are gathered. After that, there should be time to define the most important aspects of the architecture. The length of this time would vary depending on many factors. If the  project is complex, but there is a lot of certainty it makes sense that this period is larger. As certainty decreases, spend time at the beginning makes less sense as conditions will probably change. Remember Agile needs to maintain the curve cost low. Therefore, decisions that later would imply big costs should be taken at the beginning. Highsmith calls this making a "balance between anticipation and adaptation". Anticipation is up front design. Adaptation are the changes needed to react to different conditions in the course of the project.

When iterations start, for each new feature that is constructed, there is design. The design consists in accommodating the new feature in the best possible way. I don't know how you feel, but I feel design is performed with better information. It is performed based on tangible components. Personally, I do the design understanding the problem better.  Of course the design for the existing features may change to make the new design that contains all the features a better design. Don't forget that Agile is also about being able to deliver value sustainably. If design deteriorates with each new feature added, technical debt starts rising and it becomes more costly to add new features in the future. Another important point is Design is made with the YAGNI principle in mind: "Do the simplest thing that could possibly work". Let the abstractions be borne as they are needed. Don't make decisions based on assumptions about the possible necessity in the future. Ah, one last important point: "Simple is not simplistic". The design of each feature ends up with the simplest but cleanest and most elegant way you could possibly think of (I say you could possibly think of because making good designs is not something that you'll do the first day you start developing - that doesn't prevent you from following these principles, as I do :-))

Being able to change the design in each iteration relies on other technical practices such as refactoring, continuous integration and automated tests. Without them, making changes would be very risky. Refactoring in each iteration allows to have the cleanest code base each time a new change is required. I believe anyone that has developed knows there's always a couple of options to do something: the clean way and the hack. The hack is the quick and dirty option, useful in the short run and painful in the long run. The cleanest way includes refactoring and of course more work. Automated tests provide a harness that allows to apply the changes in a safer way. It would be impossible to make continuous design changes if it weren't for the automated tests that provide certain security that the existing features keep working after the changes. 

To Timebox or not to Timebox

I never thought about it until I read Johanna Rothman blog... Every time you have a task to perform, you have 2 options to tackle it. If you understand very well the scope of the task and is something that can be done in a reasonable amount of time, you just go for it. You work with the objective of finishing the task. What is the problem with this approach? If the objective is blurred or the task is very big, there is no sense of progress and we can easily fall into extending the task ad infinitum. Timeboxes bring the reality of date constraints to the task at hand. If you plan to tackle the task with timeboxes, you allocate some time, estimate what you would be able to complete in that period of time and then review the progress and take decisions based on it.

Timeboxes bring these benefits software development:

- "Timeboxing forces closure, it forces the team to deliver something concrete" (Highsmith). When exploring, it is very difficult to scope a feature. Teams are discovering and therefore, times can expand indefinitely. This contrasts with the date constraints that projects always have. Timeboxes force a date constrained exploration and it forces the development team to focus on delivering something.

- Timeboxes are short term adventures, marked by decision points at the beginning and at the end. These decisions points allow the business to take decisions. The word "short" is important here. Timeboxes are short while scope-boxes tend to be larger. 

- Timeboxes create a cadence. The team enters a cycle that repeats every 2/3 weeks. After that period, the team review how it performed, adapts and recommits for the next iteration.

So what will you do, timebox or not timebox? Scrum and XP use timeboxes while Kanban doesn't enforce them. To make the decision, bear in mind what timebox provides. I have seen a few examples where the teams were struggling to work in timeboxed iterations when it just didn't make sense to them (e.g. production environments working with Scrum where they were getting requirements all the time and therefore they couldn't freeze a chunk of work for X weeks) or the exploratory factor was very low (basically the user story always takes a certain amount of time). In these cases, timeboxing doesn't add anything and I would just opt for a scopebox approach. On the other side, if the project is highly exploratory, the timebox iteration allows the team to focus and close a piece of functionality. It forces them to scope that piece of work and not expand to provide features not required at this time. Whether you timebox or not doesn't make you more or less Agile. At least, I don't see any reference in the Agile Manifesto to timeboxing.

Conclusions

"Iterative, feature-based delivery is the cornerstone to early and often delivery of value" (Highsmith). Being able to produce a partial product allows to have a common ground for the product and the development team to discuss and keep delivering value. During an iteration, a lot of new knowledge is gained which is used by business people and developers to make better decisions in the future. The technical risk of the project is tackled as early as possible. Of course iterations don't start immediately. All Agile project start with the basic architecture and a bunch of important decisions. When iterations start, design is done incrementally. Being able to design incrementally relies on doing other engineering practices, such as refactoring and automated tests.    

Update 2/23: I just made the click with the meaning of an iteration. The key is that an iteration is fixed length. This is in contrast with a pull system like Kanban that pulls the work. So Scrum/XP are incremental and iterative and Kanban is just incremental. I am thinking that perhaps the title of the post should be changed to "Agile is incremental development" as the agile manifesto doesn't say anything about being iterative. (At least, this is my interpretation of this principle "Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.")