The [f..] last story of the iteration

I hate the last story in a Scrum iteration. If we are about to finish it, we sacrifice quality to fit it into the iteration. If it's not completed and carried over, either it's finished in the first day (best scenario, but stays the whole iteration there cluttering the task board...) or expands in a crazy way (2 hours in the last day of the sprint easily become 16 hours in the next iteration!!). WTF.

The Agile Architecture

There are some big graphs, so I published the google doc as well (it looks better): http://docs.google.com/View?id=dhpj7455_67gkwk3bfr

When I started with my first Agile project, I just knew I liked it. I felt better working this way, I felt that we delivered better results. I was thrilled by the quality of the code, I felt good with my team mates.. But they were just feelings. I couldn't explain why. With the excuse of having to complete my MsC thesis, I decided to keep investigating and learning about Agile, . I went to my first Agile course (hey Matt, it was cold in that room!), kept working in Agile projects, read some books, blogs, forums, went to some Agile conferences, became a Certified Scrum Master (wow!) and started having Agile friends :-).


So after 4 years, what is Agile? Well, what I can tell is it is not a methodology (Haven't you read at least a post that starts with "The Agile methodology..."?). The most accurate definition I have is Agile is a set of values and principles listed in the Agile Manifesto... But probably, this manifesto has too much knowledge condensed in a few words. In this post I want to define the Agile architecture (yes, this is upfront architecture!). What I mean by architecture are what I considered the most important components in an Agile development team (I am a developer, so it is from a developer's vision).

This is what I came up with:

First, I defined what I consider the most important objective of Agile: being able to deliver value sustainably, with the required quality and in a changing environment. I did a post recently in which I list why Agile provides better value.

Then I tried to list the most important components and after having them, I tried to categorize them. I think there are 3 categories:

1) Components related to the technical field. I am a developer and so I believe these components are really important in any software development team. Inside this category, I put 3 main components.

    a. Simple Emergent Design: being able to evolve the design in every iteration is crucial to being able to deliver a continuous flow of value. This component is a consequence of doing iterative and incremental development, and relies on having automated tests and keeping the code clean via opportunistic refactoring. It also relies on having the latest code base at all times (continuous integration).

    b. Technical Excellence: this includes opportunistic refactoring, clean code and ruthless automated tests. This component is big as its importance to successful agile teams as it supports doing evolutionary design, it allows to adapt to changes and it supports having a process that relies on having a lot of information on the code itself. 

    c. Continuous Integration: This is a small component that supports other components such as technical excellence and emergent design.

2) Components related with the process in place.  

    a. Light Process: Agile needs a process that allows to "welcome change, even late...", a light process that reacts and adapts to change. The ceremony and the artifacts should be the minimum that allows to coordinate the project at hand (otherwise is hard to turn). The process should encourage introspection and optimization based on it (kaizen). To accomplish this, Agile relies on a self organized but very disciplined team and the software engineering practices included in the Technical Excellence component.

    b. Incremental and Iterative Development:  I am not sure if this component should be part of the light process (probably yes). Anyway, I wanted to stress the importance of having a process where development is performed iteratively and incrementally. 

    c. Self-organized & Self-disciplined: Agile relies on a team of committed and passionate software engineers. Maintain technical excellence relies on a lot of discipline. Self-organize successfully relies on human components: good communication among all team members and the will to collaborate.

    d. Transparency: The process should encourage transparency. Transparency improves collaboration and allows to reduce the number of needed artifacts.

3) Human components

    a. Collaboration: All team members, whether in the product or development side, collaborate with each other. Success is measured for the whole team that participates in the project, not by sub-team or individually. If there is good collaboration, there's no need for artifacts that just have the purpose of make someone responsible and the process can be lighter. 

    b. Communication: The flow of information among all members should be taken to the maximum (possible) as members with more and better information translates into improvements in the results.

These are the areas where I would map the Agile values:

And the Agile principles:

Final Thoughts

I am not sure if this makes sense or not. I know making this exercise allowed me to visualize what components an Agile development team should have. I've been following some threads where they discuss what is missing from Scrum and started wondering: Is there a methodology that follows all the Agile principles? Is it straightforward to map all what is needed to make an Agile Development team from the Manifesto? (as I said before, the Manifesto seems to contain the conclusions of many years of experience of these guys). I am also interested in Process Improvements Frameworks and so I was wondering whether it would make sense to define (as CMMi does) the Agile process areas (what I called components here) and the suggested practices in each area? I am not sure about this..When you are in the Shu level, you need to follow a methodology to start. Perhaps something like this would allow to choose the methodology or know in advance what needs to be addition to the methodology to "comply" with the Agile values and principles.

Agile is Iterative and Incremental Development

Definitions

Agile is iterative and incremental development. Iterative and incremental development was borne in response to the weakness of the Waterfall model.

In Waterfall, the product is developed sequentially, delivering all the product at the end of the last phase:

from wikipedia: http://en.wikipedia.org/wiki/Iterative_and_incremental_development

The problems with this approach are:

- Specifying all the requirements at the beginning is very difficult. When the customer is able to see a working product, is very expensive to make changes.

- It's very risky. Developers starts facing the real problems after 2 entire phases, which could be in the middle of the project. A lot of money has already been expended at this moment. 

The Agile approach to overcome these problems is to deliver the software in increments. In each increment, activities from all these phases are included.

from wikipedia: http://en.wikipedia.org/wiki/Iterative_and_incremental_development

It is important to understand the different between "incremental" and "iterative": 

In incremental development, work is break up into smaller pieces which are scheduled, developed and integrated when completed. An increment is an increment in business functionality. In other words, an increment in Agile delivers something new to the customer which s/he can see and experiment with. To be able to accomplish this, work is break up based on functionality. A feature is developed in its entirety, from the UI to the back end in an iteration. 

Iterative development means the product is built in iterations, where each iteration expands the product until the project is completed. As the process is also incremental, each iteration includes parts of all phases (i.e. in each iteration, there is some analysis, some design, some code and some testing). 

When researching to write this post, I came accross this article (http://www.stsc.hill.af.mil/crosstalk/2008/05/0805Cockburn.html) written by Cockburn. Something that caught my attention about this article is he states that if after an iteration is completed and the feedback from the customer is received, there isn't time to incorporate these changes, then the process isn't really iterative. There should be scheduled time to rework, to apply the changes the customer requested, to revise the performance, to redesign or the "iterative" part disappears and we are following into the same Waterfall mistakes. To be honest, I don't understand very well why it would stop being iterative. The iterations are still there and would expand the product in each iteration but not where the customer is expecting. To be even more honest, I always saw them as one entire concept. If the delivery is incremental, is there any other way to do it other than iteratively (recursively?).

Pros & Cons

Basically the advantages are the disadvantages of the waterfall model:

+ The development team and the product team learn from what has already been constructed. The development teams face the difficulties early in the project life. The product team starts looking at the product early as well and starts materializing their ideas and suggesting the necessary changes needed to obtain the best possible product. 

+ Value can be delivered early. Doing incremental development is the basis for being able to delivered a partial release.

+ Risk is reduced. Developers "attack" the riskiest features early in the project's life.

+ It's the approach to take when exploring, when working on a not known domain. It's the way scientists work. They make an hypothesis and try to prove it via experiments. Spending a lot of time architecting and designing does not make a lot of sense in these types of projects. 

What are the cons then?

- The only disadvantage I can think of is it's difficult. Squishing all phases into a short iteration presents many difficulties to all members of the team. Developers need to accommodate the feature to the existing design. Testers need to start thinking about the test cases in the air, when the feature doesn't exist yet. All members of the team need to work concurrently on the same feature, which requires the team to coordinate the work very well. Add to this that development needs to be sustainable and therefore completing a unit of work means not only delivering the functionality. It means delivering it with the best possible design, with clean and refactored code and an automated suite that can regress this functionality in the future.

Architecting and designing in an iteration....?

This is perhaps the most controversial topic in incremental development. Detractors of Agile usually ask where (the heck) does the architecture and design fit into an iterative and incremental approach!!!

At the beginning of the project, there is a period where the vision is defined and the basic requirements are gathered. After that, there should be time to define the most important aspects of the architecture. The length of this time would vary depending on many factors. If the  project is complex, but there is a lot of certainty it makes sense that this period is larger. As certainty decreases, spend time at the beginning makes less sense as conditions will probably change. Remember Agile needs to maintain the curve cost low. Therefore, decisions that later would imply big costs should be taken at the beginning. Highsmith calls this making a "balance between anticipation and adaptation". Anticipation is up front design. Adaptation are the changes needed to react to different conditions in the course of the project.

When iterations start, for each new feature that is constructed, there is design. The design consists in accommodating the new feature in the best possible way. I don't know how you feel, but I feel design is performed with better information. It is performed based on tangible components. Personally, I do the design understanding the problem better.  Of course the design for the existing features may change to make the new design that contains all the features a better design. Don't forget that Agile is also about being able to deliver value sustainably. If design deteriorates with each new feature added, technical debt starts rising and it becomes more costly to add new features in the future. Another important point is Design is made with the YAGNI principle in mind: "Do the simplest thing that could possibly work". Let the abstractions be borne as they are needed. Don't make decisions based on assumptions about the possible necessity in the future. Ah, one last important point: "Simple is not simplistic". The design of each feature ends up with the simplest but cleanest and most elegant way you could possibly think of (I say you could possibly think of because making good designs is not something that you'll do the first day you start developing - that doesn't prevent you from following these principles, as I do :-))

Being able to change the design in each iteration relies on other technical practices such as refactoring, continuous integration and automated tests. Without them, making changes would be very risky. Refactoring in each iteration allows to have the cleanest code base each time a new change is required. I believe anyone that has developed knows there's always a couple of options to do something: the clean way and the hack. The hack is the quick and dirty option, useful in the short run and painful in the long run. The cleanest way includes refactoring and of course more work. Automated tests provide a harness that allows to apply the changes in a safer way. It would be impossible to make continuous design changes if it weren't for the automated tests that provide certain security that the existing features keep working after the changes. 

To Timebox or not to Timebox

I never thought about it until I read Johanna Rothman blog... Every time you have a task to perform, you have 2 options to tackle it. If you understand very well the scope of the task and is something that can be done in a reasonable amount of time, you just go for it. You work with the objective of finishing the task. What is the problem with this approach? If the objective is blurred or the task is very big, there is no sense of progress and we can easily fall into extending the task ad infinitum. Timeboxes bring the reality of date constraints to the task at hand. If you plan to tackle the task with timeboxes, you allocate some time, estimate what you would be able to complete in that period of time and then review the progress and take decisions based on it.

Timeboxes bring these benefits software development:

- "Timeboxing forces closure, it forces the team to deliver something concrete" (Highsmith). When exploring, it is very difficult to scope a feature. Teams are discovering and therefore, times can expand indefinitely. This contrasts with the date constraints that projects always have. Timeboxes force a date constrained exploration and it forces the development team to focus on delivering something.

- Timeboxes are short term adventures, marked by decision points at the beginning and at the end. These decisions points allow the business to take decisions. The word "short" is important here. Timeboxes are short while scope-boxes tend to be larger. 

- Timeboxes create a cadence. The team enters a cycle that repeats every 2/3 weeks. After that period, the team review how it performed, adapts and recommits for the next iteration.

So what will you do, timebox or not timebox? Scrum and XP use timeboxes while Kanban doesn't enforce them. To make the decision, bear in mind what timebox provides. I have seen a few examples where the teams were struggling to work in timeboxed iterations when it just didn't make sense to them (e.g. production environments working with Scrum where they were getting requirements all the time and therefore they couldn't freeze a chunk of work for X weeks) or the exploratory factor was very low (basically the user story always takes a certain amount of time). In these cases, timeboxing doesn't add anything and I would just opt for a scopebox approach. On the other side, if the project is highly exploratory, the timebox iteration allows the team to focus and close a piece of functionality. It forces them to scope that piece of work and not expand to provide features not required at this time. Whether you timebox or not doesn't make you more or less Agile. At least, I don't see any reference in the Agile Manifesto to timeboxing.

Conclusions

"Iterative, feature-based delivery is the cornerstone to early and often delivery of value" (Highsmith). Being able to produce a partial product allows to have a common ground for the product and the development team to discuss and keep delivering value. During an iteration, a lot of new knowledge is gained which is used by business people and developers to make better decisions in the future. The technical risk of the project is tackled as early as possible. Of course iterations don't start immediately. All Agile project start with the basic architecture and a bunch of important decisions. When iterations start, design is done incrementally. Being able to design incrementally relies on doing other engineering practices, such as refactoring and automated tests.    

Update 2/23: I just made the click with the meaning of an iteration. The key is that an iteration is fixed length. This is in contrast with a pull system like Kanban that pulls the work. So Scrum/XP are incremental and iterative and Kanban is just incremental. I am thinking that perhaps the title of the post should be changed to "Agile is incremental development" as the agile manifesto doesn't say anything about being iterative. (At least, this is my interpretation of this principle "Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.")

6 Reasons Agile Value is Different?

A few weeks/months ago there were some posts related to the importance of value in the Agile world and also some discussions of whether it should be mentioned in the Agile Manifesto. So the question that I had at this moment is how is agile value different from the value delivered by other approaches. These are the 6 reason I can think of to justify Agile value is different:

1. There is a focus on delivering what the customer needs

Waterfall relies on the customer specifying exactly what they want at the beginning of the project. Dealing with changes later is very expensive because the cost of change is high in a waterfall approach (based on Barry Boehm's famous Cost of Change of curve). Agile, on the other side, acknowledges that what the customer is expecting is difficult to express in words/models and can probably change during the duration of a project. Therefore, Agile welcomes change, even late in the development process and adapts to fulfill the customers expectations.

How is this achieved? With communication, collaboration and a lot of feedback. There is a lot of communication and collaboration during the whole duration of the project. The customer actively builds value with the development team. And the development team builds what the customer wants. Conversations don't stop with producing a requirements document or the best model. Conversations continue permanently because conversation adds context to the information. Grasping the business knowledge needed to deliver a product that provides value is really difficult and relying solely on written words is really risky. To allow for more and better feedback, software is delivered in increments, where each increment finishes with running tested features that the user can experiment with. This allows the user to really understand how the product is going to be useful and make decisions based on a working product.

This focus on the customer needs wants to be pushed even further. I read a couple of tweets a few ago where Cockburn was presenting some possibilities for the creation of a User Manifesto (http://alistair.cockburn.us/User+Manifesto). I understand this manifesto as the need to further think on the needs of our customer, on trying to solve their problems better, on fulfilling their expectations with a great piece of software.

2. Creativity and innovation is increased

The interaction among all members of the team (product side and development) added to the freedom people have in self-organized teams boosts creativity and innovation in ways that would be impossible to other approaches. Highsmith states that "one of the tenets of adaptive organization is that innovation emerges from the interaction of diverse individuals, each with it own ideas, who bring information and insight to the development process". Cockburn calls software development "a game of invention and collaboration". The interaction between produt experts and technology experts is what produces this synergy. Business experts provide the input necessary to build a product that adds value and technology experts fix the boundaries of what's possible and suggests other paths that would help create the same value.

It is important to note the difference between a waterfall approach and an agile approach in this regard. While in waterfall, all the innovation happens at the beginning, when the system is being architected and designed, in Agile there is a big leeway to innovate and create during the development process. Added to this, the change from a command and control structure to a self-organized team structure unleashes the creativity of all members (I still remember having heard at University that developers don't have to "think" much. They received the designs and just have to translate those designs into code.. Sounds easy, doesn't it?)

3. Processes are focused on delivering value

Mary and Tom Poppendieck talk about the importance of decreasing the time it takes from the moment where someone has an idea until that idea is converted to a product, something that adds value. From concept to cash.. what's in between? The Agile world has been greatly influenced by these ideas.

The ideas the Poppendiecks brought to the Agile world come from the Lean concepts that Toyota uses in their Toyota Production System (not a good moment to say so?). Lean has 7 principles, of which the first one is "Eliminate Waste". Waste is everything that doesn't add value to the final product. Eliminating waste leaves us with the useful things: the things that add value to the product. To be able to identify waste and improve the lead time, Mary suggested using a tool that is already been used in manufacturating called "Value Stream Mapping". This tool allows to visualize all the flow since a requirement is borned until it is completed, identifying the steps that add value and the ones that merely act as links. Visualizing this flow allows to think about optimizations.

These ideas are very compatible with Agile, that states that working software should go over comprehensive documentation. Working software is what produces value, is the essence. Processes and coordination mechanisms are reduced to the minimum possible to be able to produce this value. The more complex the problem, the more coordination mechanisms, formality and ceremony will be needed but under the Agile principles, it should be kept to the minimum necessary.

4. Value is delivered early

Agile delivers software incrementally. This allows to plan for intermediate releases if the project is big. If you add to this that the most valuables features are produced first, the result is an early delivery of the most important value. 

Comparing to waterfall, where all the value is delivered in the end the advantages are enormous:

- The most important problems are solved first.

- There's the possibility of putting a product in the market faster.

- Progress is measured in terms of value (i.e. in terms of a working product).
- Management can make decisions based on real progress. Their visibility is improved and the information they have to make these decisions is better.
- Partial releases means creating influx, which increases the Return on Investment.

To add to this, there are studies that show that for some systems 80% of the users use 20% percent of the functionality. Being able to delivering this 20% of the functionality provides (as you can imagine) great value.

I am deliberately leaving risk out of this point. Risk is the other factor that should be taken into account when prioritizing features. Risky features have high priority and should be attached early. It could be said that if business or technological risks cannot be overcome, the project won't be able to deliver any value (so it can be seen from the value perspective as well)

5. Value is delivered sustainably

Yeah, it is not only early delivery of value. Agile is also about being able to continue adding value to an existing product in a costly manner. The Declaration of Interdepence states that the objective is a continuous flow of value. 

Being able to deliver value for a long period of time (that's what sustainable means) requires many things in the human area, the process area and the technical area: 

On the human side, it requires a jelled team that pushes always in the same direction (even if the direction changes). It requires understanding and collaboration among members, especially between the product members and the development members. The development team needs to understands the product team decisions and the product team needs to listen to the development team. This could be particularly challenging when the direction changes too often and there is the need to do development rework because the development team could feel that the product team doesn't have a vision/objective. This is why it is very important that all members of the team at least participate in the decisions. Participatory decision making is vital to sustainability and it's vital to keep all team members committed to success. 

On the process side, it requires a light process that allows changes to be accommodated. In other words, a process that allows the team to adapt. A heavy process with a lot of formality and ceremony makes adapting difficult. Agile processes on the other side are designed to react.

On the technical side, it requires simple design, clean code and ruthless automation testing. 

    - Clean code: Design evolves, a lot of changes on the code are made continually and if the existing code is not refactored continually to accomodate all changes, it could become a mess very soon. 

    - Simple design: One of the Agile principles says "Simplicity - the art of maximizing the amount of work not done - is essential.". Ron Jeffries says "Always implement things when you actually need them, never when you just foresee that you need them." (YAGNI principle). Simple design means the team only develops code that servers a purpose. They don't try to foresee and implement features that will be needed in the future. This point could be contentious as well because a lot of people will say that to make a good OO design, we need to see the whole picture. Agile relies on evolutionary design. That is, when a new feature needs to be implemented that forces to change the design of existing ones, this should be done. Abstractions should be created at this moment and the code should be refactored. This can be done because there should be a good suite of automated tests to harness all these changes. The code needs to be clean (at the previous point said) at all times.

    - Ruthless Automated Testing: The main reason of the importance of automated tests is being able to regress the existing functionality almost continually. Automated tests provide the harness to evolve the design and to refactor the existing code without fear.   

All of these requirements aim at maintaining the cost of change low. If the cost of changing gets higher, changing is more difficult (e.g. If the code is not maintained clean or automated tests are postponed, technical debts accumulate and very rapidly delivering value becomes more difficult) and keep adding value gets difficult. In highly exploratory projects or projects with high uncertainty, this becomes specially important. In these environments, Agile methodologies are perhaps the only ones capable of delivering value.  

6. Value is the objective

As it can be seen, there is strong focus in Agile on providing value, on giving the customer what the customer needs. In fact, value is so important that Jim Highsmith suggests replacing the traditional iron triangle for one that includes value.

This is his proposition:

In the old triangle, if "scope" grows either the project will take longer (time is modified) or the cost will be higher (cost is modified). This triangle was used to take management decisions,  and to measure the success of the project. Using this triangle, a project is successful if it delivers the scope, on time and within the required budget. The new triangle has value, quality and constraints in its vertices. A successful project in the new triangle is one that delivers value with the expected quality and conforming to the constraints.

Performance measurements, processes and activities are designed to optimize value. Under Agile, a project that sticks to an outdated plan that doesn't provide any value won't be successful, even if it finishes on time and with the required cost. Processes should support the delivery of value in a sustainable way, even in changing conditions. 

Conclusion

- It is better value because the customer participates in the creation of this value. The customer has responsibility in shaping the product to fit their needs. The product is the result of the collaboration between people that know the business better and people that know how to implement the ideas. There is also more people contributing to creating the product. A group that self organizes and has freedom allows to get the value from all its members.

- Value is delivered earlier because processes and people are focused on delivering value. The most important value is delivered earlier, allowing to maximize the ROI and solve the customers problems faster. Keep adding value is possible at a reasonable cost because the cost of change is kept low.

- Finally, value is the objective. Processes exist to support the creation of value and are trimmed to the minimum necessary.

PO needs to turn, but developers refuse!

Interesting discussion we have this morning in our Agile Improvement meeting: How to maintain a good balance between the need of the PO to change things to maximize value and the need of the developers of certain stability to perform their activities.

In one side is the PO, responsible for maximizing the ROI of the project. The PO takes all his decisions with the goal of maximizing the value the product will deliver, make it more sellable and make it more competitive. 

On the other side, the developers who work hard during an iteration to finish a feature. Timeboxed feature based iterations impose this hectic rhythm. In a short period of time, the development teams delivers a feature that has some business value. During this period, the development team needs to feel this feature has been groomed to a point where is nearly frozen or at least few changes will be necessary. 

So what happens when something has already been finished and the PO judges it's incorrect: it needs to be redone. In a project with a lot of uncertainty, this is entirely possible. Having to change something when already has been completed or have to drop something is demoralizing. The team could feel that the product team does not care for their work or that they don't have any idea of where we are going. Motivation goes down because there's no reward in finishing something.

How to balance these 2 apparent contradicting needs?

With trust: There's one team - product and development go in the same direction. If a decision that implies rework or throwing work, the team needs to know it's for the best. The team trusts the PO on these decisions. And the PO respects the work of the team. 

Atacking the uncertainties as soon as possible: when the team starts with a user story, everybody (product and development) tries to eliminate the uncertainties as soon as possible. Is the prototype Ok?  What will happen if?

Maintaining the communication through all the construction of the story: This conversation continues throughout the development of the story. The ideal is the PO seated with the team full time in an opened space. Of course, this is not always possible, but we should strive to make it as similar as that as possible.

But what happens in distributed teams, where the PO is in a different timezone. Written documentation becomes more important, because it sticks and can be consulted whenever the remote team needs it. Documentation becomes more important because communication decreases. Of course, written documentation is much less effective than oral communication but good teams overcome these situations.