Questioning Scrum

I have been working with Scrum for the past 4 years. I love it. It was my introduction to the Agile world and It is so much better than what I did before. Now I am in a moment where I am questioning some things. I attended a couple of conferences about Kanban (so I am far from expert), but it was enough to plant some seeds of doubt about the things I do with Scrum. Below are some examples

Cross-functionality

- Scrum states that cross functional teams are needed, but the way to enforce it usually fails. Having iterations with stories that correspond to each area of specialization 'suggests' specialists for each story. I have struggled with many teams to be more cross-functional, to pair and to spread the knowledge, but Scrum doesn't give the tools to do that per se. On the other hand, Kanban with its 'minimize work in progress' principle enforces cross-functionality in a much better way as stories are not pulled until one of the current ones is finished. This 'forces' developers to pair or just work on areas of the system where they are not so comfortable.

Attack the risks

- Previous point also applies to tacking the risk first. There's usually the case in a scrum iteration that individuals choose the tasks they are more familiar with or they feel more comfortable with (yes, I do it all the time). Scrum 'sort of' allows this. In a self-organized team, team members decide which task they want to work with. How can we be biased to choose the task where the bottleneck is or the task that is representing the highest risk?. If there is no option other than working in the bottleneck/risky task, we don't have any other choice.

Sprints:

- Having iterations of equal size forces the team to break up the functionality in chunks that fit into a Sprint and also forces closure during the iteration to be able to finish them. A cadence is created as iterations go by and there's usually a sense of urgency because there is a lot of work to finish a story end to end. This is really nice but there are things that I don't like about these sprints. Imagine, we, the team 'commit' to a certain amount of work. When we are too careful with the estimations, all work is finished either 2/3 days before, and we (usually prefer to) relax and wait until the next sprint. What happens if we over-commit? We usually works harder to try to finish (which is not that good) and if we don't finish and the story is rolled, that story stretches in the next sprint in unthinkable ways!!. No scenario seems to be perfect. Personally, I prefer more stability. Committed and motivated people working 8 hours a day to complete the work. Does this mean stories should not be estimated? Well, I think in highly exploratory projects, stories should still be estimated even if using Kanban and the team should strive for closure in the estimated time. But the sprint fixed size sometimes is an artificial constraint that creates certain dysfunctions.

Iteration Plannings:

- I have been in many iteration planning where stories are 'guesstimated' because there is not enough information at that point (in 3 week iterations, that is entirely possible). Teams are encouraged not to pad, but when there's too much uncertainty, team members fall into this. Estimating the story when it's pulled allows in many cases to have more information, to avoid assumptions and makes this meeting faster and more productive. Probably, this is a case of 'Decide as late as possible'. For example, if one story depends on the another and many of the uncertainties will be cleared when then first story is completed, why spending so much time to speculate in estimating the second? (we do this very often in Scrum, don't we?)

- Iteration Plannings tend to be to large. And sometimes boring. Scrum suggests a ratio and the meeting should be timeboxed. If there's uncertainty in some of the stories, this could get worse. Who hasn't been in endless debates that later translate into a short 2 hours research that sheds all the necessary light? I guess this point is very related to the previous. A crisis of faith in estimating things I don't understand.   

Scrum taskboards versus kanban boards

- Kanban boards provide more information that Scrum boards. They allow to visualize flow and discover bottlenecks. Many times when working with Scrum I find my self looking at both the taskboard and the burndown to give me a real sense of the progress. With scattered tasks all over the board, usually I don't understand where we are.

- Be able to spot the bottleneck and provide an impersonal way of pushing the team is great. Constraining work in progress in certain phases forces the whole team to focus on the most critical things that will make the iteration successful. And this is not intuitive. Personally, If I looked at a scrum task board, I usually pick the tasks I am more familiar with even if they are not the highest priority or the ones that minimize the risk in the iteration. I don't think I am different than the majority :-)

Daily Standups:

- Something I read recently (and I liked) is Kanban teams do the standup looking at the board instead of looking at each other. I like that idea. I believe it foster collaboration in a less personalized way. Let me explain: sometimes I think Scrum daily standups put too much pressure on the people that are going slower or are stuck. People is looking at each other and sometimes there could be managers (chicken, but managers). I believe this could cause a dysfunction: lying about the status. Or even worse. Try to finish faster, dropping quality. Focusing on the board switches the focus from a person stuck to a task stuck. It's subtle. I like it.

All in all, I think kanban brings values that would make a team better. Minimize work in progress (and thus reduce the size of the increments), defer decisions until the last reponsible moment (estimate when you have all the needed information), focus on the problems and help resolve them (as opposed to help someone resolve it). I never used Kanban in practice. But I am at a point that with what I read/hear, I can spot moments or situations where we are not doing things in the best way with Scrum. I am not saying these are reasons to justify moving from Scrum to Kanban. And I am not saying these are shortcomings of Scrum. These are just some questions that arise from knowing that there are other ways of doing things. And I believe this questions are helping me improve how we do things now.

Agile is Self-Organization and Self-Discipline

Agile teams are self-organized teams that work in an environment that has certain constraints to achieve certain goals. A self-organized team lacks a central authority. Instead, members self-organize and lead themselves to achieve the goals.

To understand how Self-Organization works, we can start reading about "Complex Adaptive Systems". In complex adaptive systems, the solution emerges. The term "emergence" means that "Rather than being planned or controlled the agents in the system interact in apparently random ways. From all these interactions patterns emerge which informs the behavior of the agents within the system and the behavior of the system itself." Complex Adaptive Systems accurately describe a software development ecosystems. Members in this ecosystem usually take every day decisions that impact the final product and they interact with other members to do that.

Self-Organization doesn’t mean anarchy or sloppiness. On the contrary, Agile self-organized teams are very disciplined. This discipline is the one that is borne because of the high standards of the group and because no one wants to disappoint their peers and not because of rigid orders of a manager.

Self-organization needs to happen within the limits set by the organization and guided by leaders. Managers still need to set the direction, the constraints and the guidelines that the team needs to follow. It is also important to have leaders within the team to coach and spread the good practices. 

Why?

Which are the reasons that justify having this structure instead of a more traditional command and control structure. This is a small list of those:

Self Organized teams take better decisions: The fifth lean principle in the Poppendieck's "Lean Software Development" book is Empower the team. Empowered and educated employees guided by a leader are motivated employees that can take better decisions that their managers. Scientific Management, as stated by Taylor, was designed with another type of employee in mind.. very different from the software engineer. In today, knowledge-based organizations, delegating power down (empowering) has proven to be much more effective because the engineers working at the front line have much better information and because they have the knowledge to take those decisions. 

Individuals in a Self-Organized team are motivated: Motivation comes from participation. Being part of the solution… Something to keep in mind when working with knowledge-based workers is they are highly educated people. Therefore, motivation not only comes from the salary or the position, but also by being able to do something that satisfies them.  

Self-Organization improves trust and commitment: Knowledge-based organizations are totally dependent on the commitment and ideas of their employees. Knowledge is a resource locked in the human mind. As the Novel laureate economist  Friedrich Hayek has argued, "Practically every individual... possesses unique information" that can be put to use only "with his active cooperation". 

Self-Organization enables innovation: Trust and commitment makes people go beyond their call of duty by sharing their knowledge and applying their creativity. The synergy created in the interaction from individuals with this attitudes is one of the key enablers to build innovation. Contrast this to other structures were employees are just told what to do. In this case, what the person can contributed is much more bounded. In Self-Organized groups, innovations thrives.

The best solutions emerge from self-organized teams: last but not least, let’s mention one of the principles in the Agile Manifesto. I completely agree. A self-organized solution contains a mix of different backgrounds on it. There is no comparison between a solution thought by just one person and one reached by a group that went through different options and agreed on the best one.

Managers to Leaders

If members are not told what to do, you probably would wonder what happens with manager. There is a shift in the way management is performed. Managers become leaders in Agile. They don't tell its employees what to do. They fix the goals and set up the constraints to get there. And they lead by example. Lissa Adkins states in her article "The Manager Role's in Agile" that "agile managers coach, inspire, and lead teams more than they measure and manage them."

Wait a minute.. Does this mean managers don’t take decisions anymore? I had this doubt for a lot of time, until I read Highsmith “Agile Project Management” book. The answer is “yes, they still need to take some decisions”. Even more. The group expects the leader to take some decisions. I’ve seen it many times. When the leader is respected, there are a lot of people that even prefers the leader to take the decision for them. But to reach this point, there should be respect. Lots of it.

I just read the paper “Fair Process: Managing in the Knowledge Economy” by Chan Kim and Renee Mauborgne. This paper explains very clearly how people respect and even commit to decisions they have not taken if and only if the process is “fair”. A fair process is transparent and known before the decision is taken and it allows for the people affected to at least expressing their points of views. The process is as important as the outcome. 

Conclusion

Self-Organization is one of the tenets of Agile. It improves trust, motivation, commitment and most of all, it improves the quality of the produced work. Managers and organizations used to old command and control structures need to change their beliefs and their culture. Agile leaders need to understand their new role, which is by no means easier than before.

Can estimations be used to push the team? How?

In all teams, there are guys that are really performant, there are average and there is people that are slower (be it because of skill reasons, personal reasons or whatever). When tasks are estimated in ideal hours using a technique similar to planning poker (where everyone contributes), the result should be an estimate that falls into the average, right? To be more clear, if the task is taken by a fast member, probably it will be finished before the estimated time. If it's taken by an average, it would finish around the estimated time and if it's taken by a slower member, it will take more (probably much more) that what was estimated.

So these estimations should give us an idea of where we are in the Sprint, right? The question is: can we use them to 'push' the team? If they are used to push the team, how can it be done without hurting the sensibility of anyone and without causing people fear of taking the most difficult tasks?

These are some options:

- Asking the individual/s that hold the task why is taking so long in the standup.. This is brutal and I don't see any possible benefits (probably people will be terrified of taking the most difficult tasks very rapidly)
- Asking the whole team why certain tasks are taking longer. Much better cause there is depersonalization of the problem.
- Try to make the problem more visible. Using information radiators, everyone should be able to see the problem and pitch in to try to solve it.
- Minimize work in progress. This is another, more effective way, to make the problem visible. If no new tasks can be pulled because there is something stuck, the group as a whole will try to un-stuck it. Completely depersonalized and very effective. Wait... I am not using estimations this way :S

Ultimately, Agile relies on a group of motivated and committed individuals. If the speed at which the group is going is not enough to reach the objectives, it is a problem of all the team: managers, product owners, developers, etc. The group as a whole needs to resolve this problem. They need to figure out how to go faster. If the group shares the responsibility this way, they will find the best way to overcome it. However, there is a basic requisite that the group must hold: There should exist trust in the group. It should exist the trust that allows you to tell the group that you are stuck and need help without been judged. There should exist the trust that allows to treat the problems as problems of the team.

Comments???

Agile is collaboration

I categorized the collaboration component as Human because I believe it deals with human characteristics. Funny thing.. What do I know about humans? :-) Seriously.. not too much. I went to a high school with a focus on science. I studied Computer Science and I work all my life with developers!!

When I was writing it, I started doubting about having 2 different posts: one for customer collaboration and one for collaboration within the team. I think the Agile Manifesto talks only about customer collaboration, but I think the Agile community shares the feeling that team collaboration is equally important to success and makes for a better workplace. Do you agree?

What is collaboration?

To collaborate is to work jointly towards a shared objective, to help each other, to have good intentions, to share all the information and ultimately to do what's best for the team as a whole.  

Endless projects have failed in the past because of lack of user involvement. We used to believe that it was possible for the user to order what he needed and collect the product once it was finished. So, the user was involved at the beginning, defining the requirements and in the end, when the product was already built. Of course, in lots of situations, the user may have not liked what he was receiving but making changes at that point was very expensive. Therefore, the user was forced to specify correct and complete requirement definitions at the beginning of the project. Analysts and developers were 'forced' to understand these specifications which, as we all know are very difficult to do.

The Agile approach is very different as it tries to incorporate the customers throughout the creation of the product. The Agile Manifesto says "Business people and developers must work together daily through the project". So the customer is of course involved defining the requirements, but they also collaborate with the developers in their analysis and provide feedback with each increment of software. There's close collaboration between developers and the customer during the whole project. Customers and developers work together towards a share objective.

Agile collaboration does not end with customer collaboration. Collaboration needs to happen inside the team as well. Agile team members help each other, share all the information and strive for team success.  Agile builds on a team, not on isolated individuals. In a collaborative software development team, members contribute their efforts to build something bigger than themselves.

How is collaboration achieved?

The first requisite to enable customer collaboration is have a contract that allows the customers to be involved in the development process. For the business people, it may be difficult to understand that they will need to 'lose' their time working with developers to build the system. Besides this, having a business person involved in the project will probably generate changes as the project progresses. If there's a contract signed for a defined scope, it may be difficult to apply these changes as they may imply changing the conditions of the agreement. So the contract should be one that welcomes these changes.

Once changes are 'financially' feasible, it should be technically feasible. Agile relies on incremental development to be able to incorporate the customer in the creation process. That is build a bit, show it to the customer, get feedback and keep building while incorporating this feedback. This collaboration that happens in the construction of each feature helps build what the customer is expecting. Building software incrementally is not enough. For this to be sustainable, the cost of change of each new increment should not increase exponentially. If it did, those changes suggested by the customer wouldn't be welcomed at all.

How is collaboration achieved inside the same organization? Well, it may be very difficult to achieve as it deals with human nature. In any organization, there may be conflicts of interests as the interests of the organization are not exactly the same as the interests of its individuals. Cockburn says there are many "games" in place. The organization is playing its game where the project it's just another move. The individuals that work in the organization play another game which is their career. Personal interests may undermine the sucess of a project, but there are things an organization can do to improve collaboration:

Definition of Success or Failure

Either the team succeeds or the team fails. With this definition, there is no place for individualistic behaviors. What's best for the team is what's best for everyone. Individualistic behaviors don't help the team succeed. This definition of success encourages everyone to act in a collaborative way.

Productivity Rewards in the Organization

This point is very related to the previous one, but I think deserves to be mentioned separately. How does an organization rewards its employees? If rewards are made on an individual basis, employees will try to demonstrate their work and they will try to excel over everyone else. This type of reward encourages employees in selfish behaviors and diminish collaboration. I think this point is very interesting as I've seen many organizations doing Agile that have incompatible rewards schemes.

One question that I often asked myself is if a medium that favors collaboration could be unfair with high performers. Someone that works hard and also helps his/her team-mates a lot could feel his/her work is disguised under this scheme. Someone that doesn't work as hard could be 'hidden' under team goals. Probably the only way for managers to verify who works hard and who doesn't, who is talented (and ultimately who deserves a raise!) is to live in the day to day of the team. High performers Agile developers are those who help the team succeed and if you live the day to day with the team, they are easily noticeable. 

Attack dysfunctions

Patrick Lencioni describes in his book "The Five Dysfunctions of the Team" the dysfunctions that a team must overcome to be able to work well as a group.

Image taken from: http://www.thepracticeofleadership.net/2007/03/21/book-review-the-five-dysfunctions-of-a-team/

The first dysfunction is Absence of trust. Lencioni says 'this stems from their unwillingness to be vulnerable within the group'. Well.. It's hard for anyone to expose its vulnerabilities. Even more when your seniority is high. However, if team members don't trust each other, they don't ask for help and they hide information. Ultimately, they do what's best for them instead of doing what's best for the group. On the other side, teams where members trust each other are much more healthy. They ask for help, they help each other and they provide all the necessary information to anyone.

Failure to build trust generates Fear of conflict. If members don't trust each other, they don't engage in constructive arguments. Teams that cannot engage in arguments where everyone expresses their honest opinions lose much of the team's power.

Failure to engage in constructive arguments becomes the greatest cause of members not committing to the decisions taken. If members were not able to express their opinions freely and discuss them with their peers, they don't buy into what has been decided.

Members that don't commit are members that are not accountable. How can you ask a member that hasn't committed to a plan to follow it, right?

Finally, non accountability creates an environment where the five dysfunction can thrive: Inatention to results.

This book is pretty amazing. Once you read it, you begin thinking about those attitudes that were hard to understand when you just started working. For example, replying with a copy to the boss. Or you start thinking about those guys you used to admire: I used to say "this guy, I don't know what he does, but he is so politically smart!!". That sucks men... I love other attitudes now. Those that encourage transparency and bonding in the team. For example, saying in the standup 'Yes, I screw with this decision.. I apologize' or 'I really tried to make it work, but coudn't. I need help' and I also like the guys that step and say 'I will help you'. This makes a team.

Conclusion

I like seating, thinking and trying to order my ideas to write a post. I must confess I was not making the difference between customer collaboration and team collaboration. I was putting everything in the same bag. Now I think making the difference makes sense. I think the Agile Manifesto deals with customer collaboration, but I see no explicit mention to collaboration within the team in it.

Customer collaboration is of utter importance to the success of the project. A customer, someone that knows and understand what is needed, throughout the project increases exponentially the chances of success. Collaboration needs to be enabled, both at a contractual level and at a technical level.

Team members that collaborate with each other make the other half. There is nothing like a health environment for people to reach their highest potential.   

There's also a side effect benefit. Collaboration over contracts makes the process lighter and thus centered in value. I believe there are some processes or quality standards that suggest contracts to supply lack of trust. If you don't trust the other part, you want a signature to be assured that the other part is committed to do something. In contrast, if you trust the other part, you just need his acknowledgement of understanding.     

The [f..] last story of the iteration

I hate the last story in a Scrum iteration. If we are about to finish it, we sacrifice quality to fit it into the iteration. If it's not completed and carried over, either it's finished in the first day (best scenario, but stays the whole iteration there cluttering the task board...) or expands in a crazy way (2 hours in the last day of the sprint easily become 16 hours in the next iteration!!). WTF.

The Agile Architecture

There are some big graphs, so I published the google doc as well (it looks better): http://docs.google.com/View?id=dhpj7455_67gkwk3bfr

When I started with my first Agile project, I just knew I liked it. I felt better working this way, I felt that we delivered better results. I was thrilled by the quality of the code, I felt good with my team mates.. But they were just feelings. I couldn't explain why. With the excuse of having to complete my MsC thesis, I decided to keep investigating and learning about Agile, . I went to my first Agile course (hey Matt, it was cold in that room!), kept working in Agile projects, read some books, blogs, forums, went to some Agile conferences, became a Certified Scrum Master (wow!) and started having Agile friends :-).


So after 4 years, what is Agile? Well, what I can tell is it is not a methodology (Haven't you read at least a post that starts with "The Agile methodology..."?). The most accurate definition I have is Agile is a set of values and principles listed in the Agile Manifesto... But probably, this manifesto has too much knowledge condensed in a few words. In this post I want to define the Agile architecture (yes, this is upfront architecture!). What I mean by architecture are what I considered the most important components in an Agile development team (I am a developer, so it is from a developer's vision).

This is what I came up with:

First, I defined what I consider the most important objective of Agile: being able to deliver value sustainably, with the required quality and in a changing environment. I did a post recently in which I list why Agile provides better value.

Then I tried to list the most important components and after having them, I tried to categorize them. I think there are 3 categories:

1) Components related to the technical field. I am a developer and so I believe these components are really important in any software development team. Inside this category, I put 3 main components.

    a. Simple Emergent Design: being able to evolve the design in every iteration is crucial to being able to deliver a continuous flow of value. This component is a consequence of doing iterative and incremental development, and relies on having automated tests and keeping the code clean via opportunistic refactoring. It also relies on having the latest code base at all times (continuous integration).

    b. Technical Excellence: this includes opportunistic refactoring, clean code and ruthless automated tests. This component is big as its importance to successful agile teams as it supports doing evolutionary design, it allows to adapt to changes and it supports having a process that relies on having a lot of information on the code itself. 

    c. Continuous Integration: This is a small component that supports other components such as technical excellence and emergent design.

2) Components related with the process in place.  

    a. Light Process: Agile needs a process that allows to "welcome change, even late...", a light process that reacts and adapts to change. The ceremony and the artifacts should be the minimum that allows to coordinate the project at hand (otherwise is hard to turn). The process should encourage introspection and optimization based on it (kaizen). To accomplish this, Agile relies on a self organized but very disciplined team and the software engineering practices included in the Technical Excellence component.

    b. Incremental and Iterative Development:  I am not sure if this component should be part of the light process (probably yes). Anyway, I wanted to stress the importance of having a process where development is performed iteratively and incrementally. 

    c. Self-organized & Self-disciplined: Agile relies on a team of committed and passionate software engineers. Maintain technical excellence relies on a lot of discipline. Self-organize successfully relies on human components: good communication among all team members and the will to collaborate.

    d. Transparency: The process should encourage transparency. Transparency improves collaboration and allows to reduce the number of needed artifacts.

3) Human components

    a. Collaboration: All team members, whether in the product or development side, collaborate with each other. Success is measured for the whole team that participates in the project, not by sub-team or individually. If there is good collaboration, there's no need for artifacts that just have the purpose of make someone responsible and the process can be lighter. 

    b. Communication: The flow of information among all members should be taken to the maximum (possible) as members with more and better information translates into improvements in the results.

These are the areas where I would map the Agile values:

And the Agile principles:

Final Thoughts

I am not sure if this makes sense or not. I know making this exercise allowed me to visualize what components an Agile development team should have. I've been following some threads where they discuss what is missing from Scrum and started wondering: Is there a methodology that follows all the Agile principles? Is it straightforward to map all what is needed to make an Agile Development team from the Manifesto? (as I said before, the Manifesto seems to contain the conclusions of many years of experience of these guys). I am also interested in Process Improvements Frameworks and so I was wondering whether it would make sense to define (as CMMi does) the Agile process areas (what I called components here) and the suggested practices in each area? I am not sure about this..When you are in the Shu level, you need to follow a methodology to start. Perhaps something like this would allow to choose the methodology or know in advance what needs to be addition to the methodology to "comply" with the Agile values and principles.

Agile is Iterative and Incremental Development

Definitions

Agile is iterative and incremental development. Iterative and incremental development was borne in response to the weakness of the Waterfall model.

In Waterfall, the product is developed sequentially, delivering all the product at the end of the last phase:

from wikipedia: http://en.wikipedia.org/wiki/Iterative_and_incremental_development

The problems with this approach are:

- Specifying all the requirements at the beginning is very difficult. When the customer is able to see a working product, is very expensive to make changes.

- It's very risky. Developers starts facing the real problems after 2 entire phases, which could be in the middle of the project. A lot of money has already been expended at this moment. 

The Agile approach to overcome these problems is to deliver the software in increments. In each increment, activities from all these phases are included.

from wikipedia: http://en.wikipedia.org/wiki/Iterative_and_incremental_development

It is important to understand the different between "incremental" and "iterative": 

In incremental development, work is break up into smaller pieces which are scheduled, developed and integrated when completed. An increment is an increment in business functionality. In other words, an increment in Agile delivers something new to the customer which s/he can see and experiment with. To be able to accomplish this, work is break up based on functionality. A feature is developed in its entirety, from the UI to the back end in an iteration. 

Iterative development means the product is built in iterations, where each iteration expands the product until the project is completed. As the process is also incremental, each iteration includes parts of all phases (i.e. in each iteration, there is some analysis, some design, some code and some testing). 

When researching to write this post, I came accross this article (http://www.stsc.hill.af.mil/crosstalk/2008/05/0805Cockburn.html) written by Cockburn. Something that caught my attention about this article is he states that if after an iteration is completed and the feedback from the customer is received, there isn't time to incorporate these changes, then the process isn't really iterative. There should be scheduled time to rework, to apply the changes the customer requested, to revise the performance, to redesign or the "iterative" part disappears and we are following into the same Waterfall mistakes. To be honest, I don't understand very well why it would stop being iterative. The iterations are still there and would expand the product in each iteration but not where the customer is expecting. To be even more honest, I always saw them as one entire concept. If the delivery is incremental, is there any other way to do it other than iteratively (recursively?).

Pros & Cons

Basically the advantages are the disadvantages of the waterfall model:

+ The development team and the product team learn from what has already been constructed. The development teams face the difficulties early in the project life. The product team starts looking at the product early as well and starts materializing their ideas and suggesting the necessary changes needed to obtain the best possible product. 

+ Value can be delivered early. Doing incremental development is the basis for being able to delivered a partial release.

+ Risk is reduced. Developers "attack" the riskiest features early in the project's life.

+ It's the approach to take when exploring, when working on a not known domain. It's the way scientists work. They make an hypothesis and try to prove it via experiments. Spending a lot of time architecting and designing does not make a lot of sense in these types of projects. 

What are the cons then?

- The only disadvantage I can think of is it's difficult. Squishing all phases into a short iteration presents many difficulties to all members of the team. Developers need to accommodate the feature to the existing design. Testers need to start thinking about the test cases in the air, when the feature doesn't exist yet. All members of the team need to work concurrently on the same feature, which requires the team to coordinate the work very well. Add to this that development needs to be sustainable and therefore completing a unit of work means not only delivering the functionality. It means delivering it with the best possible design, with clean and refactored code and an automated suite that can regress this functionality in the future.

Architecting and designing in an iteration....?

This is perhaps the most controversial topic in incremental development. Detractors of Agile usually ask where (the heck) does the architecture and design fit into an iterative and incremental approach!!!

At the beginning of the project, there is a period where the vision is defined and the basic requirements are gathered. After that, there should be time to define the most important aspects of the architecture. The length of this time would vary depending on many factors. If the  project is complex, but there is a lot of certainty it makes sense that this period is larger. As certainty decreases, spend time at the beginning makes less sense as conditions will probably change. Remember Agile needs to maintain the curve cost low. Therefore, decisions that later would imply big costs should be taken at the beginning. Highsmith calls this making a "balance between anticipation and adaptation". Anticipation is up front design. Adaptation are the changes needed to react to different conditions in the course of the project.

When iterations start, for each new feature that is constructed, there is design. The design consists in accommodating the new feature in the best possible way. I don't know how you feel, but I feel design is performed with better information. It is performed based on tangible components. Personally, I do the design understanding the problem better.  Of course the design for the existing features may change to make the new design that contains all the features a better design. Don't forget that Agile is also about being able to deliver value sustainably. If design deteriorates with each new feature added, technical debt starts rising and it becomes more costly to add new features in the future. Another important point is Design is made with the YAGNI principle in mind: "Do the simplest thing that could possibly work". Let the abstractions be borne as they are needed. Don't make decisions based on assumptions about the possible necessity in the future. Ah, one last important point: "Simple is not simplistic". The design of each feature ends up with the simplest but cleanest and most elegant way you could possibly think of (I say you could possibly think of because making good designs is not something that you'll do the first day you start developing - that doesn't prevent you from following these principles, as I do :-))

Being able to change the design in each iteration relies on other technical practices such as refactoring, continuous integration and automated tests. Without them, making changes would be very risky. Refactoring in each iteration allows to have the cleanest code base each time a new change is required. I believe anyone that has developed knows there's always a couple of options to do something: the clean way and the hack. The hack is the quick and dirty option, useful in the short run and painful in the long run. The cleanest way includes refactoring and of course more work. Automated tests provide a harness that allows to apply the changes in a safer way. It would be impossible to make continuous design changes if it weren't for the automated tests that provide certain security that the existing features keep working after the changes. 

To Timebox or not to Timebox

I never thought about it until I read Johanna Rothman blog... Every time you have a task to perform, you have 2 options to tackle it. If you understand very well the scope of the task and is something that can be done in a reasonable amount of time, you just go for it. You work with the objective of finishing the task. What is the problem with this approach? If the objective is blurred or the task is very big, there is no sense of progress and we can easily fall into extending the task ad infinitum. Timeboxes bring the reality of date constraints to the task at hand. If you plan to tackle the task with timeboxes, you allocate some time, estimate what you would be able to complete in that period of time and then review the progress and take decisions based on it.

Timeboxes bring these benefits software development:

- "Timeboxing forces closure, it forces the team to deliver something concrete" (Highsmith). When exploring, it is very difficult to scope a feature. Teams are discovering and therefore, times can expand indefinitely. This contrasts with the date constraints that projects always have. Timeboxes force a date constrained exploration and it forces the development team to focus on delivering something.

- Timeboxes are short term adventures, marked by decision points at the beginning and at the end. These decisions points allow the business to take decisions. The word "short" is important here. Timeboxes are short while scope-boxes tend to be larger. 

- Timeboxes create a cadence. The team enters a cycle that repeats every 2/3 weeks. After that period, the team review how it performed, adapts and recommits for the next iteration.

So what will you do, timebox or not timebox? Scrum and XP use timeboxes while Kanban doesn't enforce them. To make the decision, bear in mind what timebox provides. I have seen a few examples where the teams were struggling to work in timeboxed iterations when it just didn't make sense to them (e.g. production environments working with Scrum where they were getting requirements all the time and therefore they couldn't freeze a chunk of work for X weeks) or the exploratory factor was very low (basically the user story always takes a certain amount of time). In these cases, timeboxing doesn't add anything and I would just opt for a scopebox approach. On the other side, if the project is highly exploratory, the timebox iteration allows the team to focus and close a piece of functionality. It forces them to scope that piece of work and not expand to provide features not required at this time. Whether you timebox or not doesn't make you more or less Agile. At least, I don't see any reference in the Agile Manifesto to timeboxing.

Conclusions

"Iterative, feature-based delivery is the cornerstone to early and often delivery of value" (Highsmith). Being able to produce a partial product allows to have a common ground for the product and the development team to discuss and keep delivering value. During an iteration, a lot of new knowledge is gained which is used by business people and developers to make better decisions in the future. The technical risk of the project is tackled as early as possible. Of course iterations don't start immediately. All Agile project start with the basic architecture and a bunch of important decisions. When iterations start, design is done incrementally. Being able to design incrementally relies on doing other engineering practices, such as refactoring and automated tests.    

Update 2/23: I just made the click with the meaning of an iteration. The key is that an iteration is fixed length. This is in contrast with a pull system like Kanban that pulls the work. So Scrum/XP are incremental and iterative and Kanban is just incremental. I am thinking that perhaps the title of the post should be changed to "Agile is incremental development" as the agile manifesto doesn't say anything about being iterative. (At least, this is my interpretation of this principle "Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.")