The Lean Tribe

The Agile Manifesto triggered the growth of "the so called until that moment lightweight methodologies". Scrum and XP became very popular and the tribes associated to them grew with them. However, not long after the Agile Manifesto was signed a new actor entered the scene with outreagous strength. His friends use to call him "Lean" :-)

So what is Lean? Lean has its roots in the Toyota Production System developed by Taichi Ohno. Its core idea is to maximize customer value while minimizing waste.

Lean was brought into the Agile world by Mary and Tom Poppendieck who in their book "Lean Software Development: An Agile toolkit" translated 7 lean principles used in manufacturing to software development language.

Although Lean and Agile share many of the same values and principles as they both are people centric and empower people and they both try to adapt/improve the process continually, they also have some differences as well. The main one in my opinion is its main objective: Agile was developed mostly by software developers and therefore its main objetive is "to uncover better ways of developing software". Lean has in its essence a broader objective (and a broader target) as it seeks to improve the flow of value and this could only happen at an organizational level.

The Agile world was greatly modified by the entrance of Lean. So much that probably now it is an Agile/Lean world.

So who are the referents of this new Lean tribe. Where do they get together. Do they breed with Agile folks? :D I know I may be missing many of the referents of the manufacture industries but these are the ones I know:

- Mary and Tom Poppendieck
- Eli Goldratt (RIP)
- David Anderson
- Karl Scotland
- Alan Shalloway

These folks get together in conferences like "Lean Software and Systems" (in its website, you'll be able to find all the speakers that attended in 2011)

In the last few years, the Lean tribe grew incredibly fast. Particularly, a lean tool called Kanban has gained a great deal of attention and is becoming the new "Scrum" of the moment (or the new star of the moment).

How is the relationship between Agile and Lean? Well, I believe that although there have been some fights (or discussions should I say) because sometimes tribes defend their interests, the Agile community has a lot to learn from the Lean community. Although in the Agile world, Lean is new, it really is much older than Agile.

Agile Tribes

The people that got together 10 years ago in Salt Lake City had different backgrounds, specialties, positions, etc.There were some folks already working on their own methodologies, such as Beck with XP, Schwaber and Sutherland with Scrum and Cockburn with Crystal. Some of them were more technical, specializing in OO, programming and testing while some others were managers or people more interested in the methodology aspects of software development. This is more or less what I believe were the backgrounds that depict the folks that were there:

Now I wonder how these 'tribes' (or communities or subfields of interests - I wonder what is the best name) develop over the years. How did they grow or mutated? Which are the new tribes that make up the Agile world?. Who are the new referents of these tribes? Will you help me create this map?
How do I define what a tribe is? Let's list some items that define a tribe:

• They share the same values and principles (which should enter the big Agile umbrella)
• They work with a certain methodology or method
• They have some referents
• They get together in one of more congresses (this could be a consequence of the previous)

Am I missing something to define a tribe? Hummm, yeah for sure. Will see it as we go (iteratively). For the moment, I will research for each tribe the following items:

• Reason to be (values/principles they share, objectives)
• History
• Referents
• Present/Future?

Now I just need to select the most important tribes in today's Agile map. I will start with these ones:

• Kanbaners
• Software Craftsmen
• Lean-startupers
• Scrumers
• XPs

I will focus on the first 3 groups, as the last 2 exist since the beggining of Agile.
I'd love to receive suggestions/corrections, etc.

log4jdbc: Una herramienta muy útil en desarrollo

Me estaba preguntando si existía una herramienta que permita ver los queries que se ejecutan contra la DB cuando se usa SimpleJdbcTemplate en Spring. Y.. existe. Se llama log4jdbc y la instalación es sencillisima.
Una vez instalado, en el log de la aplicación se podrán ver los queries y también el tiempo de ejecución.

(       45728) [http-8400-3         ] INFO  select DISTINCT location_code from PURCHASE_LINES where number = 'XXX'
                              | at net.sf.log4jdbc.Slf4jSpyLogDelegator(Slf4jSpyLogDelegator.java:223) [2011-05-23 21:32:21,341]
(       46995) [http-8400-3         ] INFO  select DISTINCT codes from PURCHASE_LINES where po_number = 'XXX'
 {executed in 1267 msec}      | at net.sf.log4jdbc.Slf4jSpyLogDelegator(Slf4jSpyLogDelegator.java:322) [2011-05-23 21:32:22,608]
(       46997) [http-8400-3         ] INFO  5. ResultSet.new ResultSet returned                                                                                       | at net.sf.log4jdbc.Slf4jSpyLogDelegator(Slf4jSpyLogDelegator.java:158) [2011-05-23 21:32:22,610]
(       46998) [http-8400-3         ] INFO  5. PreparedStatement.executeQuery() returned net.sf.log4jdbc.ResultSetSpy@159d796                                         | at net.sf.log4jdbc.Slf4jSpyLogDelegator(Slf4jSpyLogDelegator.java:158) [2011-05-23 21:32:22,611]
(       46999) [http-8400-3         ] INFO  5. ResultSet.next() returned true                                                                                         | at net.sf.log4jdbc.Slf4jSpyLogDelegator(Slf4jSpyLogDelegator.java:158) [2011-05-23 21:32:22,612]
(       47000) [http-8400-3         ] INFO  5. ResultSet.getMetaData() returned oracle.jdbc.driver.OracleResultSetMetaData@19e80b1                                    | at net.sf.log4jdbc.Slf4jSpyLogDelegator(Slf4jSpyLogDelegator.java:158) [2011-05-23 21:32:22,613]
(       47001) [http-8400-3         ] INFO  5. ResultSet.getString(1) returned 10176 KKKKKK                                                                      | at net.sf.log4jdbc.Slf4jSpyLogDelegator(Slf4jSpyLogDelegator.java:158) [2011-05-23 21:32:22,614]
(       47002) [http-8400-3         ] INFO  5. ResultSet.next() returned true                                                                                         | at net.sf.log4jdbc.Slf4jSpyLogDelegator(Slf4jSpyLogDelegator.java:158) [2011-05-23 21:32:22,615]
(       47003) [http-8400-3         ] INFO  5. ResultSet.getMetaData() returned oracle.jdbc.driver.OracleResultSetMetaData@113126e                                    | at net.sf.log4jdbc.Slf4jSpyLogDelegator(Slf4jSpyLogDelegator.java:158) [2011-05-23 21:32:22,616]
(       47004) [http-8400-3         ] INFO  5. ResultSet.getString(1) returned SSSSSSS                                                                           | at net.sf.log4jdbc.Slf4jSpyLogDelegator(Slf4jSpyLogDelegator.java:158) [2011-05-23 21:32:22,617]
(       47005) [http-8400-3         ] INFO  5. ResultSet.next() returned true                                                                                         | at net.sf.log4jdbc.Slf4jSpyLogDelegator(Slf4jSpyLogDelegator.java:158) [2011-05-23 21:32:22,618]


Pretty cool, ah?

Inyectando un Spring Service con PowerMock + Mockito

Ayer a la tarde estaba intentando hacer un test para nuestra aplicación que usa Spring en el backend. Una de las cosas que me di cuenta que cambiaron desde la ultima vez que use Spring es que todos los beans son inyectados automagicamente con la anotación @Autowire, por lo que no es necesario ni tener un constructor ni un setter para inyectar una dependencia. La pregunta ahora es como hago para testear estos beans? (para inyectar mocks en estos beans). Dps. de investigar un rato encuentro la libreria PowerMock que permite hacer cosas locas, como stubbear un metodo estático u obtener/setear el estado interno de un bean (rompiendo asi la encapsulación). Por ejemplo, la sintaxis para setear una variable privada es la siguiente:
        

UserRepository userRepositoryMock = mock(UserRepository.class); // declaro el mock, usando mockito

when(userRepositoryMock.safeGetLoggedUser()).thenReturn(adminUser); // seteo las expectativas
Whitebox.setInternalState(invoiceRepository, "userRepository", userRepositoryMock); // inyecto el mock, usando Powermock

Cual es el problema con el que me tope? Spring utiliza proxies para manejar las transacciones via AOP y estos proxies implementan la interfase del servicio, que no contiene los setters para sus campos. Osea que si quisieramos hacer lo mismo para un servicio (en vez de un repositorio como en el ejemplo),

Whitebox.setInternalState(scannedFileService, "userRepository", userRepositoryMock); // inyecto el mock, usando Powermock

PowerMock se quejaría (porque estamos queriendo llamar un método en una clase que no tiene ese metodo). La exception que arroja es la siguiente:

Caused by: java.lang.RuntimeException: You want me to set value to this field: 'userRepository' on this class: 'Object' but this field is not declared withing hierarchy of this class!
    at org.mockito.internal.util.reflection.Whitebox.getFieldFromHierarchy(Whitebox.java:40)
    at org.mockito.internal.util.reflection.Whitebox.setInternalState(Whitebox.java:25)
    ... 31 more

Esto se podría arreglar agregando el setter a la interfase del servicio, pero la verdad es q no me parecia limpio tener q modificar la interfase publica solamente a efectos de testear. Por suerte, existe una solución. Investigando un poco mas, encontre este foro que indica como hacer con Spring para obtener el objeto 'aconsejado' (advised) desde el proxy.

Advised advised = (Advised) scannedFileService;

ScannedFileService target = (ScannedFileService)advised.getTargetSource().getTarget();

Whitebox.setInternalState(target, "userRepository", userRepositoryMock);

et voila! Una vez obtenido el objeto aconsejado, powermock se encarga del resto. Espero sus comentarios (les parece un overkill? Agregarian los setters directamente? etc.)

Agrego dependencias de powermock en maven para aquellos que son lentos como yo y les cuesta encontrar las dependencias :-).

       
                org.powermock.modules
                powermock-module-junit4
                1.3.9
                test
       
       
                org.powermock.api
                powermock-api-mockito
                1.3.9
                test
       
 

Are Level 5 CMMI Process Areas compatible with Agile?

Introduction

After a quantitatively manage organization, the next objective is an optimizing quantitatively managed organization. An organization that is constantly improving. CMMI level 5 process areas deal with this. Improvement, change, should be done (otherwise, the organization is left behind by others) but it should be managed with care, not to overwhelm the organization.

The first process area is Organizational Innovation and Deployment (OID). Its goals and practices represent a measured way of selecting and deploying improvements. The second process area is called Causal Analysis and Resolution (CAR) and aims at resolving the root causes of defects for ever.

The specific goals and practices for OID are:

SG 1 Select Improvements

SP 1.1 Collect and Analyze Improvement Proposals

SP 1.2 Identify and Analyze Innovations

SP 1.3 Pilot Improvements

SP 1.4 Select Improvements for Deployment

SG 2 Deploy Improvements

SP 2.1 Manage the Deployment

SP 2.3 Plan the Deployment

SP 2.2 Measure Improvement Effects

and for CAR are:

SG 1 Determine Causes of Defects

SP 1.1 Select Defect Data for Analysis

SP 1.2 Analyze Causes

SG 2 Address Causes of Defects

SP 2.1 Implement the Action Proposals

SP 2.2 Evaluate the Effect of Changes

SP 2.3 Record Data

Observations

Both of these processes are designed to work in a quantitatively managed organization. CMMI states that if it isn’t quantitatively managed, it would work, but it wouldn’t be as effective. I wondered when I started reading the book if optimization wasn’t left for too late in CMMI. In one side, how would you optimize if you can’t measure? On the other, my experience with Agile is that optimization comes from the very first moments, at the standup and in the retrospective meetings where opportunities for improvement are detected. Of course all these optimizations are really subjective, as the improvement they provide can’t really be measured.

Another point that I wanted to make is that CMMI mentions that everyone should be empowered to propose optimizations. This goes very much in line with the lean approach to optimization.

Conclusion

I didn’t see any point of incompatibility in these areas. Having a framework in the organization to do Kaizen is very important for the organization as it allows to optimize with the organization’s objectives (i.e. with everyone aligned) , it allows to do it in an orderly fashion and it allows to really know what is been optimized and how much.

Are Level 4 CMMI Process Areas compatible with Agile?

Introduction

I merged the process areas in level 4, because they are really inter twinned and probably don’t make sense to do one and not the other. The first one, the Organizational Process Performance aims at “establish and maintain a quantitative understanding of the performance of the organization’s set of standard processes”. Based on these baselines, the Quantitative Project Management process area sets the objectives and “quantitatively manage the project’s defined process to achieve the project’s established quality and process-performance objectives.

The Organizational Process Performance goals and practices are listed below:

SG 1 Establish Performance Baselines and Models

SP 1.1 Select Processes

SP 1.2 Establish Process-Performance Measures

SP 1.3 Establish Process-Performance Baselines

SP 1.5 Establish Quality and Process-Performance Objectives

SP 1.4 Establish Process-Performance Models

The Quantitative Project Management goals and practices are listed below:

SG 1 Quantitatively Manage the Project

SP 1.1Establish the Project’s Objectives

SP 1.2 Compose the Defined Process

SP 1.3 Select the Subprocesses that Will Be Statistically Managed

SP 1.4 Manage Project Performance

SG 2 Statistically Manage Subprocess Performance

SP 2.1 Select Measures and Analytic Techniques

SP 2.2 Apply Statistical Methods to Understand Variation

SP 2.3 Monitor Performance of the Selected Subprocesses

SP 2.4 Record Statistical Management Data

Metrics? What metrics?

In CMMI Level 4, the organization reaches a level where the existing processes can be quantitatively measured. Objective objectives can be set based on these measures. Being able to measure is vital to being able to manage and optimize. Jurgen published a post recently that states “You cannot manage what you don’t measure”.

A very important point to keep in mind when selecting what to measure is that these metrics need to be compatible with the Agile values and practices. Wrongly selected metrics may create dysfunctional behaviours, detrimental to the organization’s objectives. Dave Nicolette has a very nice presentation regarding Agile metrics. Some of the metrics he discusses are “Running Tested Features”, “Hard Financial Value” and “Earned Business Value”. I wonder, with these metrics, how to handle at the organizational level the context differences among the projects. I mean, each project is different, runs under different circumstances, will deliver business value in a different way, etc. All these factors need to be considered to draw conclusions from the metrics.

These metrics need to provide the picture needed by upper management to take decisions and manage risk appropriately. Being able to visualize the processes used, detect and resolve problems is of utmost importance in mature organizations.

Can everything be measured?

There’s a nice quote by Albert Einstein in Dave’s presentation that says “Not everything that counts can be counted, and not everything that can be counted counts.”. Is there place for subjectivity in a mature organization? How and where would it work? Agile methods stimulate optimization since the very beginning and this is performed, in most cases, subjectively. Teams introspect on the previous iteration, try to determine how they feel and propose ideas that would optimize the process. At upper levels, it would be impossible to optimize this way. Managers need to see the broad view and the only way seems to be through measures.

Keep it light!

I must confess that after reading both process areas in level 4, I ended up a bit scared. It really sounds difficult and requirements seem pretty heavy. Agile processes are characterized by their minimum overhead. The machinery required to measure and optimize processes shouldn’t increase this overhead significantly. For this, measures should be taken in the most automatic way, only what is needed should be measured and when it’s not needed anymore it should be dropped. There should be constant improvement in the improvement processes.

Conclusion

Being able to measure, to understanding where it is standing allows an organization to set improvement objectives. These measures need to be align with the Agile philosophy. The improvement machinery should be kept light.

Is the Decision Analysis and Resolution Process Area compatible with Agile?

Introduction

The Decision Analysis and Resolution process area, a process area at level 3, aims at “formal evaluation process that evaluates identified alternatives against established criteria.”

Below are the specific goals and practices:

SG 1 Evaluate Alternatives

SP 1.1 Establish Guidelines for Decision Analysis

SP 1.2 Establish Evaluation Criteria

SP 1.3 Identify Alternative Solutions

SP 1.4 Select Evaluation Methods

SP 1.5 Evaluate Alternatives

            SP 1.6 Select Solutions

Inclusive and Fair


The process should be inclusive and fair.


Inclusive: All stakeholders should be involved in the decision. With this, I am not saying that all stakeholders need to agree on the solution. Probably this is impossible or too costly. I am just saying that everyone should be able to participate in the decision process, express their opinions, being heard. A good facilitator will increase the chances of everyone being heard and of taking a good decision. Being included in the decision process provides great encouragement to follow through the selected paths. Collaborative decision making is of utmost importance in Agile.

Fair: the process should be fair. Every person in the team should feel that it was not an arbitrary decision imposed. The decision was taken using a process that they understand and consider fair. Of course, not everyone will agree, even if a very fair process is used. However, even not agreeing, if they consider that the process was fair, they will accept. On the contrary, if the feeling is the decision was imposed through a not so transparent process, the decision taken will be very difficult to accept. An excellent paper regarding this topic can be found here.

Conclusion

I believe it would be very beneficial to have a defined formal process that can be used to take difficult decisions. To be more Agile, the process should be inclusive and fair (which are really very inter twinned)