Is Scrum (and probably Agile) Dead?

Recently, I came across an interesting article: Scrum is dead: breaking down the new open development method. I usually do have big problem with such big declarative statements like 'Scrum is dead'. Actually, it's pretty easy to refute the whole rationale by bringing about one counter example!

But, to be frank, the article is good, with the author trying to put some basic principles for the profession of software development; principles that I fully agree with and promote. On the other hand, I strongly disagree with some very fundamental ideas in the article:

Agile is Not a Methodology

First, I find the author builds upon a fundamental idea that "Agile is a methodology". Well, this is incorrect. Also, the article conveys an idea that Agile and Scrum are two sides of the same coin (a misconception about Agile. May be the author doesn't mean that, but this idea is there in the article). Actually, Agile does not prescribe any methods. Rather, there are methods which are built while Agile in mind. So, what is agile? I always preach Ahmed Sidky's definition of Agile:

"Agile is a mindset, that is in the software world, established through 4 values, grounded by 12 principles, and manifested through many many different practices"

In this sense, there is no point of discussing whether Scrum, Kanban, or Open Development are useful methodologies or not. Because the whole idea of usefulness is very situational. What is useful in one situation is not useful in another, what makes sense for one team is totally irrelevant for another. 

Is Open Development an "Evolution of Scrum"?

The other statement which I find misleading is that Open Development is an "evolution of Scrum". Well, this is also incorrect. The Open Development description as appears in the article is totally in accordance with the Agile mainstream since its inception. The stream upon which ideas like Scrum and XP was built. In such mainstream, major advances in Software development were invented, such as automated tests and Continuous Integration and Deployment; and major ideas were revived or re-established like IID (Iterative and Incremental Development) or self-organizing and cross-functional teams.

Well, when to use which? or shall we use None?

Having said all of that, I do admit that Scrum (as per the official Scrum guide) is not suitable in many cases. A better statement is that: Iterative development is not useful in many cases. Some project types cannot be led using timeboxed iterations. Usually, the reason is that the demands of the business and the development pace are much faster than a one-week timebok, which is the smallest iteration timebox. The idea is very simple: If business needs something now, and development is capable of providing this thing at this now, why wait till end of iteration, which may be 2-3 or even more than a week ahead?!

On the other hand, I recently came to a situation where I couldn't use Kanban to manage my project. The unknowns of business are so much, and the availability of the business people is limited; besides great technical risks which needs prolonged time of team collaboration and deep thinking. In this specific case, we switched from Kanban to Scrum.

Managing Flow or Managing Iterations?
Actually, what we have done is to move not from Kanban to Scrum, but from Managing Flow to Managing Iterations. Kanban and Scrum nowadays are metaphors for two more fundamental management ideas: Managing flow versus managing iterations.

So, In this project instance, I found that putting constraints on flow is stopping us rather that helping us. I found that a timebox of one or two weeks gives the team more room to act, react, and innovate. In the meanwhile, it gives us some very high level indicators about team progress.

When shall we use None?
One final thought is that even if you're doing Scrum or Kanban, it's just a starter. It just brings about a sense of discipline and cadence which enables the team to measure it's progress and improvement over time.

So, if this is only a starter, you should expect that teams will move to other practices and probably invent their own process in the near future. At this time, I would say that this team is really Agile!

It’s none of the Managers’ Business!

A team is busy refactoring a very large and overly complex several hundred thousands lines of code. This team would hang on the wall this sign and instructions:

We are busy refactoring the product code for the benefit of all of us

It is a highly technical stuff which you will not grasp, even if you tried hard

We need to concentrate. Please do not keep nagging for status and end dates

When we finish, we'll let you know

Thank you for your understanding :)

Yes, this is a joke, but we've met teams doing this spontaneously! Here is the full story:

In a recent review of three companies which tried to refactor large products (hundreds of thousands line of code), we found that decisions were taken in all three companies to suspend the refactoring effort. Two of them decided to go for a full rewrite and the third decided to minimize further development in this particular product.

This appears to be impatient managers' decision. But, analysing the three cases we reached some very interesting reasons behind these costly decisions. The number one reason was the proposition by the technical team that: "it's none of the managers' business"!

Technical teams had this attitude. Managers had almost no involvement in planning refactoring effort, and the technical teams did not spend any effort to involve busy managers or report status to them. This attitude created opposition to further expenditure on refactoring and was one of the main reasons for management wanting to either discontinue the refactoring effort or rewrite the application from scratch.

We reached the following corollary about  how managers manage refactoring effort:

Managers will not sponsor any activity unless they can track and control it!

 Other successful refactoring attempts witnessed more support from managers. The basic difference was that managers used to receive monthly reports about the status of refactoring, and would accordingly decide on whether to accelerate or slow down refactoring.

Agile Solves Problems and Introduces Others!

Recently, I got to know one of the experts of ESI, the European partner of SEI. I met him in the corridor of ITIDA during his visit to Egypt for one reason or another. We chatted for a couple of minutes, and I will quote him saying:

"Agile solves problems and introduces others!"

It was clear that he has a negative attitude towards "Agile"; may be he is talking about some bad failure patterns of Agile implementation, like those I mentioned in this blog item.

However, I would affirm that the amount of problems that Agile solves are far way bigger than the ones introduced.

I also forgot to ask him: What about CMMI-based process improvement, specially those following waterfall or phased development, does it really solve any problems? or just introduces others :)

Patterns of Agile Adoption Failure

Many teams are trying Agile methods, or rather techniques. Many of them fail, and suffer from this failure, usually because the failure induces resistance in the rest of the organization, and makes later attempts to Agile adoption much more difficult.

The following are four patterns of immature Agile implementations, where team concentrate on some superficial agile techniques without really embracing Agile values and principles:

• Usually team starts with stand-up meetings, and it fails, because it simply hurts their legs for standing 1-2 hours daily! Moreover, the meetings are boring, and take a lot of time discussing issues. The team feels that this "Agile" thing is naive and wastes their time.
• Teams do sprints:
1. Either one complete sprint for every waterfall phase. That is a sprint for analysis, another for design, another for development, and a final one for testing. So, it simply becomes sprintfall rather than waterfall !
2. Or one sprint for every big requirement or group of requirements. The problem with this approach is that no feedback loop exists. In other words, they have split a big waterfall project into smaller ones. Just bigger management overheads
• Teams use sticky notes and decorate the walls with whatever they do, even with requirements in a typical waterfall project. After a while, they lose interest in updating the board notes. Because every step take a lot of time (2-3-weeks), and they cannot see any value of maintaining two databases of requirements and tasks (board and electronic)
• Team does everything just right, but do not take feedback from customers. Somehow better, but still waterfall in the large. Teams iterate to get feedback and to adjust their development effort along the way. If they do not get feedback, the value of iterating is not realized

These are some failure patterns, there are many others; if you have any experience with this, please let us know about it :)

Software Development does not follow any predictable manufacturing process rules

In this excellent presentation about Agile Estimation, by Christoph Steindl. The author lists the difference between what is called "Predictable Manufacturing" vs. "New Product Development". These differences are following Lean principles, and applies to any endeavor.

Reflecting on these differences, it is so clear that Software Development does not following the rules of Predictable Manufacturing defined processes. This is because Software Development is an innovative, intellectual, and usually creative effort that cannot be govered or managed by any strictly defined process. True!

Something came to my mind while reading this, which is how to improve both types of processes.

For any defined process (predictable manufacturing), it is improved by getting it more and more defined to account for new special cases, or new causes for variation.

For any empirical process (new product development), it is improved by enhancing the feedback control system, which steers the development effort, and upon which everyone in the development team should depend.

Imagine what would happen if software is developed without such feedback control system?

In a next blog, I will give examples of famous mechanisms which implements these feedback control systems in software development.

All of them are victims, the customer, and the development team!

Very interesting situation. I’m now out-sourced as a project manager to manage some of the IT projects developed by the company I work for. The client is a big telecom customer of my company. In this case, I deal with the end users of the systems and see and hear how they think of the development team. On the other hand, I deal with the development team closely, and they consider me “on their side”, not on the “enemy/customer” side!

The customer is really suffering. He suffers from so many delays, unexpected results while testing, misunderstandings, unmanaged expectations, resistance of changes in requirements, et cetera. In summary, at best, the customer is suffering. The customer says that the IT was much better in the past, but now, they think that the development team is incapable, but they can’t escape dealing with them because simply speaking, all other vendors are alike or even worse!

On the other hand, the development team is also suffering. They suffer from the changing business at the customer side. They suffer from the large scale of the software size, while there is no central architecture team. They also suffer from other things, like the oppressing environment created by middle management who are also oppressed by the frustrated customer. They finally suffer from the fact that they are not appreciated by any means! All what they hear is that the customer is frustrated because so and so, but they are never rewarded of the tremendous amount of effort they exert.

Both of them, the customer and the development team, are victims. None of them should be blamed. Yes, none of them should be blamed.

The fact is that it is the methodology which should be blamed for such miserable situation. There has to be a change in the attitude, a paradigm shift in the operation of producing software. Techniques like iterative development, incremental delivery, automated tests, continuous customer involvement, and central architecture team are all critical success factors.

I’m afraid this situation is becoming the norm in software development. It started to be an accepted situation that software development is a difficult field and we have to accept all these issues as normal and cope with it; very similar to a situation where one has a disability and he has to cope with it!

I’m feeling so bad for these suffering people, both in business and in development.

Why Requirements Change?

Requirements change because this is the norm. We should never think that frequent requirements changes are sign of poor analysis capability of the team. This is one of the killing mistakes that managers used to do with the development teams!

Two reasons govern this change:

Rule #1: No Customer fully understands the requirements. At best, the customer understands the business problem; and in most cases, he cannot articulate it. What should be done is to implement part of the requirement (potentially the user interface), and let him work on it for a while to get a feeling of the solution, then start a second round of analysis.

Rule #2: the Customer is not a single person. This is always overlooked. This is why RUP says that the customer community should be dissected into groups, and each group should have at least one representative in the customer group.

The project leader/manager/master (call it whatever you like) is the responsible person to enforce these two rules. For the first rule, he should break down the vague requirements into small ones and plan for several iterations, after each of them, he goes back to the customer and get more feedback. For the second rule, he should make sure that all groups of users are represented in the customer group.