Take your time to pay for this technical debt, but let me know how you would prevent it in the future!

In a previous blog, I explained why 'Technical debt is not bad!', and explained that this idea pays for itself on the long run. However, the title of this post indicates the right question asked by senior management while negotiating time to pay for technical debts.

The idea is that: Part of fixing a problem is preventing it from re-occurring.

This does not mean that technical debt will not re-occur; it means that the specific issue that caused this type of technical debt should be resolved from the roots and hopefully not re-occur.

This way of thinking is a characteristic of high mature organizations.

Technical debt is not bad!

Technical debt is usually thought of as a notorious thing indicating bad (or no) design, and leads to projects failure after several iterations.

Actually, this is not true. Refactoring is a constant effort done by the development team even with the most intelligent designs. Developers (and their designs) matures "while" working on the project. This maturity emerges from that following facts:

• The level of understanding of the business domain gets clearer.
• Seeing the design at work develops more brighter design ideas.
• Development work is the most effective learning experience for developers. So, the more you develop, the more you learn about design concepts, and the more you see opportunities for improving the existing design.

This is why teams should find a way to incorporate paying technical debt every now and then. By incorporate, I mean to reflect on it as part of every iteration, or may be dedicate an entire iteration for it if needed.

Software Development does not follow any predictable manufacturing process rules

In this excellent presentation about Agile Estimation, by Christoph Steindl. The author lists the difference between what is called "Predictable Manufacturing" vs. "New Product Development". These differences are following Lean principles, and applies to any endeavor.

Reflecting on these differences, it is so clear that Software Development does not following the rules of Predictable Manufacturing defined processes. This is because Software Development is an innovative, intellectual, and usually creative effort that cannot be govered or managed by any strictly defined process. True!

Something came to my mind while reading this, which is how to improve both types of processes.

For any defined process (predictable manufacturing), it is improved by getting it more and more defined to account for new special cases, or new causes for variation.

For any empirical process (new product development), it is improved by enhancing the feedback control system, which steers the development effort, and upon which everyone in the development team should depend.

Imagine what would happen if software is developed without such feedback control system?

In a next blog, I will give examples of famous mechanisms which implements these feedback control systems in software development.

Why collect measurements?

Two main reasons to collect measurements:

1. To track your project. A basic task of any software project manager is to track the project progress. Examples are the defect rate, the size of finished functionality versus not finished. A very good technique which reads this measurement and displays it in a graphical format is the burn-up/burn-down charts which are heavily used in Agile development.
2. To improve the process of producing software. Very similar to six-sigma projects, CMMI level 4 and 5 have techniques which starts with an evidence of a mal-functioning process (most commonly supported by measurements).Then, it digs into the root causes, and suggests corrective actions. Finally, it designs measurements to quantify the impact of corrective actions, and start over again. This ‘Continuous Improvement’ cycle is one of the techniques used in any engineering discipline.

Measurements are an invaluable means for management, especially for software development, which is a deep intellectual engineering discipline. I may safely say that without measurements, you may not reach good management decisions what so ever. One of the sayings that is said again and again in Balanced Score-Cards environments is that “you cannot manage what you cannot measure, and you cannot measure what you cannot define”, is really true.

What to keep in mind while designing your measurements

Out of the many good advice about measurement systems for software processes, I have benefited most from these four pieces of advice:

1. Set the goal before you measure.

It is tempting to set out a range of measurements, each costs some good time in calculation, while you don't need it. It is also very common that a project manager or a process engineer says: "we will need them later on, so why not collect them anyways?", another third idea is that "collecting many measurements may give you insight about the deficiencies in the project or the process, so the more you collect, the more insight you gain"!

I have lived with these philosophies for a while, and spent so much time collecting lots of measurements just for the sake of measuring, and I really think it was pointless.

The strategy I'm following right now is to set up the next objectives (one or two, at maximum), design the measures which I really need, and implement the infrastructure which will help me collect these measures automatically.

2. Distribute measurements results and analysis to the development team

Results of measurements may be very interesting for the team to visualize. Not only interesting, but also motivating. As a practise I continuously follow, I send the results of the measrements and analysis to the team, and discuss with them their implications. This helps me to:

• Motivate the team to work towards achieving better measurements.
• Motivate the team to record measurements more accurately.
• Get the team to the point that they trust the value of the measurements as a means for project management, and as a means for process improvement.

3. Minimize the time of collecting measurements.

This is a point which is very sensitive. If the cost of collecting a measurement is high, it is more likely that the whole measurement system will collapse! Developers like to develop, not to "waste" their time counting number of hours, bugs, lines of code, et cetera. If it constantly takes time to calculate measures, it will distract the team, and demotivates them.

To mitigate this issue, automate your measurement calculation. How? this depends on the measurement and the tools used.

4. Do not use measurements for evaluating individual developers.

Measurements is a means for project tracking and process improvement. It has also a side effect of motivating the team and achieving goals. But, a common mistake is to use these measurements to pinpoint productivity issues to individual developers.

I used to work in a strong measurements environment sometime ago, in which measurements are collected to monitor all software development phases. After the project ends, the same measurements are used to evaluate developers, and calculate their mid-year bonuses accordingly.

One of these measures was the number of review issues opened in peer reviews. In time, we found developers not opening any review issues, and prefer to communicate the review results either orally or by mail, not through the issue tracking system (which automatically calculates the measure). The reason they did that is no to affect the bonuses of themselves! I have to say that I personally found myself inclined not to open any review issues just like the rest of the team. It is a feeling that you cannot resist.

If you are interested to read more about such valuable advice, you will find these advice (and others) in the following references:

• More process Patterns, by Scott Ambler.
• Rapid Software Development: Taming Wild Software Schedules, by Steve McConnell.

Why process is important from day 1?

Sometimes we here some devastating (yes literally devastating) ideas like:

• Process is pure overhead.
  
• Process is discovered while working on the project (after we start the project).
  
• Let’s start coding, we will refactor anyways.
  
• Just jump in the code, things will get crystallized as you go.
  

The problems of such sayings is that it tempts poor developers to start a project with the unhandled risks inherent in the software development endeavor.

Famous processes like RUP markets for itself as “a risk removal process”. Agile methods assume to mitigate many risks of software development, even if not having a list of risks which we keep an eye on, which is true in my humble opinion.

The fact of starting a project without a clear view of how to do configuration management, how to do estimation, how to document requirements, how to involve stakeholders and get feedback on requirements, when and how to test, what are the bugs types, and which of them are to be fixed first, how, when and by whom architecture and design is going to be done, how to do tracking and what measures will indicate progress, how you will handle change, and many others. The fact of starting the project without a clear answer of such questions is pure act of failing the project, causing some persons to suffer sometime of their lives and get disappointed and demotivated at the end of the way (the developers), and causing others to lose some money, lose business opportunities, and lose trust in software development teams (the customers).

I hope this is a clear answer to the question. Please take some time and think of how you will develop software before you take the responsibility of any projects.

The next question is: Does process limit develop creativity? This needs a separate post.

All of them are victims, the customer, and the development team!

Very interesting situation. I’m now out-sourced as a project manager to manage some of the IT projects developed by the company I work for. The client is a big telecom customer of my company. In this case, I deal with the end users of the systems and see and hear how they think of the development team. On the other hand, I deal with the development team closely, and they consider me “on their side”, not on the “enemy/customer” side!

The customer is really suffering. He suffers from so many delays, unexpected results while testing, misunderstandings, unmanaged expectations, resistance of changes in requirements, et cetera. In summary, at best, the customer is suffering. The customer says that the IT was much better in the past, but now, they think that the development team is incapable, but they can’t escape dealing with them because simply speaking, all other vendors are alike or even worse!

On the other hand, the development team is also suffering. They suffer from the changing business at the customer side. They suffer from the large scale of the software size, while there is no central architecture team. They also suffer from other things, like the oppressing environment created by middle management who are also oppressed by the frustrated customer. They finally suffer from the fact that they are not appreciated by any means! All what they hear is that the customer is frustrated because so and so, but they are never rewarded of the tremendous amount of effort they exert.

Both of them, the customer and the development team, are victims. None of them should be blamed. Yes, none of them should be blamed.

The fact is that it is the methodology which should be blamed for such miserable situation. There has to be a change in the attitude, a paradigm shift in the operation of producing software. Techniques like iterative development, incremental delivery, automated tests, continuous customer involvement, and central architecture team are all critical success factors.

I’m afraid this situation is becoming the norm in software development. It started to be an accepted situation that software development is a difficult field and we have to accept all these issues as normal and cope with it; very similar to a situation where one has a disability and he has to cope with it!

I’m feeling so bad for these suffering people, both in business and in development.

Customer and Project Team Bill of Rights

The concept is brilliant. The first time I read it I said: “Yes, this is really what I always felt missing!”

Steve McConnell, in his excellent book Software Project Survival Guide, mentions a set of rights that should be respected for both the project team, and the customer who owns the product. Please spend some time reading them carefully; this is a prescription for project success.

Customer's Bill of Rights

I have the right:

1. To set objectives for the project and have them followed
2. To know how long the software project will take and how much it will cost
3. To decide which features are in and which are out of the software
4. To make reasonable changes to requirements throughout the course of the project and to know the costs of making those changes
5. To know the project’s status clearly and confidently
6. To be apprised regularly of risks that could affect cost, schedule, or quality, and to be provided with options for addressing potential problems
7. To have ready access to project deliverables throughout the project

Project Team's Bill of Rights

I have the right:

1. To know the project objectives and to clarify priorities.
2. To know in detail what product I’m supposed to build and to clarify the product definition if it is unclear.
3. To have ready access to the customer, manager, marketer, or other person responsible for making decisions about the software’s functionality.
4. To work each phase of the project in a technically responsible way, especially to not be forced to start coding too early in the project.
5. To approve effort and schedule estimates for any work that I will be asked to perform. This includes the right to provide only the kinds of cost and schedule estimates that are theoretically possible at each stage of the project; to take the time needed to create meaningful estimates; and to revise estimates whenever the project’s requirements change.
6. To have my project’s status reported accurately to customers and upper management.
7. To work in a productive environment free from frequent interruptions and distractions, especially during critical parts of the project.

It is top priority for managers to spread this atmosphere of mutual recognition of rights. Customer, as well as project team, sometimes denies other’s rights. The customer should ensure that he is available to answer the team’s questions. Many a time, I keep waiting many days to book a meeting with the customer or even get an email feedback from him. On the other hand, customers sometimes suffer from project teams not reporting to them the true status of the project. Many things like that take place, resulting at the end of an unsuccessful project end.

Why Requirements Change?

Requirements change because this is the norm. We should never think that frequent requirements changes are sign of poor analysis capability of the team. This is one of the killing mistakes that managers used to do with the development teams!

Two reasons govern this change:

Rule #1: No Customer fully understands the requirements. At best, the customer understands the business problem; and in most cases, he cannot articulate it. What should be done is to implement part of the requirement (potentially the user interface), and let him work on it for a while to get a feeling of the solution, then start a second round of analysis.

Rule #2: the Customer is not a single person. This is always overlooked. This is why RUP says that the customer community should be dissected into groups, and each group should have at least one representative in the customer group.

The project leader/manager/master (call it whatever you like) is the responsible person to enforce these two rules. For the first rule, he should break down the vague requirements into small ones and plan for several iterations, after each of them, he goes back to the customer and get more feedback. For the second rule, he should make sure that all groups of users are represented in the customer group.

Pulling Requirements Analysis out of the project lifecycle !

The question is: Should requirements analysis be part of the software project lifecycle?

The answer is yes and no. It is very important to distinguish two types of analysis, Business analysis and software analysis.

Business analysis should be a fast, lightweight, and continuous activity. The outcome of this activity is semi-analyzed requirements which are ready to be developed. Also, these requirements should be fully characterized in terms of customer priority, marketing impact, and technical risk, and any other requirement property that the organization sees important for planning. These requirements are backlogged till the team (customer and development) is ready to work on them.

Next, once a new project is to start, several requirements are pulled out of this backlog and they start getting developed. The strategy of selection depends largely on the business priority, and the technical risk. In other situations, it may depend on other requirement attributes, like the size, stability, et cetera.

What I have described so far is all analysis work that should not be considered part of a software project, and in the meanwhile, it constitutes about 60% of the analysis work.

On the other hand, there is still some analysis work that should be done on the project. This work is done by the developers who are going to work on design and development of the requirement. They should work face-to-face with the customer or may be a subject matter expert to dig into the details of the business, They should also make some prototyping and simulation to verify that the non-functional requirements will be met.

In all cases, they need to develop an iteration, deliver to the customer, and recieve feedback. This is all part of the analysis effort. See this post for more details.