Why collect measurements?

Two main reasons to collect measurements:

1. To track your project. A basic task of any software project manager is to track the project progress. Examples are the defect rate, the size of finished functionality versus not finished. A very good technique which reads this measurement and displays it in a graphical format is the burn-up/burn-down charts which are heavily used in Agile development.
2. To improve the process of producing software. Very similar to six-sigma projects, CMMI level 4 and 5 have techniques which starts with an evidence of a mal-functioning process (most commonly supported by measurements).Then, it digs into the root causes, and suggests corrective actions. Finally, it designs measurements to quantify the impact of corrective actions, and start over again. This ‘Continuous Improvement’ cycle is one of the techniques used in any engineering discipline.

Measurements are an invaluable means for management, especially for software development, which is a deep intellectual engineering discipline. I may safely say that without measurements, you may not reach good management decisions what so ever. One of the sayings that is said again and again in Balanced Score-Cards environments is that “you cannot manage what you cannot measure, and you cannot measure what you cannot define”, is really true.

What to keep in mind while designing your measurements

Out of the many good advice about measurement systems for software processes, I have benefited most from these four pieces of advice:

1. Set the goal before you measure.

It is tempting to set out a range of measurements, each costs some good time in calculation, while you don't need it. It is also very common that a project manager or a process engineer says: "we will need them later on, so why not collect them anyways?", another third idea is that "collecting many measurements may give you insight about the deficiencies in the project or the process, so the more you collect, the more insight you gain"!

I have lived with these philosophies for a while, and spent so much time collecting lots of measurements just for the sake of measuring, and I really think it was pointless.

The strategy I'm following right now is to set up the next objectives (one or two, at maximum), design the measures which I really need, and implement the infrastructure which will help me collect these measures automatically.

2. Distribute measurements results and analysis to the development team

Results of measurements may be very interesting for the team to visualize. Not only interesting, but also motivating. As a practise I continuously follow, I send the results of the measrements and analysis to the team, and discuss with them their implications. This helps me to:

• Motivate the team to work towards achieving better measurements.
• Motivate the team to record measurements more accurately.
• Get the team to the point that they trust the value of the measurements as a means for project management, and as a means for process improvement.

3. Minimize the time of collecting measurements.

This is a point which is very sensitive. If the cost of collecting a measurement is high, it is more likely that the whole measurement system will collapse! Developers like to develop, not to "waste" their time counting number of hours, bugs, lines of code, et cetera. If it constantly takes time to calculate measures, it will distract the team, and demotivates them.

To mitigate this issue, automate your measurement calculation. How? this depends on the measurement and the tools used.

4. Do not use measurements for evaluating individual developers.

Measurements is a means for project tracking and process improvement. It has also a side effect of motivating the team and achieving goals. But, a common mistake is to use these measurements to pinpoint productivity issues to individual developers.

I used to work in a strong measurements environment sometime ago, in which measurements are collected to monitor all software development phases. After the project ends, the same measurements are used to evaluate developers, and calculate their mid-year bonuses accordingly.

One of these measures was the number of review issues opened in peer reviews. In time, we found developers not opening any review issues, and prefer to communicate the review results either orally or by mail, not through the issue tracking system (which automatically calculates the measure). The reason they did that is no to affect the bonuses of themselves! I have to say that I personally found myself inclined not to open any review issues just like the rest of the team. It is a feeling that you cannot resist.

If you are interested to read more about such valuable advice, you will find these advice (and others) in the following references:

• More process Patterns, by Scott Ambler.
• Rapid Software Development: Taming Wild Software Schedules, by Steve McConnell.