Tragedies of Performance Evaluation of Software People

I'm constantly having conflicts with managers who insist on backward methods of performance evaluation for agile teams. It is one of the miseries of software teams, because in most cases, such methods demotivate individuals and undermine team values. The issue is serious to the extent that Professor Edward Lawler wrote a recent post in Forbes magazine titled: Eliminating Performance Appraisals. In this post, prof. Edward questions whether organizations should stop doing performance appraisals altogether, because they “do more damage than good”, and “when poorly done, they create a multitude of problems”!

I felt I have a moral obligation to write something in this regard. My contribution in this post is a collection of resources and an attempt to structure the topic into three high level subtitles:

1. What's evil about performance appraisals
2. Should smarter people be rewarded more money (as a means of motivation)?
3. Using process data for the sake of individual performance

If you are a smart person suffering from a performance evaluation scheme from the middle ages, send your HR manager this post, and let him read the resources mentioned below. May be (I'm not sure) it may help.

1. What's evil about performance appraisals

First, this is an article: Should I Rank My Employees? from the Wall Street Journal. It mentions the following interesting survey:

"Stanford professors Jeffrey Pfeffer and Robert Sutton are among the most fervent critics. They cited a survey of more than 200 human resources professionals from companies employing more than 2,500 people that found that even the more than half the companies used forced ranking, the respondents 'reported that forced ranking resulted in lower productivity, inequity and skepticism, negative effects on employee engagement, reduced collaboration, and damage to morale and mistrust in leadership.' … "

Another blog by the famous software engineering management author: Esther Derby titled: Should a ScrumMaster (or any coach) Give Performance Appraisals? In this blog, Esther asks how would it be good while "psychologists know that 80 percent of people believe their performance is above average"

The topic of performance appraisal are very controversial to the extent that one of the so famous books in management in the last years is called: "Abolishing Performance Appraisals: Why They Backfire and What to Do Instead"

2. Should smarter people be rewarded more money (as a means of motivation)?

We need to change our believes that a merit system will not work unless it ranks employees into classes and every class is motivated by some more money.

This is a must-see master piece by RSA Animate: Drive: The surprising truth about what motivates us:

In this short animation, they attack very strongly the idea about motivation using financial incentives when it involves cognitive and knowledge work. At minute 1:50 till 4:15, this animation sites the results of a study done by three top universities in Engineering and Technology: MIT, University of Chicago, and Carnegie Mellon university:

"As long as the task involve only mechanical skill, bonuses worked as expected (higher pay = better performance). But, once the task called for rudimentary [basic or simple] cognitive skill, a larger reward led to poorer performance!"

Also, one of the most important books in software engineering history is Rapid Software Development: Taming Wild Software Schedules by Steve McConnell. This book was named the best Software engineering book in the year by Software Development magazine's Jolt Award. Please refer to this book to understand what motivates Software engineers. Chapter 11: Motivation is the best reference I read about motivating software engineers. In page 262, you'll find this quote: 

"At least two-dozen studies over the last 30 years have shown conclusively that people who expect to receive a reward for doing their jobs successfully don't perform as well as those who expect no reward at all (Kohn 1993). The work itself is the greatest motivator, and the more a manager stresses an external reward, the less interested the developer becomes in the work itself, and the more potential motivation is lost."

The author makes a very clear alarm that incentives has many forms which are non-monetary like:

• Dinners with company executives
• Vacation-time bonuses
• Gifts of appreciation (theater tickets or dinners for two)
• Sincere praise directed at a specific accomplishment
• Team T-shirts, polo shirts, rugby shirts, watches, pins, mugs, posters,
• Humorous or serious awards in the form of plaques, certificates, trophies, and the like
• Special events to celebrate significant accomplishments; depending on your team's preferences, and event might be a dinner at a favorite restaurant, a show, a trip, a ski day, or a dinner at the boss's house (or, for maximum effect, the boss's boss's house)
• Exceptions to company policies for the team, such as casual-dress Friday, a Ping-Pong table in your team's section of the building, free soda pop in the team refrigerator, and so on
• Special courses (outside the local area)
• Sponsorship at conferences that the organization would not ordinarily sponsor
• Grade-level promotions

You can download this book from this location.

One more article called One More Time: How Do You Motivate Employees?, which sold more than a million reprints making it the most popular article in the history of the Harvard Business Review. In this diagram, notice the particular position of the salary and it's weight as a demotivator rather than a motivator. 

Motivators and demotivators - Notice that salary (money) is not among the motivators, but rather a demotivator in case salary is not sufficient

Actually, what this diagram tells us is the same exact phrase mentioned in the RSA movie above at minute 4:50:

"If you don't pay enough, people won't be motivated. Pay people enough to take the issue of money off the table"

3. Using process data for the sake of individual performance

Forcing managers to use process data, like feature estimates, number of bugs, reported actual hours, etc. to evaluate team members will blow up the process metrics themselves. This, in turn, has a drastic effect on the process itself, making the team process-blind (not being able to improve the process based on process metrics) Being process-blind is the direct result of exploiting process metrics and reuse them as Targets, and as per Goodhart's law: "When a measure becomes a target, it ceases to be a good measure."

Why this happens? To understand the dynamics, let's recall the general rule put by Tim DeMorko, the author of the great book: Peopleware: Productive Projects and Teams's, a book which had an impact on the whole industry; the general rule is known as DeMarko's Principle:

"Effort moves towards whatever is measured"

This can happen subconsciously or even consciously; and this is considered one of the major pitfalls of measuring, as explained the this article: The seven pitfalls of performance measurement which considers: Risk of manipulation of performance data as one of them. When it comes to individual evaluation, use of process metrics becomes a real risk, and opportunity of manipulation is so widespread. Writers have been noting this again and again, to the extent that some writers like Eli Goldratt, the father of Theory of Constraints says:

"Tell me how you measure me, and I will tell you how I will behave!"

In summary, using process metrics to evaluate individuals distorts the metrics and after a while, the metrics lose their meaning and cease to reflect real process data.

A Final Word

Finally, I have to say that this topic is vast and needs a lot of effort to make a shift in the mindset of project and HR managers of software in Egypt. It is not easy, but impossible is not on my dictionary :)

Trying to Understand Why Kanban Works

Recently, I came across a book chapter explaining Little's Law; published by Massachusetts Institute of Technology. This chapter is an excellent read which is simple and yet digs into the heart of Little's Law. Reading about queuing theory is provocative and makes me relate many of the already established ideas to Lean software and Kanban.

One of the issues which I found this book chapter explaining very well is the difference between Little's law in its original formulation (which considers the Arrival Rate as one of the formula parameters) and its application in manufacturing systems (which replaces arrival rate with throughput). Here is an explanation:

Little’s Law

Little's Law state that:

L = λ W

where

L = average number of items in the queueing system
λ = arrival rate of new items to the system
W = average wait time for an item in the system

John Little argues that this law is robust and generic and holds exactly to any queuing system given an essential condition; which is "to have a finite window of observation that starts [when the system is empty] and stops when the system is empty" (p.88). Moreover, Little provides a mathematical proof for the law along with many applications in reality.

Little’s Law in Software Systems

Let's jump into software and map these concepts to Kanban. In Kanban, this is how Little's Law is stated:

WIP = Th * CT

where

WIP (average Work In Process) = average amount of unfinished items (bugs, user stories, change requests, etc.) in the development system
Th (Throughput) = Team output in unit of time
CT (Cycle Time) = Average time it takes the team to finish one item

Two basic differences between the two formulas: The first one talks about input or arrival rate, whereas the latter talks about output rate or throughput. The second issue is the condition which Little stated: that the system should start with 0 items and end at 0 items. In software, we rarely witness a system which has no maintenance requests.

To resolve these differences, Little indicated a more subtle condition of the Law to hold, which is no items to go inside the system and get lost or do not get finished, which he calls “conservation of flow” (p. 93). If we apply this condition to the system, we can easily show that the input rate = output rate, and therefore, we can relate Lambda (λ) with throughout (th).

For the second condition (system should start and end in 0 items), Little argues that the law “still applies, at least as an approximation, as long as we select a time interval that is long enough”. (p. 93)

What’s useful about this Law?
Little’s Law is a very powerful tool to use for managers. If we have two parts, and we can instantly calculate the third.

Increase team throughput

If your capabilities are fixed and want to increase throughput, the only solution is to increase WIP, and increase the team size. If this is not doable, and your input rate is high and getting higher, you may decide to decrease the cycle time instead of increasing the WIP and this may involve automating the deployment process.

Stabilize Cycle time

If you are struggling to stabilize the process so that to meet some SLA, Little’s Law gives insight about what to change in the system. You may either increase/decrease with WIP or empower your team to meet a specific SLA.

Study behavior of specific request types
Little’s Law works for a sample of item in the system as well. You may zoom into the items in the system and study the dynamics of a specific segment (like high severity bugs, for instance).

What to keep in mind while designing your measurements

Out of the many good advice about measurement systems for software processes, I have benefited most from these four pieces of advice:

1. Set the goal before you measure.

It is tempting to set out a range of measurements, each costs some good time in calculation, while you don't need it. It is also very common that a project manager or a process engineer says: "we will need them later on, so why not collect them anyways?", another third idea is that "collecting many measurements may give you insight about the deficiencies in the project or the process, so the more you collect, the more insight you gain"!

I have lived with these philosophies for a while, and spent so much time collecting lots of measurements just for the sake of measuring, and I really think it was pointless.

The strategy I'm following right now is to set up the next objectives (one or two, at maximum), design the measures which I really need, and implement the infrastructure which will help me collect these measures automatically.

2. Distribute measurements results and analysis to the development team

Results of measurements may be very interesting for the team to visualize. Not only interesting, but also motivating. As a practise I continuously follow, I send the results of the measrements and analysis to the team, and discuss with them their implications. This helps me to:

• Motivate the team to work towards achieving better measurements.
• Motivate the team to record measurements more accurately.
• Get the team to the point that they trust the value of the measurements as a means for project management, and as a means for process improvement.

3. Minimize the time of collecting measurements.

This is a point which is very sensitive. If the cost of collecting a measurement is high, it is more likely that the whole measurement system will collapse! Developers like to develop, not to "waste" their time counting number of hours, bugs, lines of code, et cetera. If it constantly takes time to calculate measures, it will distract the team, and demotivates them.

To mitigate this issue, automate your measurement calculation. How? this depends on the measurement and the tools used.

4. Do not use measurements for evaluating individual developers.

Measurements is a means for project tracking and process improvement. It has also a side effect of motivating the team and achieving goals. But, a common mistake is to use these measurements to pinpoint productivity issues to individual developers.

I used to work in a strong measurements environment sometime ago, in which measurements are collected to monitor all software development phases. After the project ends, the same measurements are used to evaluate developers, and calculate their mid-year bonuses accordingly.

One of these measures was the number of review issues opened in peer reviews. In time, we found developers not opening any review issues, and prefer to communicate the review results either orally or by mail, not through the issue tracking system (which automatically calculates the measure). The reason they did that is no to affect the bonuses of themselves! I have to say that I personally found myself inclined not to open any review issues just like the rest of the team. It is a feeling that you cannot resist.

If you are interested to read more about such valuable advice, you will find these advice (and others) in the following references:

• More process Patterns, by Scott Ambler.
• Rapid Software Development: Taming Wild Software Schedules, by Steve McConnell.