Metrics are Hard: Rewarding Performance in an Agile Environment

Recently I was reading a post by Matt Heusser titled Performance Improvement, not Process Improvement.

While I agree with the post I’m concerned with an assumption it appears to be making. It seems to be making the assumption that rewarding performance based upon metrics creates better overall performance. I don’t necessarily think this is true in an agile environment. Here are my concerns:

1. Rewarding performance based upon metrics assumes that there is a fair and equitable method of objectively measuring performance. Team Arty’s five stars are the same as Team Benny’s five stars. If they are not, their is no way to fairly reward performance across teams.
2. It also assumes that people have some impact on the metric. Sales people can generate more sales, but have no direct impact on how many defects are in the product they sell, or how well the product they sell is marketed.
3. Rewarding performance based upon metrics assumes that the behavior being encouraged is beneficial. If you measure something too granular such as sales you potentially ignore other aspects of the customers relationship. If you measure something that is not granular enough, like overall profits or gross income, you run the risk of creating a mentality where the only thing that matters is the bottom line. I don’t believe that is a healthy perspective.

Because of these issues, I find it difficult to believe that there are any metrics that are effective for measuring groups of agile teams performance.

I know what you’re thinking. What about story points? Unfortunately, story points are invalid as a performance metric. Because they are generated dynamically on each team they are completely unreliable for measuring cross team performance. They even struggle at measuring a teams current performance against past performance!

Time from start of story card to delivery to customer? How can you compare teams which work primarily on the web who can practice continuous integration with teams which have to deal with software people must download and install? One team is able to deploy within hours of creating a fix and can roll back by performing an SVN revert, whereas the other needs to be much more careful because in order to revert their changes they have to get people to actually remove and re install their software.

Defects reported in production? Maybe. I define a defect as anything that doesn’t behave in a way that customers anticipate it to. To me, this could potentially be a good metric. My concern is that it doesn’t work unless the software has reached production. It also makes it very easy to ignore defects as “design features.”

I believe rewarding performance of an agile team is not so simple as establishing a metric that they are continuously measured against. Instead I suggest setting clear measurable goals and rewarding them upon completion of each goal. Repeat this during each sprint and in every release cycle.

Remember, these are not a performance goals. Don’t say “complete 10 story cards in an iteration.” Instead set an objective. “Ship version 1.0 of our software,” or “increase server up time to 99.999% for 3 months running.”