Measuring and Managing Performance in Organizations

I teach courses on software metrics and do some research on software-related measurement. As Austin points out in his book, many of the well-known advocates of metrics in the software community are blind to the issues that he raises, or they dismiss the issues as social science hooey that won't affect serious engineering. They are so, so wrong. This is a useful, readable book, that teaches hard lessons.

Robert Austin presents an idealized model of a managed organization. Instead of looking at an organization made up of thousands of employees and a few hundred managers arranged in a hierarchy, Austin's model consists of three participants: a principal, i.e. a manager, and an agent, i.e. an employee, and finally a customer who buys the goods or services provided by the agent under the supervision of the principal. He also assumes that an agent's job consists of two activities and the customer is happy if the agent performs well in both. Austin looks at the cases where the principal can monitor neither of the two activities, where she can monitor only one of the two activities, or where she can monitor both activities. According to the model the agent will behave differently in all three cases. If the principal cannot (or will not) measure either activity, then we have delegated management, if she can measure both activities, then we have a fully supervised model, and if she can measure only one of two activities, we have a dysfunctional model. When delegating management, the assumption is that agents want to work well, that they are not deriving maximum satisfaction by exerting the least amount of effort. When supervising, the principal evaluates overall performance by measuring certain aspects of the agent's activity. Austin's conclusion is that measuring performance won't work unless you can measure everything employees should be doing (i.e. full supervision). Incomplete measurement is not only useless, it is dangerous since it motivates agents to make efforts only for what is measured. For example, if a help desk line measures performance by the number of calls an employee takes, then employees are motivated to spend very little time per call. The customer is left dissatisfied, but the measurements show that the agent is providing first class results. Austin calls this situation dysfunctional. Throughout the book, Austin emphasizes dysfunction to the point where it seems he dismisses any and all attempts at measurement, but to quote Austin, the central message of the book is that "organizational measurement is hard". It's not impossible. He suggests one method, probabilistic measurement, to mitigate dysfunction. For instance, if dysfunction comes from being unable to measure everything an agent does, e.g. you just can't have your supervisors listen to all help desk calls, the principal can carry out random samplings of performance, e.g. you can record all the calls and listen to a random selection of them each day. The agent will then expend effort along those dimensions that cannot be completely measured simply because he knows they might be. All in all, an effectively simplified model of organizations sure to spark healthy and constructive debate. Vincent Poirier, Dublin

This book provides an amazingly convincing explanation for why employee incentive programs often do more harm than good. It's often because knowledge work is too complicated to benefit from any simple measures. The core argument of the book uses some mathematical reasoning that will be accessible to anyone who stayed awake through Economics 101. This is illuminating enough, but then Austin continues to add on additional insights.I've placed this book on my shelf next to The Logic of Failure (Doerner) and Normal Accidents (Perrow). All of these books provide solid scientific arguments for the limits of management.As a software tester, the most obvious application of the book is as an explanation of exactly when counting defects (found by testers, or introduced by programmers) is likely to lead to trouble.

This book is not - a light read - long - mathematical - about software specific issues and the arcana of that discipline - a cookbook for deciding what to measure, how to measure, how to analyze, how to reportThis book describes - the uses of measurement, informational vs motivational - a (increasingly elaborated) measurement model - an objective definition of dysfunction and how it arises because of measurement - a model of "supervision" and how measurement supports (or interferes with) various kinds of supervision - a suggestion about organizational incentives - some strengths & weaknesses of well known assessement systems; e.g., ISO, SEI - the interview method and answers applying the model with 8 well-known writers on software and software management issues. The messages I got - setting up measurement systems is not easy. There are many pitfalls - picking the goal(s) that the measures will support is critical - picking the measures. Some things are too expensive to measure - deciding how much to spend - deciding what to report to whom - (to my own chagrin) that I had personnally and fully encountered most pitfalls - it's easy for those measured to subvert the measuring - partial measurement may make things worse - informational measurement (measuring and results stay with those measured) is less likely to be subverted - purely economic models are not fully adequate explanations of employee-employer relationships.

In my role as a methodologist, business process modeler, and designer of metrics and measurement programs I have long been concerned with the preverse and unanticipated effects of such measurement programs. For the first time Austin has identified what is going wrong with most types of measurements and offers a model for how to correctly construct a non-disfunctional approach to measuring things in the real world. I now understand what is wrong with the Consumer Price Index, why my marriage failed, and a lot of other inexplicable things about the world around me. I would urge every manager and professional to read at least the first few chapters of this book in order to understand the tremendous harm incorrect measurement can do and how collect and use measurements properly