Good Data or Bad Data: The Idiosyncratic Rater Effect.
Performance management redesign. It may sound boring, but it’s actually everything. Every employee is affected by your performance management system. It determines how they get paid, promoted, evaluated. It touches everybody.
We all know this, and it’s why in recent years many companies have taken radical steps to improve their performance management. One bold step many companies have tried is getting rid of ratings altogether. Recently a CEB report indicated that companies who did so are doing worse than companies who have kept them. But the whole debate about ratings vs. no ratings is a complete red herring. The question isn’t ratings or no ratings. It’s do you want good data or bad data?
Ratings generate bad data. The reason why has been known for decades. It’s called the “idiosyncratic rater effect.” Big term, but it simply means that we human beings are unreliable raters of other human beings. All the data show that when I rate you, over 60% of your rating is about me, and not you. Which is a big problem, because we are paying, promoting, training, and deploying you based on that rating — which doesn’t reflect you. So ratings are a terrible performance measurement tool, because they claim to measure one thing but actually measure another.
So, by all means, throw out your current rating system. But don’t do it because people hate ratings. Do it because ratings are bad data. It comes down to a measurement question, and nobody seems to be trying to answer it: how do we reliably reveal the true range in performance in an organization so that we can invest differentially in our people? That’s the question to ask.
6 Comments
Comments are closed.
Thanks for this post, very Interesting!
But now what? performance measurement is a must, what we can´t be measured, can´t be improved!
Do you have any suggestions?
Hi Berta,
Great question! The thing is, people can be reliable raters – but only of their own experiences. Definitely check out our Data Fluency Series, especially How to Tell Good Data from Bad Data, Do Current HR Tools Give Us Good Data and the Case Study which goes through a tool that does provide good, reliable data on talent – but only by allowing people to measure their own thoughts, feelings, and experiences. Enjoy! And please comment or let us know if you have more questions!
I wonder if the indisyncratic rater effect holds true for 360 assessments for purposes of behavioural coaching improvement? Anyone know of any such research?
Hi Rick,
Yes he discusses the 360 assessments, they are even worse because they simply increase the number of bad data inputs, leading to even more confusing and incorrect information.
They have amazing examples of how badly people assess potential in other people too, and how disengaging it can be encouraging people to build on their weaknesses rather than following their strengths, so this will likely be relevant to behavioral coaching. I strongly suggest you read his book.
I think ratings are good as long as there is an anonymous component (where the person evaluated is evaluated by someone distant, without physical characteristics).
What I am trying to say is that people should be evaluated without a previous knowledge of the person evaluated (maybe after a system of coding), and absolutely no physical description (or photo of employee, so that they could not look it up on social media or tools used, like Chat etc).
I think when there is a pictured representation of the people evaluated, it is when people are prisoners of their own biases.
Looks and nepotism are what rules our present world and makes us take bad decisions.
This would require a bit more organization but would make the system more reliable and accurate.
When there is a will there is a way.
This is brilliant! I have been a manager for many years and it’s the first time I look at ratings as “data points”. When I think about ratings and performance reviews I’ve done, I now understand why I was sometimes doubting myself and had a feeling of “who am I to give this rating”. So I usually look at other data points to measure people’s performance in terms of their progress on specific projects and objectives. All in all, a super difficult exercise when you want to do it right and unfortunately many managers, me included, don’t allocate enough time in this area, because of all the other functional responsibilities and ongoing work. Insightful video, thanks!