I admit I designed or helped to design 5 assessment systems in my career. No, I’m not proud. Frankly, I’m pretty sure the first 3 brought net negative value, i.e., they created more harm than value.
So when we set out to reinvent our salary system, I said publicly (and more than once): “An assessment system? Over my dead body.”
Fast forward 7 years, and it was time to eat my own words. While the change to transparent salaries was a big thing, and literally no one would instead go back to what we had before, we saw issues piling up.
An interesting fact, among others, some of the issues were:
- Raises not happening frequently enough
- Too little money spent on salary increases
And yes, that was all in a system where anyone could self-set their own salary.
Long story short, a part of the best way out we could devise meant introducing an assessment system.
The Fallacy of Assessment
Assessment systems are a standard in organizations, big and small. There are broadly accepted good practices, like including perspectives from everyone around (so-called 360 assessments) or focusing on facts (outcomes, observable behaviors, etc.).
The bottom line is the aspiration to improve the objectivity of any given assessment.
And that’s fool’s gold.
The fallacy of assessment systems in a professional context is that there’s no such thing as objectivity. There can’t be.
As Marcus Buckingham and Ashley Goodall eloquently explain in Nine Lies About Work, we’re fundamentally incapable of answering questions like “How good are Pawel’s software development skills?” or “How good of a leader is Pawel?”
That holds true even if we pretend to assess observable artifacts, like Pawel’s code or his behaviors during staff meetings. We could observe but a fraction of what a person employs to deliver the outcomes.
To make things worse, even when we consciously focus on observing Pawel’s visible behaviors and outcomes of his work, we will only see a tiny part of it. After all, we have our work, too, let alone all the other people in the team that we need to take care of as well.
Curiously enough, the closest person to objectively assess anything about Pawel will be himself. He knows most about his trials and tribulations as well as triumphs. He knows when he thrives and when he struggles.
Except we intuitively dismiss his assessment as subjective.
Thinking: Fast and Slow
One of the fundamental observations Dan Kahneman made in his Thinking: Fast and Slow was that whenever our brain faces a difficult question, it subconsciously changes it to a similar albeit simple-to-answer one.
Another one is that we typically make snap decisions and only then look for arguments supporting our choice (also actively dismissing those that would go against it).
Couple these two with our assessment example and the challenge described above. The question about Pawel’s development skills is difficult. The most honest answer I can give is that I don’t really know, and the best I could do would be to spend a lot of time trying to inform myself better, but a) I don’t want to do this, and b) it would only improve my answer by a thin margin.
But here’s a simple question that I can answer instantaneously. What is my opinion about Pawel’s development skills? Yes, the change may look subtle, but it makes all the difference.
The new question doesn’t force me to consider facts, outcomes, and observable behaviors alike. It literally goes for my judgmental opinion, which, of course, may take data into account. It may also include all the prejudice, bias, hearsay, and other sources of misinformation.
Obviously, it is explicitly subjective.
Yet, our lazy brain answers the simple subjective question while pretending it deals with a difficult and more objective one. Oh, and once it has the answer, it follows up by justifying it with all the supporting arguments it can find.
All the efforts to make the assessment system “more objective” are then thwarted by our brains.
No wonder so many people consider their assessment systems unfair.
The Way Out
Following Buckingham and Goodall’s advice, one might ask what would happen if we ditched the pretense of objectivity and embraced subjectivity. While I can’t give a general answer, here’s a story of what happened at Lunar.
The starting point was that we had a 360 assessment in place. We designed it around observable behaviors. The measured satisfaction with the assessment (and, broader, the salary system) was a whopping 80%.
If it ain’t broken, don’t fix it, they say. So that’s precisely what we did. We redefined our categories of flexibility, experience, effectiveness, and people skills with straightforward and subjective questions, such as:
- I would always want [that person] to be my leader.
- When a team requires multiple organizational and technical skills, I would always want [that person] to be on the team.
- Whenever there’s a role no one is willing to take, I would always count on [that person] to take the responsibility.
- etc.
We answer them on a scale from “I strongly disagree” to “I strongly agree.” Also, the “always” qualifier is important as it stretches the answers across the scale.
Initially, there were about a dozen of such questions. We wanted to ensure that all the aspects of the existing solution are covered.
After another iteration of the “old” assessments, I suggested we experimentally try the new approach and compare the results. It should have been an easy sell.
That’s when all hell broke loose.
I didn’t appreciate how much resistance there would be against trying something new, even when it wasn’t supposed to affect anything. Not unless we would have validated the outcomes, at least.
The new approach was considered explicitly subjective (as opposed to the perceived objectivity of the old one). It was expected its outcomes would affect the fairness of the payroll. People feared we would lose a lot of sophistication and details in assessments as the new questions were necessarily broad and generic. For example, we stopped mentioning any specific technologies we work with. Then, there was concern that people would start playing their favorites (I like you, so sure, I would love you to be my leader).
The Experiment
No amount of discussion could get everyone on board, but at least I could play the “let’s try it” card. If we didn’t like the outcome, nothing would change, after all.
What have we learned?
It was easier to answer the questions. We got significantly more answers in the new scheme (up from 47% to 65% of all possible responses).
It also took much less time to answer the subjective questions than when we pretended to be objective (about a half time spent on the activity, despite providing more responses).
The best thing?
The results correlated almost perfectly with the old system. The correlation coefficient was 0.98. That’s math for “these series are as identical as it gets.”
We literally generated the same results (or, in terms of quantity, better) with half the effort.
Many still felt uncomfortable with a blatant admission that we use individual opinions as assessments. Nonetheless, the experiment’s outcome spoke for itself.
We have been doing it all along; we’ve just cultivated an illusion of objectivity.
Summary
Two years down the timeline, no one looks back. We reduced the set of questions to just five. And if I wanted to get super-radical, we could stick with just one: “I would always want [the person] to be my leader.” This single response correlates most with all the categories we used to assess.
When you consider it, it makes perfect sense. There are many interdisciplinary traits and skills we’d expect from an ideal leader. We’d want them to support us as a person and professional. We’d turn to them with problems, both technical and interpersonal. We’d look for guidance and challenge in them. We’d want them to make the team better. Be fair to everyone. And many more.
If someone does well on all those accounts (and more), it is only fair to expect them to shine in a traditional skill/trait-based assessment, too.
Still, we want to stress some other aspects of our work explicitly as well. But that still leaves us with only five questions, which, in most cases, we can easily answer from the top of our heads.
I won’t say that we somehow started loving assessments. No one does that. But we:
- get the same quality
- but better quantity
- by spending much less time on the activity
- and are similarly satisfied with the system
What’s not to love?
Subscribe RSS feed