3 problems with annual engagement surveys

by Matt Grimshaw
December 27, 2016

Doing an annual engagement survey is now HR orthodoxy, which in our opinion is exactly why you should be looking for ways to do it differently or better than your competitors.

Here’s a few challenges with annual surveys that might get you thinking of ways to improve your approach…

1. Nothing else in your business moves this slowly!

Can you think of anything else in your business that you measure just once a year? Sales? Profit? Quality? Customer feedback?

Imagine you suddenly switched to measuring customer feedback just once a year, how do you think your managers would react? My bet is that they’d conclude that delighting customers was no longer important to the business.

The danger with annual surveys is that they encourage managers to think employee engagement is ’non-core’ and critically, that it’s something that the organisation manages in a way that’s detached from day-to-day business. If it’s measured and discussed once a year, then employee engagement goes in the same bucket as CSR and sustainability. Nice things to do when we’ve finished with the real work of making money.

Companies with high performance cultures take the opposite view — they believe that sales result from happy team members not the other way round.

2. Managers aren’t equipped to improve their scores

Engagement surveys are a good way of identifying relative performance. As a manager you can compare your results to your peers and you can track your own performance over time. In other words, you can judge whether you’re doing ‘better’ or ‘worse’.

To make this information useful you need to be able to make meaningful comparisons.

The most meaningful comparison will be between your current performance and your past performance. In this context, the challenge with annual surveys is obvious — you don’t have many data points. Even an experienced manager who has been in role for 5 years will only have 5 sets of results to look back on. If this manager had access to a monthly measure, they’d give themselves the chance to learn twelve times faster.

The other reference point for managers should be an internal benchmark. In our view, most organisations make the mistake of encouraging their managers to compare their scores to the company average. It’s far more useful for managers to compare themselves with the company’s pioneers (typically the top 5-10%), particularly if the organisation understands what these pioneer managers are doing differently, why and how their approach can be replicated.

If the average or under-performing manager can replicate the approach of the pioneer manager then engagement in their team will improve. But again, the challenge with a single annual measure, is that it denies this manager the opportunity to improve in an iterative evidence based way. You can transform the operational effectiveness of your annual engagement survey if you also offer managers the ability to choose a small number of specific survey items that they can measure on a daily, weekly or monthly basis. This allows them to focus on implementing specific changes and to measure whether they are having the intended effect.

If you buy your engagement survey from a big consultancy, the final source of comparison is to reference their external benchmark. If you want your managers to do this, be clear on why you think this is meaningful and useful for your managers, and always ask your consultancy provider which companies make up the benchmark dataset. If it doesn’t include all your major competitors, we’re not sure why you’d bother…!

3. Managers start gaming the system

In his book, Thinking Fast and Slow, Daniel Kahneman cites an experiment on life satisfaction surveys as an example of ‘substitution’. Substitution is when people automatically but subconsciously substitute a difficult question they don’t know the answer to, for an easier one they do know the answer to.

“How satisfied are you with your life?” is a tough question. To answer it properly, you ought to spend some time reflecting on your whole life, allocating relative weighting to past experiences and perhaps also considering what your future might hold.

Given we are all naturally disposed to mental laziness, it’s no surprise then, that when most people are asked to rate their life satisfaction, they actually tell researchers the answer to an easier question like “How happy are you this month?”.

We think annual engagement surveys are almost certainly susceptible to the same flaws. If once a year you ask employees a question like “Does your manager do a good job of recognising performance?” it’s very unlikely the person will reflect back on the whole year or how they’ve seen their manager recognise the performance of other team members. Instead they’ll answer on the basis of the most readily available information to them (availability bias) which is usually their last interaction with their manager.

As a rule of thumb, we would argue that engagement survey questions are likely to be answered on the basis of an employees’ previous 3 or 4 weeks experience.

We’ve seen instances where this limitation rewards wily managers for really cynical behaviour. A month before the survey, these managers make a point of engaging positively with their team, making a fuss over recognition, giving out freebies and prizes and postponing difficult conversations. They’re rewarded with a spike in engagement that coincides with the company’s engagement survey. Then they go back to normal. Their engagement results look fine and they avoid scrutiny from senior managers, but it’s a false positive that masks underlying disengagement and poor management.

Enjoyed this blog? Sign up to our monthly newsletter to receive more great content just like this.