Your Survey Is Lying to You

There is a famous line, usually misattributed to Henry Ford, about how if he had asked people what they wanted they would have said faster horses. Whether he said it or not, the point holds, and it is one that should give pause to anyone running a public consultation, local authority or even a small business.

People are very good at answering the question in front of them, and not nearly as good at telling you what they actually need. This is not a flaw in your residents or your service users. It is simply how human beings work.

We answer surveys with our stated preferences, reflecting the socially acceptable, top-of-mind version of ourselves. What we do in real life is governed by our revealed preferences, which are messier, more honest and frequently contradict the form we filled in last week.

A great example of this from a local business recently, a yoga studio, the owner reviewed feedback from customers, the resounding ask was a 7am class. The 7am class is made live, but the numbers are so low in practice the session has to be cancelled.

The gap between what we ask for and what we actually do is where a lot of good intentions, and a lot of council budgets, quietly go to die. I have seen the pattern more times than I can count, and I have a great deal of sympathy for the teams it catches out, because they are usually doing everything they were told to do.

An organisation runs a consultation, gets a respectable number of responses, and treats those responses as the truth. Residents say they want more evening events, so the events are programmed and almost nobody comes. People say they would use a new digital service, so it is built, and the phone lines stay just as busy. The consultation was not wrong, exactly. It captured what people said. It just did not capture what they would do, and those are two very different things.

The trouble is that a survey leans, by its nature, towards the answerable. It asks closed questions because closed questions are easy to count, and easy-to-count answers feel like they offer us more certainty. But the things that drive behaviour, the friction, the embarrassment, the habit, the thing someone would never write down on a form from the council, do not fit in a tick box. So they get left out, and you end up with a confident, well-presented report about a version of your community that does not quite exist.

For councils and service providers this matters more than it might first appear, because it erodes trust in both directions.

You consult, people respond, you act on what they said, the thing does not work, and now everyone is a little more cynical than before. Residents conclude that consultation is theatre. The organisation concludes that residents do not know what they want. Both walk away slightly more disengaged, which is the exact opposite of what the whole exercise was for. So what do you do instead?

You do not throw surveys out, they have a place for scale and for measuring things you already understand. But you do stop asking them to carry the whole weight of your project or requirement. The most reliable thing you can do is watch what people do, not only collect what they say. Where do people already gather, and what do they already do, when nobody is asking them to fill in a form.

A library that is full at ten on a Tuesday morning is telling you something no questionnaire will. A consultation event that three people attend is also telling you something, and it is rarely "we just needed to advertise it better." Go to where people are rather than waiting for them to come to a portal. The voices a survey reaches are, almost by definition, the voices already willing to fill in surveys. The people you most need to hear from, the time-poor, the wary, the ones who long ago stopped believing you were listening, are precisely the ones who never click the link.

Reaching them takes more effort and returns far more truth. When you do ask questions, ask open ones, in person where you can, and resist the urge to lead. "What would make this easier for you" tells you more than "would you use this new app, yes or no." The first invites a real answer. The second gently invites the answer you were hoping for. And then close the loop, because this is the part that builds the relationship for next time. The fastest way to lose people is to ask for their time, go quiet, and never tell them what changed.

People are remarkably forgiving when you cannot do everything they asked. They are far less forgiving about being consulted and then ignored. None of this costs more than running a survey and acting on the wrong conclusion, which is what a lot of well-meaning organisations are doing right now without realising it. It is simply more honest about how people actually behave, rather than how they describe themselves in a particular given moment.

The goal of listening was never to collect responses. It was to understand people well enough to serve them properly. A survey can absolutely be part of that. On its own, it tends to confirm what you already assumed, and that is expensive and dangerous territory to be in.

*Anoushka Emilee is co-founder and Communications Director of JAM Group Media.*