Garbage In, Garbage Out: The Danger of Misdirection in Customer Surveys

Article by: Rowan Jackson 

Market researchers report that response rates are down dramatically. They attribute this to increased survey fatigue, evidenced by more and more surveys not being completed or even started. Far too many surveys are at once competing for our attention and stockpiling in our inboxes. But there is another, less frequently acknowledged reason for plunging response rates.

Many customer surveys that come our way invite us to respond to questions that don’t apply to us and therefore seem irrelevant. This is a matter of poor design, rather than over saturation, resulting in bad participant experiences and inferior data. A vicious cycle indeed.

In the software field, the result of problem-laden products is characterised as “Garbage In, Garbage Out” (or GIGO for short); that is to say that incorrect or poor quality input will invariably produce faulty output. Developers stress how vital it is to start with the right data and ask the right questions of the customer from the outset to ensure that their needs and expectations are met.

The GIGO challenge is especially prominent in the world of customer surveys. A few years ago, we conducted a survey for an upmarket sports club. Its leadership team had created a questionnaire based on what they wanted to ask club members. Their questions were perfectly logical and might possibly have yielded some useful information had they been used. However, the CEO’s goal was to reverse the decline in membership and engagement within the club. When he suggested that they needed to be more customer-driven, we gently called attention to the fact that the questionnaire itself was not customer-driven, but rather it was management-driven. It consisted solely of questions that management deemed important to them, not what members considered to be priorities.

We took on the task of helping and began by gathering the expectations of club members using methods specifically fit for this purpose. A customer-driven list surfaced and with it the critical topics that had not occurred to the leadership team and therefore went unaccounted for on their own internal survey. When we redesigned the survey, the questions within it were based around the very expectations reported directly by members. Responses were high as the questions stimulated members’ emotional investment in the outcome, and a list of action areas soon emerged with five key priority targets.

The highest priority area for improvement concerned food and beverages, which came as a surprise to the management team. The issue was addressed quickly and of little cost, while resolving an underlying problem that had been driving members away. That, along with other key areas for improvement, would never have been uncovered at all if surveying had been left to the devices of management. Money would have been unnecessarily spent on making unilateral improvements which wouldn’t affect membership engagement. The key to creating a successful survey was member collaboration and gathering expectations data.

Another real-world example of GIGO in action was a global software development company that wished us to assess their performance against the expectations of their customers, which is work we often undertake. The president said that he and the management team were planning to base their strategic positioning in the coming year on the theme of innovation. They were astonished to learn that innovation never surfaced as an expectation of any customer. On the contrary, what customers stressed was that they wanted the company to deliver on time, to the required quality, and to budget. In other words, their customers wanted them to go back to basics. This renewed focus led to the company taking the right actions and continued rapid growth. Trust the data to give you the right answers.

Here are a few questions to help you avoid GIGO and the Error Mountain:

1. Are the questions in your customer or employee survey driven by what they tell you is most important to them (i.e. Expectations Data)?
2. Does your survey method identify, measure and prioritise the areas for improvement so you have a clear focus that will drive targeted investment?
3. Do you use an independent external organisation to create and administer the survey to help you avoid the dangers of inherent bias as outlined in our first example above?

If the answer to any of the above questions is ‘no,’ contact us below for more information on how Promising Outcomes can help you gather, analyse and act upon state-of-the-art customer and employee data.

Explore Expectation-Based Insights for Your Organisation