Good anecdotes gone bad: Surveys
A few months ago, we wrote about the power of anecdotes (Anecdotes all around you — put them to work), describing how stories engage audiences more effectively than stats and leave more lasting impressions than data. Since then, we’ve been reminded of one setting in which anecdotal information can get in the way: the survey.
We’ve noticed this with a number of organizations. Good for-profit and not-for-profit organizations occasionally plan surveys to learn what their customers, employees or other audiences think. When the surveys are conducted and the results are in, a team reviews the data, crunches the numbers and recommends changes that will address problems or build on strengths.
So far so good. Unfortunately, too often, just as this process is winding up, someone on the board or in the front office gets a copy of the survey report and reads the responses to open-ended questions … and everything gets messy. Why? Because they read a few compelling individual concerns or complaints and they want to “fix” them.
As a result, an organization, hypothetically, might learn that 90 percent of employees love the soap in the restrooms, but one person got a nasty rash from it, another gets sick when she smells it, and another thinks it’s ridiculous that the company is spending so much money on soap when employees could just bring their own (admit it: There’s one in every office).
OK: That’s an exaggeration, but we have seen plenty of situations in which the majority of respondents are happy with a policy or program, but one or two comments grabs a leader’s attention enough to make him or her wonder if some companywide change is in order.
While this individual concern is admirable (better to have that problem than have to deal with leaders who don’t care about people), it can throw one heck of a monkey wrench into a carefully crafted process. In fact, as we work with organizations planning to conduct surveys, we often suggest that they avoid open-ended questions or, if they feel such questions are necessary, that they not let their leaders see the answers.
All of which begs the question: Should open-ended questions even be included on surveys? On a limited basis, maybe, because they can help provide context for trends revealed in data. But they should not be a big part of your survey, because they seldom lend themselves to quantification and metrics (that is what surveys are for) and they too often distract from the real information.
So, does this contradict what we wrote back in January? No. In fact, we would suggest that it proves that anecdotes are far more powerful than stats. As a result, once you have your survey results and you’re preparing your report, definitely quote respondents’ comments that support the prevailing results, but — after careful and sensitive review — downplay comments that contradict a survey’s finding.
