Customer surveys’ low rating
Handwritten responses and mailings have yielded to screen taps and AI analysis. Feedback is immediate, but critics question whether it’s reliable.
The polite but insistent phone calls started coming for Brian Wille not long after he purchased his Odyssey minivan at Darrell Waltrip Honda in Franklin, Tenn.
Was he satisfied with his purchase? Was the contract clear? Was he given a tour of the service area? Was the waiting room clean? Could he please rate these categories on a numerical scale — and please affirm that everything was excellent?
“As a consumer, I think it’s funny,” said Wille, who works as a banker. “They’re not getting good data because they coach you to give them a five. Nobody wants to give them a three because they don’t want to deal with the follow-up calls asking if they could have offered coffee instead of tea.”
Darrell Waltrip Honda’s aggressive post-sales outreach — legendary in its corner of Tennessee — is one example of the increasingly widespread corporate practice of hunting down customer thoughts and fixing them into a quantitative data set. Although some call it a valuable peephole into the thoughts of customers, others see customer service surveys as a breeding ground of bad data, awkward interactions, nervous employees and coached responses.
Technology has only supercharged both the ubiquity of the surveys and the lingering questions about their effectiveness. What was once a plodding exercise involving handwritten documents and postage stamps has evolved into a juggernaut that administers real-time ratings via screen taps and interprets them through artificial intelligence. Customer surveys are just one tool in the burgeoning field of “customer experience management,” which, as the consulting firm Gartner puts it, attempts to nurture “greater customer satisfaction, loyalty and advocacy.”
Medallia, a Menlo Park, Calif., company that’s one of the biggest in the multibillion-dollar business, estimates that it processes
2.5 billion pieces of data each month in its quest, among many other improvement missions, to help hotels find out whether customers liked the rooms and whether their check-in experience was pleasant.
“These companies have made it easy to send surveys really inexpensively,” said Martha Brooke, the founder of the Portland, Ore., company Interaction Metrics and a vocal critic of survey culture.
“It’s really a travesty — it’s gotten out of control,” she said. “It’s all written in this pleading language. The customer just wants to give them five stars and be on with it.”
The 1990s obsession with finely measured management — the culture of Six Sigma and the infamous “vitality curve” first used to rate the performance of General Electric employees — has found explosive new life in an era of smartphone apps and artificial intelligence data crawlers. Both the driver and the passenger on a ride-hailing platform can rank each other on a five-star scale, holding a small piece of each other’s fates. Food delivery apps can light up your phone with questions before you even bring dinner into the kitchen. Call-center workers have to deliver unsatisfying news to irate customers — say, explain why that refund they want isn’t available — and then sweetly ask them to comment on how much they liked being told “no.”
Can such data be trusted? Many business school professors are skeptical, and those who have studied the issue point to the natural psychological bias among customers who wish to be perceived as nice people and have no wish for a wage-earning employee to get into trouble on their account. So performance thought to be poor, mediocre or merely adequate is often rated as excellent — especially if the rating is sought when the employee is standing right there.
In an influential 2019 paper for the National Bureau of Economic Research titled “Reputation Inflation,” John J. Horton, Joseph M. Golden and Apostalos Filippas observed an online labor marketplace and noted a dramatic uptick in good performance rankings when those providing the data knew their answers were not confidential and could cost somebody a future job.
“As the potential to harm is what makes ratings effective, reputation systems, as currently designed, sow the seeds of their own irrelevance,” the authors concluded.
Companies, indeed, often seek qualitative performance data for internal personnel reasons — to calculate bonus structures, for example, and gauge whom to promote and whom to fire. The pressure to harvest anything less than top rankings can result in awkward conversations between employees and customers, even if the encounter is over the telephone. The fear of a bad survey, for example, pervaded the work environment at a company called Teleperformance in Shreveport, La., said former employee Adriana Bourgeois, who spent the majority of her $10-per-hour job taking calls for the billing department of AT&T and knowing that the customer had the power to destroy her quantitative scores as soon as she hung up.
“People are upset with the company, but the only way to express that is to rate the employee poorly,” she said. “It was the bane of everybody’s existence.”
In order to rig their own ratings, she said, employees learned certain tricks, such as framing a routine rebate given to every caller as an extra-special perquisite tailored just for them. “A lot of it was just kissing butt,” she said. (Teleperformance did not return a message seeking comment.)
Employees who can’t change corporate policy but must relay unfavorable news to customers are potential victims of their own surveys in another way. Companies might be using surveys as prophylactic shields against angry customers who might otherwise vent in public online forums, said Giorgos Zervas, a professor at Boston University.
“Before the customer goes public on Yelp, it allows the company to intercept them, in a sense,” he said. “I think, ‘Aha, somebody is listening.’ And that completely changes my perspectives.”
Zervas did his own study by watching how online hotel rankings changed when the managers posted responses to irate lodgers. The bad reviews got both fewer in number but lengthier in words, suggesting that the appearance of “listening” was likely to cut down on the number of brief angry bursts without defensible logic behind them.
But even customer venting done in a private inquiry, and expressed in a series of icy 1-to-10 ratings, could leave the company with a skewed sense of its image and the competence of its employees, Zervas said — especially when the subject of the survey, the front-line sales associate, is aware of the survey’s power and is in a position to help alter the results.
“If I give chocolate to my students and they give me better ratings, does it say anything about whether they learned anything in class? No. And that’s the problem with methodology that relies blindly on metrics,” he said.
Is there a way for companies to get better data from surveys?
Vivek Astvansh, an assistant professor at the Kelly School of Business at the University of Indiana in Bloomington, said part of the key is to shift the responsibility for the survey away from the person making the sale to avoid questions of survey-gaming and even potential bribery.
When he went to buy a car recently, the associate offered him free floor mats in exchange for a positive rating on the imminent survey.
“That made me realize there’s more going on here and it might be unethical,” Astvansh said. “It’s very hard for me to say ‘no’ to a human being, particularly if that person has serviced me. Asking me right then puts me under a pressure situation.”
Brooke of Interaction Metrics agreed that truly good data cannot be gleaned from rankings because the human factor implicit in a face-to-face encounter with a sales associate is going to rest on subtle matters of personal chemistry that defy expression with a five-star chart. “Mostly customers don’t think of you all that much,” she said. “They’re responding to you on all kinds of subjective levels. But they aren’t parsing or thinking much about ‘Why don’t I like that guy?’ The survey is at the level of rational reasoning, but most people don’t think that way. Nobody’s writing a thesis about you. Mostly they’re responding emotionally.”
Medallia, which has designed voice and text customer surveys for six of the world’s 10 biggest banks, offers a package of live-time surveys to its clients in which fragments of linguistic response from customers are picked over by an artificial intelligence program called Athena, which recognizes and interprets patterns within the narrative of a customer feedback form to highlight larger issues for the company’s attention. Certain words a customer might use — such as “clean,” “rude,” or “aisles” — can be plucked out and highlighted to attain a rough picture of crowd wisdom.
The software is designed to operate at high volume and without human intervention to curate the findings. “You can easily prioritize the biggest impact areas to take action on and identify which customer segments, regions and business units require the most attention,” the company says in a brochure.
Elizabeth Carducci, senior vice president for vertical market solutions, said the ease of completing surveys on the phone — either through screen taps or voice responses — makes it consequently easier for clients to give shorter surveys and tap into larger pools of data. What used to take three weeks with a mailed, handwritten survey can now take less than three minutes.
“We very specifically automate these processes,” she said. “To do this on an ongoing basis of scale means you need to automate everything. Three weeks down to three minutes or less. We can look what’s coming in and automatically rate how you’re reaching out to customers.”
The quicker and more painless the survey, the bigger the rate of response. Anything above 25% is considered a rousing success. Carducci recalled one client — a cruise ship line — that used to ask passengers to complete a 100-question survey with a pen before they disembarked. Medallia persuaded the company to shorten the list to 40 questions answered electronically, and the response rate shot up.
Historians of formal customer surveys trace their origin to the Institute of Social Research at the University of Michigan in 1946, where researchers first began asking Americans about their purchasing habits. The trade evolved into a humming profit center at the kitchen table of a Detroit accountant and marketer named James David Power III, who started sending out detailed questionnaires to car buyers in 1968. Writers from the automotive press took notice of some of the design imperfections Power identified merely through listening, and a spate of 1980s television commercials bragging about high rankings from J.D. Power & Associates helped freeze the name in the public consciousness as a mark of status.
Just as the film industry conceived of the Oscars — which also relies on the curation of a mass poll — as a selfpromotional tool for the film industry as a whole, the Initial Quality award from J.D. Power is enough to boost a brand name as well as legitimize the practice of customer service surveys as a forum of rigorous review.
Beyond the endless repetition of the name on television, part of the reason why J.D. Power commands enduring respect is the timeconsuming nature of the Costa Mesa company’s surveys, said Marc Cannon, vice president and chief marketing officer of Auto Nation and a three-decade veteran of the car business. Its surveys are administered not by sales associates but by third parties, removing the psychological bias.
Brooke agreed that the only known way to correct for potentially f lawed data in surveys is to go the expensive way — using a third party to contact customers in a live phone call after the transaction has come to a close, as well as supplementing the results with actions observed on the front line.
“If you want to know how the Marriott lobby is doing, position yourself to watch it for an hour in the morning,” she said. The aggressive outreach at Darrell Waltrip Honda used to be supplemented with a robocall with the folksy voice of the namesake NASCAR Cup Series champion, thanking the customer for their purchase and encouraging them to fill out the survey. But this had to be discontinued, General Manager
William Berryman said, because too many people were calling the dealership asking to speak to Waltrip, in the mistaken belief that he was putting in ordinary workdays.
Anyone who gets hired to work in sales for the dealership has the survey ethic drilled into them from the start, Berryman said. Calls are made to customers at regular intervals — two days, one month and six months — and there’s an expectation at Darrell Waltrip that those calls will be entered into the associate’s day planner. Both sales and service associates are expected to make the calls, and to let departing customers know to expect them. Berryman makes many of them himself. The practice brings in at least 300 completed surveys a month.
“We want the feedback,” Berryman said. “We want the good, the bad and the ugly.”
He contested the idea that survey results are used only for personnel reasons or that they don’t have an effect on how a business conducts itself. One pattern that he noticed in the surveys about the service department was a dislike of the shuttle van that ferried customers back and forth to their houses while the car was up on the rack. Berryman figured it would cost him only a little more to pay for Uber rides. The change has resulted in a distinct uptick in reported satisfaction.
“We do more than just walk through the motions,” he said. “It’s huge for us.”
Chris Scott, a geologist from Nashville, has bought at least five vehicles from Darrell Waltrip Honda over the last quarter-century and has no intention of doing business elsewhere. But he has grown so jaded by the surveys that he stopped responding to them years ago, especially after a technician told him that anything less than a perfect score was regarded as “terrible” inside the dealership.
“Their service is really quite good and they are not an otherwise high-pressure outfit,” he said. “But they sure want you to fill out that survey.”
‘It’s gotten out of control. It’s all written in this pleading language. The customer just wants to give them five stars and be on with it.’
— MARTHA BROOKE, a critic of survey culture