RETHINKING CROWDSOURCING
Social ties can skew results. Illustrations by Raj Verma
WHEN the Swiss soft drink company Rivella was looking to launch new flavours in 2012, it used an open innovation platform to ask consumers for ideas and received 800 responses. As managers sorted through them, they noticed that one in particular – for a health-oriented ginger-flavoured drink – appeared to be extremely popular. But on closer examination they saw that much of the buzz around it was coming from just a handful of participants who were working feverishly to elicit votes and comments. “It was a very small group of consumers who were rallying one another and generating a lot of noise,” says Silvan Brauen, who oversaw Rivella’s innovation pipeline. Despite the strong online feedback, the company concluded that the ginger flavour would flop in the market and abandoned the idea.
That buzz is an example of social bias, and new research shows that it’s a hazard companies should be aware of when tapping into consumers’ opinions during crowdsourced innovation exercises. To understand how social bias can skew results, Reto Hofstetter, a marketing professor at University of Lucerne in Switzerland, studied 87 crowdsourcing projects posted by 18 companies during a 14-month period on Atizo, one of the leading European open innovation platforms, which is routinely used by companies, including BMW and Nestlé. In all, Hofstetter’s team examined 31,114 ideas submitted by 1,917 consumers.
A key part of the Atizo system is a process for consumer voting. The companies in the study received 358 suggestions, on average; sorting and evaluating large numbers of suggestions requires significant management resources. To make that task easier, Atizo lets consumers “like” and comment on others’ ideas, much in the way that people can like posts on Twitter and Facebook. And the likes and comments are influential: Every company the researchers studied used the voting system as a first screen to help them judge ideas and decide which consumers to reward for their submissions.
As Hofstetter examined the voting system, however, he discovered that it wasn’t as meritocratic as it appeared. As often happens on social media, when someone liked an idea, that idea’s progenitor tended to reciprocate, liking an idea the other person had submitted. What’s more, Atizo has a mechanism that lets users “friend” one another, and the researchers discovered that users were far more likely to vote for online friends’ ideas than for those submitted by people with whom they had no connection. The data showed that these social biases had a lot to do with which ideas received the most votes and comments – but when the researchers spoke with the companies, they learned that the firms were unaware of that fact. “I didn’t see any evidence that the companies were de-emphasising the likes,” Hofstetter says. “On the contrary, the likes played a very large role in informing their decisions about which ideas to reward and develop.”
To further probe whether consumer voting holds real predictive value, the researchers conducted interviews and surveys at companies more than a year after the brainstorming was completed to see how things had panned out. They gave managers a randomly ordered list of the crowdsourced ideas each had received and asked them to rate each one in response to the statement “This idea was useful to implement, or this idea had a great impact on the success of an innovation.” The results showed no correlation between the ideas consumers preferred and the ones that led to successful products.
Seeking to understand the disconnect, the researchers had 145 outside evaluators independently rate each of the crowdsourced ideas for feasibility, originality, and customer benefit; they then
compared the evaluators’ ratings with the results of the crowdsourced voting. They found that consumers undervalue feasibility and overvalue moderate originality, whereas firms prefer feasible and either highly original or very common ideas.
The bottom line, the researchers write: “Online consumer votes are unreliable indicators of actual idea quality.”
That’s not to say that crowdsourcing isn’t a useful technique. But the study suggests that firms should look beyond likes, comments, and other signs of consumer preference and find moreeffective ways to evaluate the ideas that are generated. Hofstetter and his colleagues cite prior research showing that product developers who identify “idea hubs,” or similar ideas submitted by different people, may get better results. One platform replaced simple “likes” with more-complex evaluative questions, which reduced the incidence of reciprocal voting. And the researchers suggest that platforms develop algorithms or other mechanisms to control for social ties.
Adrian Gerber, the CEO of Atizo 360°, agrees that companies should pay less attention to consumer voting and give more weight to their own criteria when winnowing hundreds of ideas down to a handful of viable ones. As crowdsourcing evolves, he says, companies are beginning to show a preference for a more carefully selected crowd, one that brings special expertise to the innovation puzzle. For instance, they are increasingly using open innovation platforms with groups of employees or suppliers rather than consumers. Some firms also seek out specialised consumers whose views may be especially relevant. To brainstorm new kinds of avalanche protection gear, Mammoth, the Swiss outdoor clothing and equipment company, asked Atizo to convene an online group of mountaineers who have design or engineering experience. As that example suggests, the right answer may come from tapping a small number of the right people rather than from polling a crowd of random idea generators.
FIRMS SHOULD LOOK BEYOND LIKES, COMMENTS, AND OTHER SIGNS OF CONSUMER PREFERENCE AND FIND MOREEFFECTIVE WAYS TO EVALUATE THE IDEAS THAT ARE GENERATED