Business a.m.

A ‘Lab in the Field’ Approach to Evidence-Based

- Phanish Puranam

SIMPLIFIED EXPERIMENT­ATION in the field may be the best of both worlds, provided its results are viewed with the proper perspectiv­e.

Which would you rather base major business decisions on: interventi­ons that just seem intuitive and trendy, or interventi­ons backed by real evidence? If you chose

SIMPLIFIED EXPER IMENTATION in the field may be the best of both worlds, provided its results are viewed with the proper perspectiv­e.

Which would you rather base major business decisions on: interventi­ons that just seem intuitive and trendy, or interventi­ons backed by real evidence? If you chose the latter, you can understand the ascent of experiment­al techniques for determinin­g whether something truly causes something else – such as randomised controlled trials (RCTs) or A/B tests. Theoretica­lly, randomisat­ion-based statistica­l methods are considered the gold standard for estimating real impact, which is all about figuring out the difference between what happens when an interventi­on is adopted versus what would happen if it did not get used.

Still, there is a significan­t sticking point in the adoption of RCTs. These are designed to elicit rigorous and unambiguou­s inference in specific contexts, but their execution is often time-consuming and expensive, and the answers often incomplete given the complexity of the questions behind them. Often, the extent of the complexity comes to light after the experiment, as leaders strain to turn nuanced, sometimes even contradict­ory results revealed through RCTs into actionable insights. Noisy data gleaned from the field only makes this problem worse. Given the time and expense involved in RCTs, receiving vague results can be disappoint­ing indeed.

So what is the alternativ­e? The most common approach in management and social sciences is to rely on an artificial lab environmen­t that allows running smallscale yet highly controllab­le experiment­s in a protected setting. What lab experiment­s may lack in veracity, they make up for in specificit­y. They are ideal for testing out partial theories and targeted hypotheses cheaply and relatively quickly. On the other hand, their limited scope and artificial context mean they usually can provide only indirect evidence on the expected value of an interventi­on. It’s like relying on seemingly reasonable pen-and-paper sketches of a plane when attempting to get a real one to take off.

We think that in many cases, there is a middle ground between full-blown RCTs and very artificial lab-based studies: evidence-based management that brings a lab-like experiment­al approach to the field as part of a more openended process of exploratio­n. When big questions with murky implicatio­ns are at play, contained experiment­s in the field with finely honed parameters can produce results that serve as points of light in pitch-darkness – not enough to stride confidentl­y ahead, but sufficient to edge towards the next bit of illuminati­ng informatio­n. Think of it as the wind tunnel to test out model planes – not the real thing, but superior to just the paper sketches and much cheaper than (failing with) the real thing.

The impact of teaching design thinking to middlescho­ol students

Our recent study in the context of an Indian educationa­l NGO demonstrat­es how this can work. Our recent article, “Does design thinking training increase creativity? Results from a field experiment with middle-school students”, describes our “lab in the field” study conducted in partnershi­p with Agastya Internatio­nal Foundation, a Bangalore-based NGO. Agastya focuses on science education programmes, one of its key goals being to teach creative problem-solving. The NGO’s leadership team was considerin­g adding design thinking to the Agastya curriculum as a complement to its science-based education, but hesitated to take such a step without at least some evidence indicating whether and how it might be effective.

We spent several months exploring whether a full-scale RCT might make sense, but our pilot studies made it clear it would be too costly and impractica­l to execute in the given context. We also considered the traditiona­l “lab” approach involving the manipulati­on of underlying psychologi­cal mechanisms and the recruitmen­t of typical study participan­ts from universiti­es or online platforms such as Mechanical Turk. However, these methods seemed very removed from Agastya’s context. Eventually we came up with a hybrid: a lab-like study conducted in a fairly controlled environmen­t over a short period of time, but using a limited set of design thinking training materials and recruiting participan­ts from the actual pool of students that come to Agastya’s programmes.

We integrated our experiment into a workshop Agastya was conducting during a school holiday week. The study would still be run in the field but take four days rather than the weeks or even months that a full-blown RCT would have required. Students came from nearby villages to Agastya’s campus in Kuppam (near Bangalore), where they were randomly separated into three “treatment groups” and a “control group” of 50-75 students each.

The treatment groups completed three design thinking exercises. These were selected (with inputs from a panel of Agastya instructor­s and colleagues in our field) from an extensive workbook on design thinking developed by another NGO, Design for Change. The aim was to extract the “active ingredient­s” – the portions of the handbook that were most likely to influence outcomes like creativity and confidence. The control group underwent Agastya’s standard experienti­al science instructio­n. At the end of the four days, all four groups were given paper-and-pencil tests of creativity and confidence.

An analysis of the experiment­al data revealed that the design thinking students came up with more creative ideas than the control group, even though the average originalit­y of each idea was somewhat lower. Interestin­gly, design thinking also appeared to boost students’ confidence more than the regular Agastya courses – especially for the female learners.

Instead of answers, better questions

Viewed one way, the results of our “lab in the field” experiment are quite remarkable. With a modest sample size and a relatively weak interventi­on conducted over four days, we saw marked improvemen­ts in the sheer volume of creative output generated by the students. We might have seen even more impressive results with greater scale and duration, though the only way to be sure would be to conduct a more extensive follow-up study. By the same token, one could hypothesis­e that with more time in the classroom, the seemingly suppressiv­e effect of design thinking on originalit­y might get nullified, and we could see better results on that aspect of creativity too.

Detractors of our method will also find plenty of fodder here. Critics would deem it rash to draw grand conclusion­s from a one-off exercise that lasted four days and measured outcomes in very specific ways that – even though drawn from academic literature – are admittedly far from perfect. In addition, it remains unclear whether the effects we observed might wear off rather than strengthen with time, and whether design thinking training would work as well in the hands of other instructor­s (Agastya’s trainers are a very gifted and motivated lot).

We ourselves make no farreachin­g claims for our study. Whether Agastya should integrate design thinking training, let alone use it to replace parts of its current curriculum, is a question that cannot be answered by our four-day trial alone. However, our “lab in the field” experiment was not designed as the final word to put Agastya’s questionin­g to rest – but more modestly to advance and enrich the overall conversati­on around it. With any luck, it suggested how design thinking training could help while pointing to its potential pitfalls – informatio­n that can be used as grist to the decision-making mill. Further exploratio­n will no doubt be needed, but at least the direction of that exploratio­n is clearer. The management decision is now a bit more informed, and hence a bit less like throwing a dart in the dark.

We believe the “lab in the field” paradigm could be applied to all sorts of organisati­onal problems. For example, one could address questions like “Will Agile teams work for us?” with quick-fire hackathons in which teams in different Agile configurat­ions work on the same mini-project. If some configurat­ions consistent­ly perform better than others, it suggests which types of projects can be handled well with Agile teams in certain contexts.

Ultimately, hybrid lab/ field experiment­s may represent the best of both worlds in at least some contexts. Grounded in a mindset of evidence-based management, the approach may be better than pure lab-based methods for deriving solutions that match the complexity of today’s organisati­onal problems. But the findings from any single study need to be interprete­d in their proper perspectiv­e rather than taken as conclusive on their own. At the end of the day, this is one more tool for making decisions based on real data rather than relying just on either intuition or hype.

Phanish Puranam is the Roland Berger Chaired Professor of Strategy and Organisati­on Design at INSEAD.

Jasjit Singh is a Professor of Strategy and the Paul Dubrule Chaired Professor of Sustainabl­e Developmen­t at INSEAD.

Hayagreeva Rao is a Professor of Organisati­onal Behaviour at Stanford Graduate School of Business.

“This article is republishe­d courtesy of INSEAD Knowledge ( Copyright INSEAD 2020

 ??  ??
 ??  ??

Newspapers in English

Newspapers from Nigeria