Cutting through the noise
Astudy of 1.5 million cases found that when judges are passing sentences on days following a loss by the local city’s football team, they tend to be tougher than on days following a win. The study was consistent with a steady stream of anecdotal reports beginning in the 1970s that showed sentencing decisions for the same crime varied dramatically — indeed scandalously — for individual judges and also depending on which judge drew a particular case.
A study at an oncology centre found that the diagnostic accuracy of melanomas was only 64 per cent, meaning that doctors misdiagnosed melanomas in one of every three lesions.
When a large insurance company, concerned about quality control, asked its underwriters to come up with estimates for the same group of sample cases, their suggested premiums varied by an eye-popping median of 55 per cent, meaning that one adjuster might have set a premium at $9,500 while a colleague set it at $16,700.
If employers rely on only one job interview to pick a candidate from among a similarly qualified group, the chances that this candidate will indeed perform better than the others are about 56 to 61 per cent. That’s “somewhat better than flipping a coin, for sure but hardly a fail-safe way to make important decisions,” Daniel Kahneman, Olivier Sibony and Cass R Sunstein explain in this tour de force of scholarship and clear writing.
These inconsistencies are all about noise, which the authors define as “unwanted variability in judgments.”
Sometimes we treasure variability — in artistic tastes, political views or picking friends. But in many situations, we seek consistency: Medicine, criminal justice, child custody decisions, economic forecasts, hiring, college admissions, fingerprint analysis or business choices about whether to greenlight a movie or consummate a merger.
Despite its prominence in so many realms of human judgment, the authors note that “noise is rarely recognized,” let alone counteracted. We are living in a moment of rampant polarisation and distrust in the fundamental institutions that underpin civil society. Eradicating the noise that leads to random, unfair decisions will help us regain trust in one another.
Despite the authors’ intimidating academic credentials, they take pains to explain their various categories of noise, the experiments and formulas that they introduce, as well as their conclusions and solutions.
Some decision hygiene is relatively easy. “Occasion noise” — the problem of a judge handing out stiffer sentences depending on whether a favourite sports team won or lost — can, like bias, be
recognised during NOISE: A Flaw in
a “noise audit” and Human Judgment
presumably dealt Author: Daniel with. “System Kahneman, Olivier noise,” in which Sibony and Cass R insurance Sunstein adjusters, doctors, Publisher: Harper project planners or Collins business
strategists assess ~699 the same facts
454 with that
unfortunate variability, requires a more energetic decision hygiene. However, as the authors point out, the steps of decision hygiene “can be tedious. Their benefits are not directly visible; you might never know what problem they prevented from occurring.”
Acompellingexampleof “decomposing”adecisioninvolvesacase studyofacorporatemerger.ratherthan thebankersandexecutiveteamgivingthe company’sboardtheusualproorcon presentation,theceofirsttaskedvarious seniorexecutivestocomeupwiththeir assessmentsonsevenaspectsofthemerger,rangingfromtalentoftheteamtobe acquiredtothepossiblefinancialbenefits. Importantly,therewereseparateteams workingoneachaspect,sothattheir judgmentwasnotcolouredbypositiveor negativenoiseemanatingfromanother verdict,fallingintothetrapofwhatthe authorscall“excessivecoherence.”
The authors are sensitive to the costs of noise reduction, a point they illustrate in part with the story of the company that tangled itself up in an annual employee review process that included an overly complicated feedback questionnaire. Forty-six ratings on eleven dimensions for each rater and person being rated is just too much.
Beyondbureaucracyandcost,there’sa lossofdignitywhenpeoplearetreatedlike numbersinsteadofindividuals.thinkof Jackwelch,theformerceoofgeneral Electric,whomadeitasetpracticetofirea percentageofhislowestperformerseach year,evenifmanywerestillperforming well.inothersituations,theopposite approachcancreateproblems:rating everyoneindividuallywithnocomparisons,suchasthestandardsthatallowover 98percentofthefederalcivilservantwork forcetobejudged“fullysuccessful.”
The trick is finding the right balance, not looking for perfect fairness or accuracy, which will always be illusory. A digital body scan examined might be an efficient way to check for melanoma, but I’d rather trust the terrific doctor who checks me every few months. Then again, I wouldn’t mind if he checked his conclusion against the algorithm.
Noise is about how our most important institutions can make decisions that are more accurate and credible. That its prescriptions will not achieve perfect fairness and credibility, while creating pitfalls of their own, is no reason to turn away from this welcome handbook for making life’s lottery a lot more coherent.