Courtroom psychology tests may be unreliable

Courts are not properly screening out unreliable psychological and IQ tests, allowing junk science to be used as evidence, researchers have concluded. Such tests can sway judges or juries and influence whether someone gets custody of a child or is eligible for bail or capital punishment.

The scientists looked at hundreds of different psychological tests used in recent court cases and found that a third of those exams weren’t reviewed in the field’s most prominent manuals. Of those that were reviewed, just 40% were graded favorably. Nearly a quarter were deemed unreliable.

“There’s huge variability in the psychological tools now being admitted in U.S. courts,” said Tess Neal, an Arizona State University psychology professor and co-author of the study published Saturday in the journal Psychological Science in the Public Interest.

“There’s a lot of stuff that looks like it’s junk and should be filtered out by the courts, but it’s not being filtered out,” said Neal.

Legal challenges to the validity of psychological tests occurred in less than 3% of cases, the researchers found.

“This paper is highly significant, in part because many people’s fates are determined by these tests,” said Dan Simon, an expert on law and psychology at the University of Southern California Law School, who was not involved in the research.

The new study is not the first critique of how science is used in the courts.

In 2009, the National Research Council released an extensive report on courtroom science that found that “testimony based on faulty forensic science analyses may have contributed to wrongful convictions of innocent people.”

The critique prompted calls for reform, and only partial progress has been made, Simon said.

“Courts are supposed to sift out the junk science from the good science, as laid out in the federal rules of evidence” — a set of national guidelines that require that “testimony is the product of reliable principles and methods.”

“But that’s not happening,” Simon said.

The new study examined 876 court cases in the U.S. between 2016 and 2018, and found the most commonly used psychological test was the Minnesota Multiphasic Personality Inventory, which has generally positive reviews in the professional literature.

The second most common was the Rorschach test — sometimes colloquially called the inkblot test. While the test, first developed in 1921, has its defenders, some scientists regard it as dangerously ambiguous and subjective.

Robert MacCoun, a professor of law and psychology at Stanford University who was not involved in the study, said that he’s received unsolicited catalogs advertising new psychological tests from vendors for many years. Those brochures used to include data about test effectiveness, but “by the end of the 1990s those numbers had disappeared.”

Lawyers and judges, who are not experts in testing methods, must rely on the expertise of psychologists to perform due diligence on tests they present as evidence, said Harvey Fishbein, a criminal defense attorney in Manhattan who was not involved in the study.

“If psychologists are not willing to regulate their own field, it’s a real problem.”