CU Boulder scientist says data “fatally flawed”

New research from a University of Colorado scientist argues that the data used to support new regulations requiring some female athletes to medically lower their testosterone is “fatally flawed.”

The International Association of Athletics Federations announced new regulations in April that require certain female athletes with higher levels of testosterone to medically reduce that level, or they won’t be allowed to compete.

The association published a paper in the British Journal of Sports Medicine in 2017 claiming that elite women runners with high testosterone levels performed as much as 3 percent better than those with lower levels.

A paper from Roger Pielke Jr., director of the Sports Governance Center at CU, published Friday found significant errors in the data that the association used in its research.

Pielke, who authored the paper along with Erik Boye, a professor emeritus at the University of Oslo, and Ross Tucker, a University of Cape Town exercise physiologist, said their paper doesn’t discuss gender and testosterone levels, but rather talks about scientific integrity.

“Before you can even get to the point of discussing and debating testosterone in female athletes, we believe that the scientific base should be solid,” Pielke said. “The evidence isn’t even enough to begin to have that discussion.”

Pielke will be an expert witness at the international Court of Arbitration for Sport in Switzerland later this month, where South African Olympic sprinter Caster Semenya is challenging the new regulations. The hearing is not open to the public.

The first issue with the study that Pielke and his colleagues ran into was a lack of transparency. Normally, Pielke said that scientists share data with colleagues after they publish a paper.

The association had both performance data and medical data. Pielke requested the performance data, as he expected the medical data to be confidential, but said it was difficult to even get that.

“Finally, they gave us 25 percent of what we asked for,” he said.

They then replicated the association’s statistics, and found that their results were very close. But they ran into issues when trying to recreate the data set.

Pielke said they tried using publicly available information on performances of female athletes. They found that the association’s data set sometimes used the same woman twice, included disqualified athletes who had been doping, copied performances more than once and also held data that Pielke couldn’t find anywhere

else — what he calls “phantom times.”

The three researchers found that 17 percent to 32 percent of the data was erroneous, which is “a big number if you’re doing research that only has 1,000 data points,” Pielke said.

In response, the association corrected some of the errors. Still, though, Pielke said they found errors when they replicated the statistics.

The results from the corrected research didn’t match up with the association’s initial findings, which “tells you that the data errors were actually important.”

“All of this is such a scientific mess ... In most settings, if you’re gonna be implementing policies, you’d say, ‘Let’s clean this up,’ ” Pielke said.

Bad science leads to bad policies and regulations, he added.

This instance is important, Pielke said, as sports begins to rely more on science for detecting those using drugs.

“It forces us to ask, do we really want sport to have solid and robust science? Because, if we do, then it can’t hide its data, it can’t do it in-house,” he said. “It seems to me that the sport community is not quite ready to adhere to the 21st century and the standards we uphold.”

Pielke said the sports community needs to do two things if it wants to collect data to guide regulations. First, it needs to hire a third party to do the research. The association used its own researchers for the report, which Pielke said wouldn’t be allowed in other industries.

Second, he said that any of its research needs to “follow the basic norms of science.” The data needs to be shared so that others can try to replicate it, and then “let the chips fall where they may.”

The association has grappled with the issue of gender for a decade, Pielke said, and issues surrounding gender verification date back a half-century.

In 2011, the association tried to get similar regulations in place, but the Court of Arbitration for Sport found that there was a lack of evidence to support them