Understanding terms when interpreting study results

DEAR DR. ROACH >> In a recent column discussing the use of aspirin, you made reference to the results of a study that were neither “statistically significant” nor “clinically meaningful.” While I am quite familiar with the concept of statistical significance and believe it is very helpful in interpreting the results of a study, I am not familiar with the latter concept. Are you referring to a poorly designed study or one with too small of a sample? I am wondering whether “clinically meaningful” adds any benefit to the scientifically accepted concept of statistical significance. — P.J.B. DEAR READER >> Statistical significance is a concept central to understanding medical or other scientific studies. Often, one group (an experimental group, who may get a new treatment, for example) is compared with another group (the control group, who got some other treatment, usually the standard treatment or, if there is no standard treatment, a placebo). The difference in the outcome is looked at between the two groups.

A statistician employs one of several methods to calculate the likelihood that the difference between the two groups could have happened by chance, called the p-value. If the p-value is less than 5 percent, then that is usually considered statistically significant. The lower the p-value, the less the likelihood that the observed difference between the two groups could have happened by chance if the two treatments were identically effective.

Clinical significance, or clinical meaningfulness, refers to the real-world usefulness of the intervention. While the term “clinical significance” is subjective and therefore an opinion, the term nevertheless is useful. For example, with a very large trial, a small difference in effectiveness between the two treatments could have a very significant p-value, as low as 0.0001, with only a 1 in 10,000 chance that the two treatments are equivalent. However, the effectiveness may be 50.1 percent in one group and 49.9 percent in the other group. Although statistically significant, the clinical significance is marginal.

If a result is not statistically significant, it cannot be considered clinically meaningful, even if the difference is large, because there is not enough evidence to reject the hypothesis that the difference could have occurred by chance.

It is logically impossible to prove the absence of a difference between two treatments. All we can do is show that it is unlikely that the two treatments are identically effective. Unfortunately, this means that concepts long regarded as true sometimes can be upended by new information.