CAD software-What's wrong with this picture?
Detection systems draw mixed reviews in radiology
Early detection” has long been the buzzword in cancer care. So when computer-assisted detection, or CAD, software hit the marketplace 12 years ago, manufacturers and providers alike envisioned a day when the technology would become so sophisticated it might all but eliminate the possibility of missing an early cancer.
Today CAD, which uses computer-generated markings to draw a radiologist’s attention to questionable areas on an image, is far from being flawless in its function or ubiquitous in its use. But a growing number of provider organizations are making the technology part of their image-reading options and protocol.
According to a February 2005 study published in the American Journal of Roentgenology, 10% of mammography facilities in the U.S. had adopted CAD technology within three years of the U.S. Food and Drug Administration’s approval of the first CAD device in 2002.
“A lot of the CAD systems are acting as an assistant to radiologists—especially in breast imaging—so that we can accurately assess the image,” say Gillian Newstead, a professor of radiology at the 548-bed University of Chicago Medical Center.
CAD currently is being used in breast mammography, lung imaging and colon-cancer screening, but despite its promise, many proponents of the technology say its development has stalled in recent years, and available CAD systems haven’t kept pace with the imaging modalities with which they are paired.
“I did an evaluation of mammography CAD systems in 2002, and since then the technology’s underlying algorithms haven’t changed at all,” says Jason Launders, medical physicist and senior project officer with the healthcare-technology analysis group ECRI Institute based in Plymouth Meeting, Pa.
Other industry experts express similar concern about the state of CAD technology. “There have been no new CAD device applications cleared by the FDA since 2006,” says David Fisher, managing director of the Medical Imaging & Technology Alliance, a group representing radiology equipment manufacturers. “We estimate that between 10 and 20 CAD applications have gone to the FDA, but none have been cleared.”
Fisher declined to say which companies have submitted CAD products for FDA review within the past four years, but he says he believes that manufacturers’ and FDA reviewers’ confusion over the premarket approval process is to blame for the lack of product approvals.
Now some observers are worried that the FDA’s proposed new guidance on the CAD technology approval process could make getting new and updated products into the hands of providers nearly impossible. They say added requirements for clinical studies and design verification will make CAD development too costly, particularly since the technology provides limited reimbursement opportunities for providers. Mammography CAD is the most widely covered form of the device’s use, with Medicare and most private payers reimbursing providers for CAD services. Lung CT and colon-screening CAD is less often covered.
“These can be expensive add-on software systems to buy,” Launders says. “It can cost $50,000, so it’s the kind of thing hospitals knock off the order when they’re looking to buy new imaging technology.”
Throwing down a new gauntlet
In October 2009, after two years of work by a radiological device advisory panel, the FDA released proposed guidance for its CAD technology approval process. Contained in two documents, the draft guidance is the agency’s first attempt to standardize and clarify the CAD product-approval application process.
“The goal is to let industry know what we hope to see in an application and what criteria we need them to meet,” says Nicholas Petrick, deputy director of the FDA’s Center for Devices and Radiological Health’s science and engineering laboratories office.
The first of the two documents details proposed new rules for the 510(k) submission process—which covers CAD product updates, moderate-risk devices and new products that are equivalent to already approved CAD technology. The second document directs manufacturers on how to design and conduct their clinical-performance assessment for both 510(k) and new, high-risk CAD products approved under the pre-market application, or PMA, process.
“We think the process they’ve put forward is excessively burdensome,” Fisher says of the draft guidance.
Under FDA regulations, companies are required to provide more performance evidence and clinical-testing data for PMA applications than for those approved through the 510(k) process. But critics say that the bar for proving the safety and efficacy of CAD products under the 510(k) process would be raised too high if the agency’s current draft guidance is approved.
“This substantially bumps the requirements of a 510(k) device up to that of a PMA,” says Robert Nishikawa, a University of Chicago medical physicist who has consulted with a number of companies on radiology product
development. The change would likely add time and expense to the approval process.
Under the proposed 510(k) application guidelines, manufacturers would be required to detail how they designed the algorithms that guide their CAD technology to identify potential areas of concern on an image. CAD devicemakers are worried, however, that the requirement would force them to disclose proprietary information.
“We spend millions and millions of dollars to design those algorithms, and we don’t want to reveal those trade secrets,” says Julian Marshall, director of product management for workflow solutions for Hologic, a Bedford, Mass.-based developer of mammography CAD software. He argues that the FDA should be able to glean safety and efficacy information from stand-alone testing of CAD devices. But the validity of stand-alone testing is also being challenged by the advisory panel that wrote the guidelines.
Previously, CAD manufacturers have been able to submit stand-alone test analysis—which looks at how accurately a CAD system marks images from previously determined cases—with their 510(k) applications to demonstrate that their devices perform on par with others already on the market. But the FDA’s proposed guidance would also require devicemakers to perform clinical studies on 510(k)-eligible devices. The guidance would, in most cases, require devicemakers to conduct multiple reader, multi- ple case studies where radiologists’ readings of an image would be compared against the new CAD system’s assessment of an image.
“One of the problems is that the manufacturers don’t have access to their competitors’ algorithms, so it’s difficult for them to know if their product is equivalent,” Petrick says. “So in those cases, I think we’ll definitely need to have reader studies that show the product is equivalent.” Critics of the draft guidance say the requirement is not realistic. “Having run multiple reader, multiple case trials, I’m very sensitive to how difficult logistically it is to put together these studies,” says Abe Dachman, professor of radiology with the University of Chicago Medical Center. “Finding radiologists who are willing to read the cases two and three times is difficult, and companies want to avoid doing multiple reader, multiple case studies because it’s so expensive.”
Dachman was unable to estimate the cost of multiple reader clinical studies, but Hologic’s Marshall says the expense easily runs into hundreds of thousands of dollars.
The FDA also would require that 510(k) clinical studies include a control group to show if there is a difference between how accurately radiologists assess images when they are aided by CAD technology on a first read and when they are unaided by the technology on a first read.
Certainly, the findings on the benefits of CAD are mixed. An April 5, 2007
New England Journal of Medicine Influence of Computer-Aided Detection on Performance of Screening Mammography,
found for example that the use of CAD was associated with reduced image interpretation accuracy by physicians and increased rates of biopsy, but was not clearly associated with improved detection of invasive breast cancer.
Single Reading With Computer-Aided Detection for Screening Mammography,
published Oct. 16, 2008, in the same medical journal, found that double reading of a mammogram image by two radiologists produced more accurate conclusions than a single read of an image coupled with CAD assistance.
Newstead says the challenge of CAD technology is that because it is more sensitive to potential anomalies than the human eye, it often calls attention to perfectly healthy tissue. Some studies have found, for example, that CAD typically generates two to four false-positive marks on a single mammography study. That means more patients are likely to be recalled for unnecessary biopsies and other tests, particularly when CAD is being used by a less-experienced radiologist who may not be skilled in determining if a CAD marking is superfluous.
“CAD is not perfect,” says Chris Flowers, an associate professor of radiology with the University of California at San Francisco’s breastimaging program. “It can mark things that aren’t an area of concern, and it can miss things that are and that can be dangerous. It can be a distraction to the reader when there’s a positive finding that actually represents nothing.”
Flowers believes that the FDA’s request for information about algorithm designs and additional clinical studies for CAD 510(k) applications are positive moves given uncertainty about the benefits of CAD. “I’m concerned that anytime you are going to allow a computer-aided device to be involved in detection you have to be careful to make sure the algorithms are accurate.”
Maryellen Giger is a professor of radiology at the University of Chicago and a member of the medical center’s CAD imaging team.
Newstead: CAD is used as “an assistant to radiologists.”