Technique evaluates reliability of imaging measurements used in medical decision-making

Advances in medical imaging and artificial intelligence (AI) have revolutionized many aspects of medical practice in recent years. Quantitative measurements derived from medical images are increasingly being used to support diagnosis and clinical decision-making. Consequently, many new quantitative imaging tools, including those based on AI, are being developed.

But evaluating the reliability of these imaging tools can be challenging. In clinical settings, the true value of the item being measured — for example, the actual size of a cancerous tumor inside the body — is often unknown. That makes it hard to know if a given measurement tool works as intended.

To address this concern, researchers at Washington University in St. Louis have developed a technique that researchers and clinicians can use to assess the reliability of medical imaging tools. The new method may also be useful for regulators who evaluate new AI-backed medical imaging tools as they come to market.

Results from this project, led by Abhinav Jha, an associate professor of biomedical engineering at the McKelvey School of Engineering and of radiology at WashU Medicine Mallinckrodt Institute of Radiology (MIR), were published in IEEE Transactions on Medical Imaging.

“Medical images are increasingly being used not only for visual interpretation, but also to derive quantitative measurements of clinically relevant properties,” said Jha, who is a research member at Siteman Cancer Center, based at Barnes-Jewish Hospital and WashU Medicine. “We want these measurements to be precise. Otherwise, they can impact the clinical decisions that are being made based on them.”

Published with fellow Siteman research members Daniel L.J. Thorek, and Barry A. Siegel, MD, both faculty at MIR, and Jingqin (Rosy) Luo, professor of surgery at WashU Medicine, the new study shows how a technique called NGSE-Corr can be used to objectively determine which quantitative imaging methods work best — even in the absence of a “gold standard,” or true value for the quantity being measured.

“Being able to compare different quantitative imaging methods with clinical data gives us greater confidence in the evaluation,” Jha said. “However, with clinical data, gold standards can be expensive and time-consuming to obtain, and in many cases, may simply be unavailable. Thus, validation with clinical data has been a long-standing challenge.

“I am excited about this technique because it represents a promising step toward evaluation with clinical data without requiring gold standards,” he said.

The first author of the study is Yan Liu, a student in the imaging science PhD program at WashU, an innovative interdisciplinary program that brings together expert faculty from the McKelvey School of Engineering and WashU Medicine.

For the new study, the WashU researchers built upon a previously proposed mathematical formulation for quantitative imaging, bringing new insights related to the concept of “correlated noise.” In actual clinical settings, if a doctor were to use multiple different quantitative imaging methods to measure certain clinical properties of the same tumor, then any random fluctuations in the measurements offered by these tools would be correlated, not independent.

Jha modified the original formulation to be more useful for evaluating imaging tools built to measure the same item. He and his team then ran a variety of numerical experiments that showed that their NGSE-Corr method could accurately rank different quantitative imaging methods on the basis of precision. 

Then the researchers conducted a virtual imaging trial to rank three quantitative SPECT methods for measuring regional activity uptake in computer-generated patients with bone metastatic castrate-resistant prostate cancer treated with radium-223. The ranking task was meant to determine which imaging tool was most fitting in the given situation. 

“Without any knowledge of the ground truth, our methodology was able to accurately rank the imaging methods for 91% of the trials,” in groups of 50 virtual patients, Jha said. “We also identified the most precise method for 95% of the trials.” Running the trials with higher numbers of patients improved these results even further.

Having worked in quantitative imagery for close to two decades, Jha is particularly excited about the ways that the new NGSE-Corr technique can help build confidence in these technologies, so that he and other innovators can continue to build on today’s tools for even broader applications.

“This tool has a lot of potential applications for different types of people in the community,” Jha said. “It’s useful for researchers who are developing new quantitative imaging methods, including AI-based methodologies; physicians who are looking at using different tools to make decisions; and regulators who are interested in evaluating technologies.”

Click on the topics below for more stories in those areas

• Research
• Biomedical Engineering