Using Adversarial Images to Assess the Robustness of Deep Learning Models Trained on Diagnostic Images in Oncology

Marina Z. Joel, Sachin Umrao, Enoch Chang, Rachel Choi, Daniel X. Yang, James S. Duncan, Antonio Omuro, Roy Herbst, Harlan M. Krumholz, Sanjay Aneja

Research output: Contribution to journalArticlepeer-review

17 Scopus citations

Abstract

PURPOSE Deep learning (DL) models have rapidly become a popular and cost-effective tool for image classification within oncology. A major limitation of DL models is their vulnerability to adversarial images, manipulated input images designed to cause misclassifications by DL models. The purpose of the study is to investigate the robustness of DL models trained on diagnostic images using adversarial images and explore the utility of an iterative adversarial training approach to improve the robustness of DL models against adversarial images. METHODS We examined the impact of adversarial images on the classification accuracies of DL models trained to classify cancerous lesions across three common oncologic imaging modalities. The computed tomography (CT) model was trained to classify malignant lung nodules. The mammogram model was trained to classify malignant breast lesions. The magnetic resonance imaging (MRI) model was trained to classify brain metastases. RESULTS Oncologic images showed instability to small pixel-level changes. A pixel-level perturbation of 0.004 (for pixels normalized to the range between 0 and 1) resulted in most oncologic images to be misclassified (CT 25.6%, mammogram 23.9%, and MRI 6.4% accuracy). Adversarial training improved the stability and robustness of DL models trained on oncologic images compared with naive models ([CT 67.7% v 26.9%], mammogram [63.4% vs 27.7%], and MRI [87.2% vs 24.3%]). CONCLUSION DL models naively trained on oncologic images exhibited dramatic instability to small pixel-level changes resulting in substantial decreases in accuracy. Adversarial training techniques improved the stability and robustness of DL models to such pixel-level changes. Before clinical implementation, adversarial training should be considered to proposed DL models to improve overall performance and safety.

Original languageEnglish (US)
Article numbere2100170
JournalJCO Clinical Cancer Informatics
Volume6
DOIs
StatePublished - 2022
Externally publishedYes

ASJC Scopus subject areas

  • General Medicine

Fingerprint

Dive into the research topics of 'Using Adversarial Images to Assess the Robustness of Deep Learning Models Trained on Diagnostic Images in Oncology'. Together they form a unique fingerprint.

Cite this