HER2-positive
breast cancer is a highly heterogeneous
tumor, and about 30% of patients still suffer from recurrence and
metastasis after
trastuzumab targeted
therapy. Predicting individual prognosis is of great significance for the further development of precise
therapy. With the continuous development of computer technology, more and more attention has been paid to computer-aided diagnosis and prognosis prediction based on
Hematoxylin and
Eosin (H&E) pathological images, which are available for all
breast cancer patients undergone surgical treatment. In this study, we first enrolled 127 HER2-positive
breast cancer patients with known recurrence and
metastasis status from Cancer Hospital of the Chinese Academy of Medical Sciences. We then proposed a novel multimodal deep learning method integrating whole slide H&E images (WSIs) and clinical information to accurately assess the risk of relapse and
metastasis in patients with HER2-positive
breast cancer. Specifically, we obtained the whole H&E staining images from the surgical specimens of
breast cancer patients, and these images were adjusted to size 512 × 512 pixels. The deep convolutional neural network (CNN) was applied to these images to retrieve image features, which were combined with the clinical data. Based on the combined features. After that, a novel multimodal model was constructed for predicting the prognosis of each patient. The model achieved an area under curve (AUC) of 0.76 in the two-fold cross-validation (CV). To further evaluate the performance of our model, we downloaded the data of all 123 HER2-positive
breast cancer patients with available H&E image and known recurrence and
metastasis status in The
Cancer Genome Atlas (TCGA), which was severed as an independent testing data. Despite the huge differences in race and experimental strategies, our model achieved an AUC of 0.72 in the TCGA samples. As a conclusion, H&E images, in conjunction with clinical information and advanced deep learning models, could be used to evaluate the risk of relapse and
metastasis in patients with HER2-positive
breast cancer.