The incidence of breast cancer continues to rise worldwide. Due to the heterogeneity of breast cancer

there is an overlap between benign and malignant tumor images that only using a type of images cannot obtain satisfactory classification results. This paper proposes a two-stage breast cancer diagnosis system based on ultrasound and mammogram images. In the first phase

an abstaining classification method is used to classify breast ultrasound (BUS) images

in which some BUS tumors are classified with high confidence

and the uncertain tumors are not classified. These unclassified tumors are then classified using mammogram images in the second stage. Supplemented by mammogram information

the system can diagnose breast cancer by utilizing multimodal image information to screen for unrecognizable ultrasound images. The ultrasound and mammogram images used in this study were provided by the Cancer Hospital of Harbin Medical University and the First Affiliated Hospital of Harbin Medical University. The abstaining method and the two-stage diagnosis system were validated in experiments. Compared with diagnostic systems using only BUS features

the proposed two-stage diagnostic system provided better performance with the accuracy of 92.59%

AUC of 0.933 3

G-mean of 93.09%

sensitivity of 86.67%

specificity of 100%

positive predictive value of 100%

negative predictive value of 85.71%

and Matthew’s correlation coefficient of 0.861 9. Experimental results demonstrate that adding mammogram information can increase the performance of the diagnosis system that uses BUS images only.