Both feature representation and classifier performance are important factors that determine the performance of computer-aided diagnosis (CAD) systems. In order to improve the performance of ultrasound-based CAD for breast cancers, a novel multiple empirical kernel mapping (MEKM) exclusivity regularized machine (ERM) ensemble classifier algorithm based on self-paced learning (SPL) is proposed, which simultaneously promotes the performance of both feature representation and the classifier. The proposed algorithm first generates multiple groups of features by MEKM to enhance the ability of feature representation, which also work as the kernel transform in multiple support vector machines embedded in ERM. The SPL strategy is then adopted to adaptively select samples from easy to hard so as to gradually train the ERM classifier model with improved performance. This algorithm is verified on a B-mode ultrasound dataset and an elastography ultrasound dataset, respectively. The results show that the classification accuracy, sensitivity and specificity on B-mode ultrasound are (86.36±6.45)%, (88.15±7.12)%, and (84.52±9.38)%, respectively, and the classification accuracy, sensitivity and specificity on elastography ultrasound are (85.97±3.75)%, (85.93±6.09)%, and (86.03±5.88)%, respectively. It indicates that the proposed algorithm can effectively improve the performance of ultrasound-based CAD for breast cancers with the potential for application.