Heart sound is one of the common medical signals for diagnosing cardiovascular diseases. This paper studies the binary classification between normal or abnormal heart sounds, and proposes a heart sound classification algorithm based on the joint decision of extreme gradient boosting (XGBoost) and deep neural network, achieving a further improvement in feature extraction and model accuracy. First, the preprocessed heart sound recordings are segmented into four status, and five categories of features are extracted from the signals based on segmentation. The first four categories of features are sieved through recursive feature elimination, which is used as the input of the XGBoost classifier. The last category is the Mel-frequency cepstral coefficient (MFCC), which is used as the input of long short-term memory network (LSTM). Considering the imbalance of the data set, these two classifiers are both improved with weights. Finally, the heterogeneous integrated decision method is adopted to obtain the prediction. The algorithm was applied to the open heart sound database of the PhysioNet Computing in Cardiology(CINC) Challenge in 2016 on the PhysioNet website, to test the sensitivity, specificity, modified accuracy and F score. The results were 93%, 89.4%, 91.2% and 91.3% respectively. Compared with the results of machine learning, convolutional neural networks (CNN) and other methods used by other researchers, the accuracy and sensibility have been obviously improved, which proves that the method in this paper could effectively improve the accuracy of heart sound signal classification, and has great potential in the clinical auxiliary diagnosis application of some cardiovascular diseases.