Denoising methods based on wavelet analysis and empirical mode decomposition cannot essentially track and eliminate noise, which usually cause distortion of heart sounds. Based on this problem, a heart sound denoising method based on improved minimum control recursive average and optimally modified log-spectral amplitude is proposed in this paper. The proposed method uses a short-time window to smoothly and dynamically track and estimate the minimum noise value. The noise estimation results are used to obtain the optimal spectrum gain function, and to minimize the noise by minimizing the difference between the clean heart sound and the estimated clean heart sound. In addition, combined with the subjective analysis of spectrum and the objective analysis of contribution to normal and abnormal heart sound classification system, we propose a more rigorous evaluation mechanism. The experimental results show that the proposed method effectively improves the time-frequency features, and obtains higher scores in the normal and abnormal heart sound classification systems. The proposed method can help medical workers to improve the accuracy of their diagnosis, and also has great reference value for the construction and application of computer-aided diagnosis system.