The requirement for unconstrained monitoring of heartbeat during sleep is increasing, but the current detection devices can not meet the requirements of convenience and accuracy. This study designed an unconstrained ballistocardiogram (BCG) detection system using acceleration sensor and developed a heart rate extraction algorithm. BCG is a directional signal which is stronger and less affected by respiratory movements along spine direction than in other directions. In order to measure the BCG signal along spine direction during sleep, a 3-axis acceleration sensor was fixed on the bed to collect the vibration signals caused by heartbeat. An approximate frequency range was firstly assumed by frequency analysis to the BCG signals and segmental filtering was conducted to the original vibration signals within the frequency range. Secondly, to identify the true BCG waveform, the accurate frequency band was obtained by comparison with the theoretical waveform. The J waves were detected by BCG energy waveform and an adaptive threshold method was proposed to extract heart rates by using the information of both amplitude and period. The accuracy and robustness of the BCG detection system proposed and the algorithm developed in this study were confirmed by comparison with electrocardiogram (ECG). The test results of 30 subjects showed a high average accuracy of 99.21% to demonstrate the feasibility of the unconstrained BCG detection method based on vibration acceleration.