Alzheimer’s disease (AD) is an irreversible neurodegenerative disorder that damages patients’ memory and cognitive abilities. Therefore, the diagnosis of AD holds significant importance. The interactions between regions of interest (ROIs) in the brain often involve multiple areas collaborating in a nonlinear manner. Leveraging these nonlinear higher-order interaction features to their fullest potential contributes to enhancing the accuracy of AD diagnosis. To address this, a framework combining nonlinear higher-order feature extraction and three-dimensional (3D) hypergraph neural networks is proposed for computer-assisted diagnosis of AD. First, a support vector machine regression model based on the radial basis function kernel was trained on ROI data to obtain a base estimator. Then, a recursive feature elimination algorithm based on the base estimator was applied to extract nonlinear higher-order features from functional magnetic resonance imaging (fMRI) data. These features were subsequently constructed into a hypergraph, leveraging the complex interactions captured in the data. Finally, a four-dimensional (4D) spatiotemporal hypergraph convolutional neural network model was constructed based on the fMRI data for classification. Experimental results on the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database demonstrated that the proposed framework outperformed the Hyper Graph Convolutional Network (HyperGCN) framework by 8% and traditional two-dimensional (2D) linear feature extraction methods by 12% in the AD/normal control (NC) classification task. In conclusion, this framework demonstrates an improvement in AD classification compared to mainstream deep learning methods, providing valuable evidence for computer-assisted diagnosis of AD.