Considering the small differences between different types in the diabetic retinopathy (DR) grading task, a retinopathy grading algorithm based on cross-layer bilinear pooling is proposed. Firstly, the input image is cropped according to the Hough circle transform (HCT), and then the image contrast is improved by the preprocessing method; then the squeeze excitation group residual network (SEResNeXt) is used as the backbone of the model, and a cross-layer bilinear pooling module is introduced for classification. Finally, a random puzzle generator is introduced in the training process for progressive training, and the center loss (CL) and focal loss (FL) methods are used to further improve the effect of the final classification. The quadratic weighted Kappa (QWK) is 90.84% in the Indian Diabetic Retinopathy Image Dataset (IDRiD), and the area under the receiver operating characteristic curve (AUC) in the Messidor-2 dataset (Messidor-2) is 88.54%. Experiments show that the algorithm proposed in this paper has a certain application value in the field of diabetic retina grading.