Autoimmune pancreatitis (AIP) is a unique subtype of chronic pancreatitis, which shares many clinical presentations with pancreatic ductal adenocarcinoma (PDA). The misdiagnosis of AIP often leads to unnecessary pancreatic resection. 18F-FDG positron emission tomography/ computed tomography (PET/CT) could provide comprehensive information on the morphology, density, and functional metabolism of the pancreas at the same time. It has been proved to be a promising modality for noninvasive differentiation between AIP and PDA. However, there is a lack of clinical analysis of PET/CT image texture features. Difficulty still remains in differentiating AIP and PDA based on commonly used diagnostic methods. Therefore, this paper studied the differentiation of AIP and PDA based on multi-modality texture features. We utilized multiple feature extraction algorithms to extract the texture features from CT and PET images at first. Then, the Fisher criterion and sequence forward floating selection algorithm (SFFS) combined with support vector machine (SVM) was employed to select the optimal multi-modality feature subset. Finally, the SVM classifier was used to differentiate AIP from PDA. The results prove that texture analysis of lesions helps to achieve accurate differentiation of AIP and PDA.