Objective To investigate the relationship between skin/pectoral muscle invasion and the prognosis of male breast cancer. Methods Clinical data and follow-up information of 79 male breast cancer patients who received treatment between September 2008 to April 2020 in West China Hospital were retrospectively reviewed, to analyze the clinicopathological features of male breast cancer and prognostic value of skin/pectoral muscle invasion. Results Among 79 male breast cancer patients, a total of 23 patients (29.1%) were with skin/pectoral muscle invasion at diagnosis. All the patients were followed up, with a median follow-up period of 63.3 months (1.0–204.5 months). Within follow-up period, 8 patients (10.1%) suffered from relapse, 19 patients (24.7%, 19/77) suffered from metastasis, and 4 patients (5.1%) died. Multivariate Cox proportional risk regression model suggested that patients with skin/pectoral muscle invaded had poor disease free survival [RR=4.48, 95%CI (1.08, 18.52), P=0.038]. Conclusions Skinor pectoral muscle invasion might be a valuable prognostic factor for male breast cancer patients. However, limited by sample size, the conclusion should be proved by further high-level studies.