Background: Low lumbar skeletal muscle mass and density have been associated with adverse outcomes in different populations with colorectal cancer (CRC). We aimed to determine whether skeletal muscle mass, density, and physical performance are associated with postoperative complications and overall survival (OS) in older CRC patients. Methods: We analysed consecutive patients (≥70 years) undergoing elective surgery for non-metastatic CRC (stage I-III). Lumbar skeletal muscle mass and muscle density were measured using abdominal CT-images obtained prior to surgery. Low skeletal muscle mass and low muscle density were defined using commonly used thresholds and by gender-specific quartiles (Q). The preoperative use of a mobility aid served as a marker for physical performance. Cox regression proportional hazard models were used to investigate the association between the independent variables and OS. Results: 174 Patients were included (mean age 78.0), with median follow-up 2.6 years. 36 Patients (21%) used a mobility aid preoperatively. Low muscle density (Q1 vs Q4) and not muscle mass was associated with worse postoperative outcomes, including severe complications (p < 0.05). Use of a mobility aid was associated with more complications, including severe complications (39% vs 17%, p = 0.004) and OS (HR 2.65, CI 1.29–5.44, p = 0.01). However, patients with mobility aid use and low skeletal muscle mass had worse OS (HR 5.68, p = 0.003). Conclusion: Low skeletal muscle density and not muscle mass was associated with more complications after colorectal surgery in older patients. Physical performance has the strongest association for poor surgical outcomes and should be investigated when measuring skeletal muscle mass and density.