In cerebral small vessel disease (cSVD), whole brain MRI markers of cSVD-related brain injury explain limited variance to support individualized prediction. Here, we investigate whether considering abnormalities in brain tracts by integrating multimodal metrics from diffusion MRI (dMRI) and structural MRI (sMRI), can better capture cognitive performance in cSVD patients than established approaches based on whole brain markers. We selected 102 patients (73.7 ± 10.2 years old, 59 males) with MRI-visible SVD lesions and both sMRI and dMRI. Conventional linear models using demographics and established whole brain markers were used as benchmark of predicting individual cognitive scores. Multi-modal metrics of 73 major brain tracts were derived from dMRI and sMRI, and used together with established markers as input of a feed-forward artificial neural network (ANN) to predict individual cognitive scores. A feature selection strategy was implemented to reduce the risk of overfitting. Prediction was performed with leave-one-out cross-validation and evaluated with the R2 of the correlation between measured and predicted cognitive scores. Linear models predicted memory and processing speed with R2 = 0.26 and R2 = 0.38, respectively. With ANN, feature selection resulted in 13 tract-specific metrics and 5 whole brain markers for predicting processing speed, and 28 tract-specific metrics and 4 whole brain markers for predicting memory. Leave-one-out ANN prediction with the selected features achieved R2 = 0.49 and R2 = 0.40 for processing speed and memory, respectively. Our results show proof-of-concept that combining tract-specific multimodal MRI metrics can improve the prediction of cognitive performance in cSVD by leveraging tract-specific multi-modal metrics.