Research has confirmed that the visual working memory representation of objects' roughness is robust against illumination changes in the human ventral visual cortex and intraparietal sulcus, but not yet against visual distractors during memory maintenance. Thus, this study investigated the effects of visual distractors on roughness-related brain regions during the maintenance phase using multi-voxel pattern analysis (MVPA). We conducted an fMRI experiment in which participants were asked to memorize a sphere's roughness against visual distractors, presented during the delay period in random trials. Region of interest-based MVPA showed no contribution of the ventral visual cortex and intraparietal sulcus to the roughness memory, regardless of behavioral performance. Post hoc searchlight MVPA revealed an above-chance decoding performance level in the brain regions presumably related to haptic processing when no visual distractors were shown. In contrast, when visual distractors appeared in the delay period, decoding performance exceeded the chance level in the ventral visual cortex. These results suggest that when visual distractors are presented during the memory phase, both visual and haptic processing are related to visual working memory for roughness, and the weighting of modality changes depending on the presence of visual distractors.