Previous research suggests that older adults' motor performance is associated with cognitive function. Although this has been reported especially for executive function, it is not yet clear for various types of working memory (WM). In fact, age-related decline in WM is more severe for faces than other types of visual objects. The present study focused on the relationship between diverse WM and two types of motor performance (mobility and manual dexterity), which are implicated in pathological decline. To measure diverse WM, we adopted N-back tasks using three distinct types of stimuli (numbers, locations, and faces). Mobility was measured with the timed up and go test and manual dexterity was measured with the Pegboard Test. Participants were community-dwelling older adults (age: mean 78.6 years). Comparisons of younger and older adults' N-back performances indicated that WM for faces is more sensitive to aging compared with WM for the other stimuli. Correlation analyses within the older group indicated that WM tasks mainly correlated with mobility, but less so with manual dexterity. Among the three types of WM, spatial WM and face WM had significant partial correlation coefficients with mobility after age and general cognitive decline were controlled. These results indicate that visually encoded WM is associated only with mobility, although general cognitive function is related to both motor abilities. The selective association between the visually encoded WM and mobility is discussed in terms of the interactive processes between executive processing and perceptual encoding, where dynamic visual processing for locomotion plays a role.