The aim of this cross-sectional study was to investigate the influence of replacing sedentary time with time engaged in one of two levels of physical activity on sleep quality using an isotemporal substitution model. The participants were 70 community-dwelling older Japanese adults (approximately 70% female). Physical activity types were measured using a triaxial accelerometer and categorized based on intensity as sedentary, light-intensity, and vigorous-intensity. The Pittsburgh Sleep Quality Index assessed subjective sleep quality. Objective sleep parameters were assessed using an actigraph. A series of multi-linear regression models analyzed the statistical relationships. Our findings showed that replacing 30 min of sedentary activity per day with an equal period of light-intensity physical activity significantly influenced sleep quality parameters. However, there was no significant difference in sleep quality when light-intensity activity was replaced with vigorous-intensity activity. Engaging in one activity type means less available time for other types of activity; habitual replacement of sedentary activity with light-intensity physical activity might have long-term benefits on the sleep quality of older people.