To evaluate the cognitive decline at the early period in the cognitively healthy elderly, we modified the visual working memory task and measured cerebral blood flow responses during the task-set and task period using near-infrared spectroscopy (NIRS). We found that prefrontal activation was larger during the task execution than during the task-set period. However, with repetition of the task sessions, the activation during task-set period tended to be larger and the task-induced activation to be smaller in the frontal pole area. Furthermore, the activation during the task-set period was significantly correlated with the scores of several neuropsychological tests for prefrontal function. As a result, the temporal characteristics of prefrontal activation might reflect the cognitive function of the elderly people.