It has been well-documented that brain regions related to a task are activated during the task performance. We investigated whether brain activity and functional connectivity during the rest period are affected by the preceding task. Participants performed visual search tasks with three search conditions, which were followed by a rest period. During the rest period, participants were asked to look at the display that did not show any visual stimuli. In the result, brain activity in occipital and superior parietal regions would be deactivated by the preceding task during the rest period after visual search tasks. However, the activity of the inferior frontal gyrus during the rest period, which is also part of the attention network, was not affected by the brain activity during the preceding visual search task. We proposed a new model for explaining how the cognitive demands of the preceding visual search task regulate the attention network during the rest period after the task. In this model, the cognitive demand changes with task difficulty, which affects the brain activity even after removing the visual search task in the rest phase.