Visual working memory (VWM) is known as a highly capacity-limited cognitive system that can hold 3-4 items. Recent studies have demonstrated that activity in the intraparietal sulcus (IPS) and occipital cortices correlates with the number of representations held in VWM. However, differences among those regions are poorly understood, particularly when task-irrelevant items are to be ignored. The present fMRI-based study investigated whether memory load-sensitive regions such as the IPS and occipital cortices respond differently to task-relevant information. Using a change detection task in which participants are required to remember pre-specified targets, here we show that while the IPS exhibited comparable responses to both targets and distractors, the dorsal occipital cortex manifested significantly weaker responses to an array containing distractors than to an array containing only targets, despite that the number of objects presented was the same for the two arrays. These results suggest that parietal and occipital cortices engage differently in distractor processing and that the dorsal occipital, rather than parietal, activity appears to reflect output of stimulus filtering and selection based on behavioral relevance.