Contingent attentional capture occurs when a stimulus property captures an observer's attention, usually related to the observer's top-down attentional set for target-defining properties. In this study, we examined whether contingent attentional capture occurs for a distractor that does not share the target-defining property at a physical level, but does share that property at an abstract level of representation. In a rapid serial visual presentation stream, we defined the target by color (e.g., a green-colored Japanese kanji character). Before the target onset, we presented a distractor that referred to the target-defining color (e.g., a white-colored character meaning "green"). We observed contingent attentional capture by the distractor, which was reflected by a deficit in identifying the subsequent target. This result suggests that because of the attentional set, stimuli were scanned on the basis of the target-defining property at an abstract semantic level of representation.