Resistance to distraction is a key component of executive functions and is strongly linked to the prefrontal cortex. Recent evidence suggests that neural mechanisms exist for selective suppression of task-irrelevant information. However, neuronal signals related to selective suppression have not yet been identified, whereas nonselective surround suppression, which results from attentional enhancement for relevant stimuli, has been well documented. This study examined single neuron activities in the lateral PFC when monkeys covertly tracked one of randomly moving objects. Although many neurons responded to the target, we also found a group of neurons that exhibited a selective response to the distractor that was visually identical to the target. Because most neurons were insensitive to an additional distractor that explicitly differed in color from the target, the brain seemed to monitor the distractor only when necessary to maintain internal object segregation. Our results suggest that the lateral PFC might provide at least two top-down signals during covert object tracking: one for enhancement of visual processing for the target and the other for selective suppression of visual processing for the distractor. These signals might work together to discriminate objects, thereby regulating both the sensitivity and specificity of target choice during covert object tracking.