The ability to switch between tasks is critical for animals to behave according to context. Although the association between the prefrontal cortex and task switching has been well documented, the ultimate modulation of sensory-motor associations has yet to be determined. Here, we modeled the results of a previous study showing that task switching can be accomplished by communication from distinct populations of sensory neurons. We proposed a leaky-integrator model where relevant and irrelevant information were stored separately in two integrators and task switching was achieved by leaking information from the irrelevant integrator. The model successfully explained both the behavioral and neuronal data. Additionally, the leaky-integrator model showed better performance than an alternative model, where irrelevant information was discarded by decreasing the weight on irrelevant information, when animals initially failed to commit to a task. Overall, we propose that flexible switching is, in part, achieved by actively controlling the amount of leak of relevant and irrelevant information.