An important ability for mobile robots is to process multiple tasks in complex environments. Since the sensor resources on a robot are limited, it is necessary to distribute the sensors attention to different tasks along the time scale. This paper proposes a temporal attention control method which aims at detecting multiple objects and estimating their poses with a single actuated camera. The proposed method is based on three criteria which are partially inspired by human behavior: (i) minimization of the overall object poses perception uncertainty and minimization of the variance of the perception uncertainty of different objects; (ii) minimization of the camera movements for completing the tasks; (iii) maximization of the number of objects in the cameras field of view. The proposed approach use Kalman filters to estimate object poses and to determine the perception uncertainty. The method was evaluated with both simulation and experiment on actual robot. The results show that the proposed approach is able to switch the camera's attention according to the objects poses and movements efficiently with a low frequency of the camera movements.