Despite a number of studies of computational models of visual saliency, it is still difficult to predict where human would attend in a scene. One of the problems is the effect of self-motion on retinal images. Usually saliency is calculated with local motion signals, and high degree of saliency is found in motion components caused by self-motion. Since human observers usually ignore the motion caused by self-motion, the prediction accuracy of attention locations falls off. We developed a framework that reduces the saliency of the motion components caused self-motion, using a physiological model of optic flow processing that extracts object motion among self-motion signals.