In this paper, the authors propose a robust background estimation algorithm, which is a technique for adaptively estimating background in response to changes in the environment based on robust statistics, and a technique for automatically adjusting the adaptation rate in response to changes in the environment. In monitoring systems and the like using stationary cameras, it is necessary to detect moving objects from the captured image series, and recognize and identify the detected subject. The most simple and representative method for moving object detection, which is used in many fields, is the background subtraction method for detecting moving objects by the difference between the background image and the current scene. This method requires a background image which does not include a moving object. However, it is difficult to acquire a background image under conditions in which moving objects are continually included in the scene. Also, when the environment changes, adaptive revision must be performed in response to the changes in the background image. Furthermore, in an environment where sections of the background change over time in different ways, it is important to determine the appropriate number of frames needed to estimate the background. In this paper, therefore, the authors propose a method for adaptive background estimation using M-estimation, a known robust statistics technique. Also, the authors propose a technique for adjusting the number of frames necessary for background estimation by varying the adaptation rate to the environment using robust template matching. © 2007 Wiley Periodicals, Inc.