This paper presents a novel algorithm for estimating complex human motion from 3D video. We base our algorithm on a model-based approach which uses a complete surface mesh of a 3D human model to be matched with 3D video data. This type of method usually works well against partly incomplete input data. However it fails to estimate what we call "complex motion": where some parts of the body touch each other for a long period. It is because the touching deteriorates the visibility of the neighbouring surface, which causes matching failures.
In order to solve this problem, we introduce a "visibility" measure for each mesh vertex that represents how it is occluded or missed on the observed surface. Using the "visibility" we selectively suppress the outliers caused by low observability while traditional surface matching algorithms try to find corresponding area for the entire surface and cannot converge to the real posture by definition. Our algorithm shows improvements over naive surface matching algorithm on both synthesized and real 3D video.