This paper presents a new approach to estimate the kinematic structure underlying a sequence of 3D dynamic surfaces reconstructed from multi-view video. The key idea is a mesoscopic surface characterization with a tree-structure constraint. Combined with different levels of surface characterizations, namely macroscopic and microscopic characterizations, our mesoscopic surface characterization can cope with shape estimation errors and global topology changes of 3D surfaces from the real world to estimate kinematic structure. The macroscopic analysis focuses on global surface topology to perform temporal segmentation of 3D video sequence into topologically-coherent sub-sequences. The microscopic analysis operates at the mesh structure level to provide temporally consistent mesh structures using a surface alignment method on each of the topologically-coherent sub-sequences. Then, the mesoscopic analysis extracts rigid parts from the preprocessed 3D video segments to establish partial kinematic structures, and integrates them into a single unified kinematic model. Quantitative evaluations using synthesized and real data demonstrate the performance of the proposed algorithm for kinematic structure estimation.