This study develops a vision-based displacement measurement system for remote monitoring of vibration of large-size structures such as bridges and buildings. The system consists of one or multiple video cameras and a notebook computer. With a telescopic lens, the camera placed at a stationary point away from a structure captures images of an object on the structure. The structural displacement is computed in real time through processing the captured images. A robust object search algorithm developed in this study enables accurate measurement of the displacement by tracking existing features on the structure without requiring a conventional target panel to be installed on the structure. The efficacy of the system in remote measurement of dynamic displacements was demonstrated through a shaking table test and a field experiment on a long-span bridge.