This research project, as a part of Special Project for Earthquake Disaster Mitigation in Urban Areas initiated by the MEXT, aims to extend the existing robotics and human interface technologies and to establish a novel human-machine interacting structure to enable effective use of semi-autonomous robots in place of humans for searching survivors from rubble under human remote instructions. Collaboration and coordination of the human and robot intelligence and skills are to be developed from both hardware and software viewpoints to make the results of this project contribute to improvement of the search technology, an important issue in the urban robot-assisted search and rescue activities. Recent research efforts and results are described in this paper.