© 2019 Institute of Control, Robotics and Systems - ICROS. Recently, welfare vehicles are widely used among the aged. However, it is hard for inexperienced beginners to control the welfare vehicle in narrow places such as rooms and corridors. For the problem, the driving assist system for the welfare vehicle which is steered from an objective viewpoint with a transmittable Head Mounted Display (HMD) has been proposed in complex driving areas. Also, the autonomous driving system which can be used in safe areas for the driver has been studied. The advanced function which switches the driving assistance and the autonomous driving is needed because the welfare vehicles drive in the complex driving situations. In this paper, an autonomous driving system is proposed using the wearable MR-device which has collision avoidance function using the spatial mapping. As the system uses the virtual vehicle to avoid the obstacle, this system will be easy to switch the driving assistance and the autonomous driving. The obstacles are modeled by the spatial mapping installed in wearable HMD. Using the virtual vehicle drawn by the hologram, the autonomous system is steered using A∗ and NavMesh system on Unity. So, the welfare vehicle follows the virtual vehicle with the virtual platoon control constructed on the welfare vehicle. The effectiveness of proposed real-time obstacle avoidance system is evaluated by driving experiments.