When a large-scale disaster occurs, each evacuee should move to an appropriate refuge in a speedy and safe manner. Most of the existing studies on the refuge assignment consider the speediness of evacuation and refuge capacity while the safety of evacuation is not taken into account. In this paper, we propose a refuge assignment scheme that considers both the speediness and safety of evacuation under the refuge capacity constraint. We first formulate the refuge assignment problem as a two-step integer linear program (ILP). Since the two-step ILP requires route candidates between evacuees and their possible refuges, we further propose a speedy and reliable route selection scheme as an extension of the existing route selection scheme. Through numerical results using the actual data of Arako district of Nagoya city in Japan, we show that the proposed scheme can improve the average route reliability among evacuees by 13.6% while suppressing the increase of the average route length among evacuees by 7.3%, compared with the distance-based route selection and refuge assignment. In addition, we also reveal that the current refuge capacity is not enough to support speedy and reliable evacuation for the residents.