We estimate ground motions from the record observed at the JMA Kobe station in the Sannomiya district in Chuo Ward, Kobe, where heavily damaged buildings were concentrated during Hyogo-Ken Nambu earthquake, and use them to examine the damage distribution of buildings. We use a two-dimensional finite element method to model local topography, shallow sub-surface layers, and a deep basin structure formed by the Rokko faults. The resultant ground motion without holocene layers has peak accelerations or velocities well exceeding 800 cm/sec^2 or 100 cm/sec, respectively, in the narrow zone 750m away from the fault. These peak values could be more on the surface since holocene layers would amplify them. Therefore, we carry out nonlinear analyses of holocene layers with five representative sites to estimate surface ground motions after we verified our scheme by comparing the results of the nonlinear analysis with surface records at Fukiai Gas supply station in Chuo Ward. Then, we evaluate maximum ductility response of buildings which are subjected to ground motions with or without holocene layers. Variation of ductility responses with respect to the distance from the fault shows good correspondence with the area of damage concentration. These results suggest that both a deep basin structure and shallow holocene layers have to be taken into account to simulate the strong ground motion and hence to explain damage concentration in Kobe during the Hyogo-ken Nambu earthquake.