Nonmagnetic impurity scattering effects in the vortex states are studied in the Born and the unitary limits for chiral superconductors by Eilenberger theory. We compare the spatial structure of the pair potential and local electronic structure in chiral p-wave superconductors with those in two types of chiral d-wave superconductors; d(1 +/-) d(xz) +/- id(yz) and d(2 +/-) d(x2-y2 )+/- id(xy) pairing Similar behaviors of the pair potentials are seen in p +/- - and d(1 +/-) -wave superconductors. In chiral d-wave superconductors, due to lack of Majorana properties relating to the s-wave Cooper pair amplitude, zero-energy vortex bound states are easily destroyed by the impurity scattering, compared to the Born limit in the chiral p_-wave superconductors. This decay is also seen in the unitary limit in the chiral p_ -wave superconductors. These differences are also discussed in the relation to s-wave Cooper pair amplitude and the local scattering rate. The impurity effects are also compared with those in s-wave superconductors.