Non-magnetic impurity scattering effects on the vortex core states are theoretically studied to clarify the contributions from the sign-change of the pairing function in anisotropic superconductors. The vortex states are calculated by the Eilenberger theory in superconductors with px-wave pairing symmetry, as well as the corresponding anisotropic s-wave symmetry. From the spatial structure of the pair potential and the local electronic states around a vortex, we examine the differences between anisotropic superconductors with and without sign-change of the pairing function, and estimate how twofold symmetric vortex core images change with increasing the impurity scattering rate both in the Born and the unitary limits. We found that twofold symmetric vortex core image of zero-energy local density of states changes the orientation of the twofold symmetry with increasing the scattering rate when the sign change occurs in the pairing function. Without the sign change, the vortex core shape reduces to circular one with approaching dirty cases. These results of the impurity effects are valuable for identifying the pairing symmetry by observation of the vortex core image by the STM observation.