Atomistic simulation of tension of a thin film containing a notch on its surface is performed and its mechanical stability is analyzed by solving the eigenvalue problem of the Hessian matrix taking into account all the degrees of freedom of the atoms in the system. The value of an eigenvalue designates the curvature of the potential energy landscape in the direction of the corresponding eigenvector, which indicates the deformation mode
meaning that the system is unstable when any eigenvalue is zero or negative. At a strain where all the eigenvalues are positive, atomic fluctuation due to finite temperature may cause instability such as a dislocation nucleation. We have revealed that the path of the activated instability, dislocation emission from the notch, can be written by a linear combination of the eigenvectors having small eigenvalues. The energy landscape has a much lower hill along the mixed-mode path than along any single-mode paths.