Milling of thin-walled mechanical parts is limited by chatter, which affects surface quality. This paper concerns vibration analysis in milling with respect to nonlinear and non-stationer signals. Initially a stability lobe diagram was created to determine cutting conditions. Therefore, hammering test was conducted to determine modal parameters. Following, milling tests were conducted under different cutting conditions. Data obtained in milling were analyzed by Hilbert-Huang Transform (HHT) and were compared with Fast Fourier Transform (FFT) results. HHT decomposes original signals into a series component, called IMFs, by empirical mode decomposition (EMD). Each IMF separated tooth passing, natural and chatter frequencies within frequency bandwidth used. Thus, the EMD made it easy to recognize the chatter growing as cutting depth was increased. The Hilbert transform is then applied on each IMF. The energy associated to chatter frequency increased in the Hilbert spectrum. The increasing of energy indicated the occurrence of chatter. HHT revealed the transition of machining state, from stable to unstable. The results showed that the HHT has an advantage over the FFT.