General movements (GMs), a type of spontaneous movement, have been used for the early diagnosis of infant disorders. In clinical practice, GMs are visually assessed by qualified licensees; however, this presents a difficulty in terms of quantitative evaluation. Various measurement systems for the quantitative evaluation of GMs track target markers attached to infants; however, these markers may disturb infants' spontaneous movements. This paper proposes a markerless movement measurement and evaluation system for GMs in infants. The proposed system calculates 25 indices related to GMs, including the magnitude and rhythm of movements, by video analysis, that is, by calculating background subtractions and frame differences. Movement classification is performed based on the clinical definition of GMs by using an artificial neural network with a stochastic structure. This supports the assessment of GMs and early diagnoses of disabilities in infants. In a series of experiments, the proposed system is applied to movement evaluation and classification in full-term infants and low-birth-weight infants. The experimental results confirm that the average agreement between four GMs classified by the proposed system and those identified by a licensee reaches up to 83.1 ± 1.84%. In addition, the classification accuracy of normal and abnormal movements reaches 90.2 ± 0.94%.