This paper proposes a markerless infant movement assessment system for general Movements (GMs) evaluation. This system calculates twenty-five types of evaluation indices related to the movements of an infant such as movement frequency and rhythm of movement from binary images that are extracted from video images using the background difference and the frame difference. Movement discrimination based on GMs is also performed using a neural network. Medical doctors thus can intuitively understand the movements of infants without long-term observations. This will be helpful in supporting their diagnoses of disabilities and diseases in the early stages. The distinctive feature of this system is that the movements of infants can be measured without using any markers for motion capture and can discriminate movements based on GMs automatically using a neural network. In the experiments conducted during the study, evaluation and classification of infant movements based on GMs are demonstrated using the proposed system for full-term infants and low birth weight infants. The results revealed that the proposed system can evaluate infant movements similarly to a licensed evaluator and can classify GMs with a certain accuracy (average classification rates: 76.2 ± 2.83% for four types GMs classification, 92.9 ± 1.98% for normal/abnormal classification).