Enhancing the antitumor effect of radiation, while reducing damage to organs, is a significant challenge in radiation therapy for head and neck malignancies. One promising radiosensitizer is gold. The present study aimed to determine whether gold nanoparticles (AuNPs) have the potential to enhance the effects of X-ray irradiation on head and neck cancer cells. The human head and neck carcinoma cell line HSC-3 was used. Total cell number and the levels of cell proliferation and apoptosis were compared between control cells and cells treated with 5-nm AuNPs alone at four concentrations (0.1, 0.4, 1.0 and 10.0 nM), X-ray irradiation alone at three doses (2, 4 and 8 Gy), or a combination of 4 Gy X-ray irradiation and 1.0 nM AuNPs. Analysis of variance and Tukey-Kramer testing were performed to compare the different groups. The total number of cells significantly decreased following 4 and 8 Gy X-ray irradiation, compared with in the control group (control vs. 4Gy, P=2.19×10-4; control vs. 8Gy, P=1.28×10-6). The combination of 4 Gy X-ray irradiation and 1.0 nM AuNPs significantly reduced the total number of cells compared with 4 Gy X-ray irradiation alone (P=2.95×10-4). Cell proliferation was not affected by AuNP treatment alone, 4 Gy X-ray irradiation alone or the combination of X-ray irradiation and AuNPs. The combination of 4 Gy irradiation and 1.0 nM AuNPs significantly increased the number of apoptotic cells compared with 4 Gy irradiation alone (P=0.0261). In conclusion, AuNPs combined with X-ray irradiation enhanced the cytotoxic effect on human head and neck cancer cells in vitro, through the induction of apoptosis, but not inhibition of cell proliferation.