The Fist-Edge-Palm (FEP) task, a manual hand task, has been used to detect frontal dysfunctions in clinical situations: its performance failures are observed in various prefrontal cortex (PFC)-related disorders, including schizophrenia. However, previous imaging studies reported that the performance of the FEP task activated motor-related areas, but not the PFC. Here, we aimed to investigate the relationships between the performance of the FEP task and PFC functions. Hemodynamic activity in the PFC, including the dorsolateral PFC (area 46) and frontal pole (area 10), was recorded. Healthy young subjects performed the FEP task as well as a palm tapping (PT) task (control task) three times. The subjects also completed a Wisconsin Card Sorting Test (WCST) and Schizotypal Personality Scale (STA) questionnaire. We found that hemodynamic activity (Oxy-Hb) in the PFC increased in the first trial of the FEP task but decreased considerably in the second and third trials compared to the PT task. The number of performance errors in the FEP task also decreased in the second and third trials. Error reduction (i.e., learning) in the FEP task between the first and second trials was negatively correlated with schizotypal trait and the number of perseveration errors in the WCST. Furthermore, changes in the PFC hemodynamic activity between the first and second trials were positively correlated with error reduction in the FEP task between the first and second trials, and negatively correlated with the number of perseveration errors in the WCST. These results suggest that learning in the FEP task requires PFC activation, which is negatively associated with perseveration errors in the WCST. The results further suggest that the FEP task, in conjunction with near-infrared spectroscopy, may be useful as a diagnostic method for various disorders with PFC dysfunction.