Several therapeutic regimens, including neoadjuvant chemoradiation therapy (NACRT), have been reported to serve as anticancer immune effectors. However, there remain insufficient data regarding the immune response after NACRT in pancreatic ductal adenocarcinoma (PDAC) patients. Data from 40 PDAC patients that underwent surgical resection after NACRT (NACRT group) and 30 PDAC patients that underwent upfront surgery (US group) were analyzed to examine alterations in immune cell counts/distribution using a multiplexed fluorescent immunohistochemistry system. All immune cells were more abundant in the cancer stroma than in the cancer cell nest regardless of preoperative therapy. Although the stromal counts of CD4+ T cells, CD20+ B cells, and Foxp3+ T cells in the NACRT group were drastically decreased in comparison with those of the US group, counts of these cell types in the cancer cell nest were not significantly different between the two groups. In contrast, CD204+ macrophage counts in the cancer stroma were similar between the NACRT and US groups, while those in the cancer cell nests were significantly reduced in the NACRT group. Following multivariate analysis, only a high CD204+ macrophage count in the cancer cell nest remained an independent predictor of shorter relapse-free survival (odds ratio = 2.37; P = 0.033). NACRT for PDAC decreased overall immune cell counts, but these changes were heterogeneous within the cancer cell nests and cancer stroma. The CD204+ macrophage count in the cancer cell nest is an independent predictor of early disease recurrence in PDAC patients after NACRT.