Purpose: The therapeutic effect of systemic treatment for breast cancer (BC) generally depends on its intrinsic subtypes. In addition, tumor infiltrating lymphocytes (TILs) are considered to be an independent factor for tumor shrinkage and disease prognosis. High TILs at baseline or after primary systemic chemotherapy are reported to be associated with better survival in triple-negative or human epithelial growth factor receptor 2 (HER2)-positive BCs. However, the prognostic value of TILs in estrogen receptor (ER)-positive and HER2-negative (ER+/HER2−) BC is still controversial. Methods: We assessed TIL score (low, intermediate, and high) before and after primary systemic chemotherapy in every subtype of BC, and compared the clinical outcomes. Biopsy specimens of 47 triple-negative, 58 HER2+ and 91 ER+/HER2− BCs were used to assess TILs before treatment. To assess TILs after treatment, we examined residual invasive carcinoma in surgically resected samples of 28 triple-negative, 30 HER2+ and 80 ER+/HER2− BCs. Results: A high TIL score in triple-negative BC before treatment resulted in a significantly higher proportion of pathological complete response (pCR). In contrast, ER+/HER2− BC exhibited fewer instances of pCR than other subtypes. Although not statistically significant, ER+/HER2− cases with a high TIL score also tended to achieve pCR (p = 0.088). Moreover, we revealed that low TIL BCs after chemotherapy, but not at baseline, had significantly better relapse-free survival in ER+/HER2− BC (p = 0.034). Conclusion: Pathological examination of TILs after treatment may be a surrogate marker for prognosis in ER+/HER2− BC.