This cross-sectional study aimed to demonstrate the association between physical frailty subdomains and oral frailty. This study involved community-dwelling older adults (aged ≥65 years). Physical frailty was assessed with the Japanese version of the Cardiovascular Health Study criteria. Oral frailty was defined as limitations in at least three of six domains. Logistic regression analysis was used to analyze the association between physical frailty risk and oral frailty. In addition, we examined the association between physical frailty subdomains (gait speed, grip strength, exhaustion, low physical activity, and weight loss) and oral frailty. A total of 380 participants were recruited for this study. Overall, 18% and 14% of the participants were at risk of physical frailty and had oral frailty, respectively. Physical frailty risk (odds ratio (OR) = 2.40, 95% confidence interval (CI): 1.22–4.75, p = 0.012) was associated with oral frailty in multivariate analysis. In secondary analysis, among physical frailty subdomains, gait speed (OR = 0.85, 95% CI: 0.73–0.97, p = 0.019) was associated with oral frailty. The present findings suggest that physical frailty is closely related to oral frailty. Among physical frailty subdomains, decreased gait speed in particular is an important indicator related to the development of oral frailty.