Onoceroids are a group of triterpenes biosynthesized from squalene or dioxidosqualene by cyclization from both termini. We previously identified a bifunctional triterpene/sesquarterpene cyclase (TC) that constructs a tetracyclic scaffold from tetraprenyl--curcumene (C-35) but a bicyclic scaffold from squalene (C-30) in the first reaction. TC also accepts the bicyclic intermediate as a substrate and generates tetracyclic and pentacyclic onoceroids in the second reaction. In this study, we analyzed the catalytic mechanism of an onoceroid synthase by using mutated enzymes. TCY167A produced an unnatural tricyclic triterpenol, but TCY167L, TCY167F, and TCY167W formed small quantities of tricyclic compounds, which suggested that the bulk size at Y167 contributed to termination of the cyclization of squalene at the bicyclic step. Our findings provide insight into the unique catalytic mechanism of TC, which triggers different cyclization modes depending on the substrate. These findings may facilitate the large-scale production of an onoceroid for which natural sources are limited.