Bifidobacteria are among the first colonizers of the infant gut, and human milk oligosaccharides (HMOs) in breastmilk are instrumental for the formation of a bifidobacteria-rich microbiota. However, little is known about the assembly of bifidobacterial communities. Here, by applying assembly theory to a community of four representative infant-gut associated Bifidobacterium species that employ varied strategies for HMO consumption, we show that arrival order and sugar consumption phenotypes significantly affected community formation. Bifidobacterium bifidum and Bifidobacterium longum subsp. infantis, two avid HMO consumers, dominate through inhibitory priority effects. On the other hand, Bifidobacterium breve, a species with limited HMO-utilization ability, can benefit from facilitative priority effects and dominates by utilizing fucose, an HMO degradant not utilized by the other bifidobacterial species. Analysis of publicly available breastfed infant faecal metagenome data showed that the observed trends for B. breve were consistent with our in vitro data, suggesting that priority effects may have contributed to its dominance. Our study highlights the importance and history dependency of initial community assembly and its implications for the maturation trajectory of the infant gut microbiota.