We investigate a possibility of primordial black hole (PBH) formation with a hierarchical mass spectrum in multiple phases of inflation. As an example, we find that one can simultaneously realize a mass spectrum which has recently attracted a lot of attention: stellar-mass PBHs ($\sim\mathcal{O}(10)M_\odot$) as a possible source of binary black holes detected by LIGO/Virgo collaboration, asteroid-mass ($\sim\mathcal{O}(10^{-12})M_\odot$) as a main component of dark matter, and earth-mass ($\sim\mathcal{O}(10^{-5})M_\odot$) as a source of ultrashort-timescale events in Optical Gravitational Lensing Experiment microlensing data. The recent refined de Sitter swampland conjecture may support such a multi-phase inflationary scenario with hierarchical mass PBHs as a transition signal of each inflationary phase.