In recent studies, it has been revealed that robots can form concepts and understand the meanings of words through inference. The key idea underlying these studies is "multimodal categorization" of a robot's experience. However, previous studies considered only nonhierarchical categorization methods, which led to nonhierarchical concept structures. Our concepts have a hierarchical structure, thus ensuring that the resulting inferences are more efficient and accurate. In this paper, we propose a novel hierarchical categorization method. The method involves extending multimodal latent Dirichlet allocation (MLDA) to hierarchical MLDA using the nested Chinese restaurant process, which makes it possible for robots to acquire concepts in a hierarchical structure. We show that a robot can form a hierarchical concept structure based on self-obtained multimodal information. Moreover, by focusing on the common features of each category in the hierarchy, the robot is able to infer unobserved information including word meanings.