This paper presents a novel method for image sentiment analysis called a deep metric network via heterogeneous semantics (DMN-HS). The contribution of the proposed method is introduction of the image captioning into image sentiment analysis to reflect a global impression that cannot be represented by classical visual features extracted from images. In order to consider a sentiment correlation between visual and captioning features, the proposed method newly designs a network to integrate these heterogeneous semantics features (HS features). Furthermore, with consideration of relations among sentiments based on the HS features, the proposed method constructs a sentiment latent space by introducing the center loss concerning relationships between different sentiments and enables the classification of image sentiments. From experimental results, the performance improvement via DMN-HS is confirmed.