In this paper, we describe, first, the present status and the tasks toward the future of energy systems in Japan from the viewpoint of 3E+S. 3E+S means energy security, environmental conservation, economic efficiency, and safety. Second, we describe the future energy society in 2030 and 2050, respectively. We introduce a large picture of the total energy system considering the energy resources, conversion, transportation, storage, consumption, control, and system architecture. For the image of 2030, the Japanese government has already announced the middle-term (2030) energy demand and supply on July 16, 2015. For 2050, we assume three cases regarding the renewable energy installations: the status quo of 2030 covering 22-24% of the total consumption, and 50 and 100% of the total consumption. In the last section, we describe smart technologies to support the envisioned society in 2030 and 2050. We also introduce the reference architecture model. To achieve 80% reduction of greenhouse gas (GHG) emission in 2050, the innovations at the consumer side, such as the large-scale renewable energy resources and storage installations, utilization of electrified vehicles, distributed energy management systems including virtual power plants, and system integration, will become very important in addition to innovations at the supply side. (c) 2017 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.