Facial attractiveness can be judged on the basis of visual information acquired in a very short duration, but the absolute level of attractiveness changes depending on the duration of the observation. However, how information from individual facial parts contributes to the judgment of whole-face attractiveness is unknown. In the current study, we examined how contributions of facial parts to the judgment of whole-face attractiveness would change over time. In separate sessions, participants evaluated the attractiveness of whole faces, as well as of the eyes, nose, and mouth after observing them for 20, 100, and 1,000 ms. Correlation and multiple regression analyses indicated that the eyes made a consistently high contribution to whole-face attractiveness, even with an observation duration of 20 ms, whereas the contribution of other facial parts increased as the observation duration grew longer. When the eyes were averted, the attractiveness ratings for the whole face were decreased marginally. In addition, the contribution advantage of the eyes at the 20-ms observation duration was diminished. We interpret these results to indicate that (a) eye gaze signals social attractiveness at the early stage (perhaps in combination with emotional expression), (b) other facial parts start contributing to the judgment of whole-face attractiveness by forming aesthetic attractiveness, and (c) there is a dynamic interplay between social and aesthetic attractiveness.