Video data is a sequence of video frames and their temporal continuity is an essential property of video stream. In this paper, we present a framework of constructing video continuity model from large-scale video archives with unsupervised learning. Our method estimates the similarity distribution of continuous frame pairs by applying simple assumption on the minimum duration of continuous video segments and then determines discontinuous frame pairs as outliers. The optimal estimation is pursued with measuring the separation between the estimated distribution of continuous frame pairs and that of discontinuous frame pairs. In order to verify the validity of the obtained model, the model is applied to the shot boundary detection. The results of experimental evaluation demonstrate the feasibility and the effectiveness of our method.