We present a novel method that performs spatio temporal editing of 3-D multiparty interaction scenes for free-viewpoint browsing, from separately captured 3-D video data. The main idea is to first propose the augmented motion history volume (aMHV) for individual motion representation. Then, by modeling the correlations between different aMHVs, we can define a multiparty interaction dictionary, describing the spatiotemporal constraints for different types of multi-party interaction events. Finally, a constraint satisfaction and a global optimization method synthesize natural and continuous 3-D multiparty interaction scenes. Evaluations with real data demonstrate the effectiveness of our method.