A system of Brownian motions in one dimension all started from the origin and conditioned never to collide with each other in a given finite time interval [Formula presented] is studied. The spatial distribution of such vicious walkers can be described by using the repulsive eigenvalue statistics of random Hermitian matrices and it was shown that the present vicious walker model exhibits a transition from the Gaussian unitary ensemble (GUE) statistics to the Gaussian orthogonal ensemble (GOE) statistics as the time t goes on from 0 to T. In the present paper, we characterize this GUE-to-GOE transition by presenting the graphical expansion formula for the moments of positions of vicious walkers. In the GUE limit [Formula presented] only the ribbon graphs contribute and the problem is reduced to the classification of orientable surfaces by genus. Following the time evolution of the vicious walkers, however, the graphs with twisted ribbons, called Möbius graphs, increase their contribution to our expansion formula, and we have to deal with the topology of nonorientable surfaces. Application of the recent exact result of dynamical correlation functions yields closed expressions for the coefficients in the Möbius expansion using the Stirling numbers of the first kind. © 2003 The American Physical Society.