Edge computing is a promising computing paradigm that satisfies QoS requirements of delay-sensitive applications. In edge computing, server migration control is indispensable for managing client mobility. As a server migration method for edge computing, the method based on Q-learning has been proposed. However, the method assumes that there is only one application client and the number of destination edge servers is limited to one. In this paper, we propose a server migration method using Q-learning that copes with realistic situations where there are multiple application clients and destination edge servers. The contributions of this paper are as follows: 1) we clarify that, under the situation with multiple application clients and multiple destination edge servers, a straightforward server migration method using Q-learning (RL method) does not scale due to state space explosion, and 2) we propose a server migration method using Q-learning (RL-DR method) that reduces the dimensionality of state space by abstracting the numbers of application clients at all locations into a center of the gravity (COG) of application clients. The simulation results show that 1) RL method shows up to 248% worse performance than conventional server migration methods because of state space explosion and 2) RL-DR method achieves up to 38.3% better performance than the conventional methods.