This study deals with the formation control problem of swarm robots using position sensitive detector (PSD) proximity distance sensors based on light-emitting diodes (LEDs). These proximity distance sensors are lightweight and quickly responsive, and are expected to enhance the mobility and flexibility of swarm robots. However, as each sensor has a narrow detection angle, the formation control problem becomes more difficult than when wide-directional distance sensors (such as cameras and laser rangefinders) are used. To overcome this difficulty, we design a two-part motion controller that controls both position and attitude. The attitude controller is necessary for continuous detection of other robots through the narrow detection angles. The designed controller is distributed in the sense that it requires only information on measured values of each robot’s own sensors. Next, we derive an appropriate sensor arrangement (positions and detection angles) that achieves the desired formation pattern. Finally, the effectiveness of the proposed method is demonstrated in an experiment performed by six omni-wheeled robots equipped with LED-based PSD proximity distance sensors.