The Smith equivalence of real representations of a finite group has been studied by many mathematicians, e.g. J. Milnor, T. Petrie, S. Cappell-J. Shaneson, K. Pawalowski-R. Solomon. For a given finite group, let the primary Smith set of the group be the subset of real representation ring consisting of all differences of pairs of prime matched, Smith equivalent representations. The primary Smith set was rarely determined for a nonperfect group G besides the case where the primary Smith set is trivial. In this paper we determine the primary Smith set of an arbitrary Oliver group such that a Sylow 2-subgroup is normal and the nilquotient is isomorphic to the direct product of a finite number of cyclic groups of order 2 or 3. In particular, we answer to a problem posed by T. Sumi.