In this paper we give a delay dependent stability criterion for stochastic delay systems with polynomial nonlinearity. An important benefit of this result is that when an invariant set of the system is available, we can incorporate this information to reduce the conservativeness. The resulting condition is given in terms of a semi-algebraic problem which is known to be efficiently solvable via sums of squares (SOS) relaxations. This work is originally motivated by the design of quantum spin control systems in the face of feedback delays. The effectiveness of the proposed method is evaluated by designing a globally stabilizing control law for the spin-1/2 system.