We explore the region of small sea quark masses below m(PS)/m(V) 0.5 in two-flavor QCD using a mean-field improved clover quark action and an RG-improved gauge action at a similar or equal to 0.2 fm on 12(3) x 24 and 16(3) x 24 lattices. We find that instability of the standard BiCGStab algorithm at small quark masses can be mostly removed by the BiCGStab(DS-L) algorithm, which employs L-th minimal residual polynomials with a dynamical selection of L. We also find singular spikes of DeltaH in the HMC algorithm at moderate values of Deltatau. Nature of the spike is studied. We also study finite-size effects and chiral properties of meson masses.