Recently, an efficient reinforcement learning method has been proposed, in which the problem of approximating the value function is naturally decomposed into a number of sub-problems, each of which can be solved at small computational cost. While tins method certainly reduces the magnitude of temporal difference error, the value function may be overfitted to sampled data. To overcome this difficulty, we introduce a robust approximation to tins context. Computer experiments show that the value function learning by our method is much more robust than those by the previous methods.