We propose an innovative learning algorithm for enhancing the generalization ability of support vector machines (SVMs), when the Gausssian radial basis function (RBF) is used and when the parameter σ is very small. As learning patterns it uses not only the prescribed learning patterns but also newly inserted patterns in their neighbourhoods. In spite of the inserted many patterns, the size of the proposed optimization problem can be reduced to be same as the original one by using the averaging method. Many simulation results show the effectiveness of the proposed algorithm. © 2005 IEEE.