Expectation Propagation (EP) extends belief propagation by approximating messages with expectations of statistics, in which users can choose the statistics. In this paper, we discuss how a choice of statistics affects accuracy of EP's estimates. We approximate estimation error of EP via perturbation analysis based on information geometry. By comparing the approximated estimation error. we show that adding statistics does not necessarily improve the accuracy of EP. A numerical example confirms validity of our analytical results.