Factor analysis is one of the most popular methods of multivariate statistical analysis. This technique has been widely used in the social and behavioral sciences to explore the covariance structure among observed variables in terms of a few unobservable variables. In maximum likelihood factor analysis, we often face a problem that the estimates of unique variances turn out to be zero or negative, which is called improper solutions. In order to overcome this difficulty, we employ a Bayesian approach by specifying a prior distribution for model parameters. A crucial issue in Bayesian factor analysis model is the choice of adjusted parameters including hyper-parameters for a prior distribution and also the number of factors. The selection of these parameters can be viewed as a model selection and evaluation problem. We derive an information criterion for evaluating a Bayesian factor analysis model. Our proposed procedure may be used for preventing the occurrence of improper solutions and also for choosing the appropriate number of factors. Monte Carlo simulations are conducted to investigate the efficiency of the proposed procedures.