Obesity has reached epidemic proportions worldwide. Obesity results in reduced serum testosterone levels, which causes many disorders in men. Lifestyle modifications (increased physical activity and calorie restriction) can increase serum testosterone levels. However, it is unknown whether increased physical activity or calorie restriction during lifestyle modifications has a greater effects on serum testosterone levels. Forty-one overweight and obese men completed a 12-week lifestyle modification program (aerobic exercise training and calorie restriction). We measured serum testosterone levels, the number of steps, and the total energy intake. We divided participants into two groups based on the median change In the number of steps (high or low physical activities) or that in calorie restriction (high or low calorie restrictions). After the program, serum testosterone levels were significantly increased. Serum testosterone levels in the high physical activity group were significantly higher than those in the low activity group. This effect was not observed between the groups based on calorie restriction levels. We found a significant positive correlation between the changes in serum testosterone levels and the number of steps. Our results suggested that an increase in physical activity greatly affected the increased serum testosterone levels in overweight and obese men during lifestyle modification.