The hammering test has been widely used for inspection of social infrastructures because of its accuracy and efficiency of operation. In order to automate and apply the method to actual inspection sites, it is important for the system to calibrate itself against various environmental noise sources, such as strong winds, that influence the accuracy of the acoustic diagnosis. In this paper, a boosting based diagnostic algorithm with an updation rule for the weak learners that constitute the whole defect detector is proposed. Both clean and defective hammering sounds are templatized in the frequency domain, and inner parameters of the weak learners are optimized by using the pair of frequency template vectors. Furthermore, a new updation rule for template vectors is proposed. In experiments, our method was applied to defect detection of concrete test-pieces having slant cracks, which are considered to be serious defects in inspection sites. From the results, the effectiveness of the proposed method was confirmed.