A complex amplitude hologram can reconstruct perfect light waves. However, as there are no spatial light modulators that are able to display complex amplitudes, we need to use amplitude, binary, or phase-only holograms. The images reconstructed from such holograms will deteriorate; to address this problem, iterative hologram optimization algorithms have been proposed. One of the iterative algorithms utilizes a blank area to help converge the optimization; however, the calculation time and memory usage involved increases. In this study, we propose to reduce the computational complexity and memory usage of the iterative optimization using scaled diffraction, which can calculate light propagation with different sampling pitches on a hologram plane and object plane. Scaled diffraction can introduce a virtual blank area without using physical memory. We further propose a combination of scaled diffraction-based optimization and conventional methods. The combination algorithm improves the quality of a reconstructed complex amplitude while accelerating optimization.