Measuring extremely low magnetic signals of diamagnetic materials such as water and ethanol with high sensitivity is important in various fields. Ethanol is widely used as a solvent in areas such as chemical industry and drug discovery; therefore, it is important to measure its concentration with high sensitivity. We have already reported a hybrid-type magnetometer using a high temperature superconductor superconducting quantum interference device (HTS-SQUID), which can measure and evaluate magnetic characteristics by vibrating and rotating a sample. However, the signal-to-noise ratio (SNR) of the system needs to be improved as magnetic signals from diamagnetism in water and ethanol are extremely low in magnitude. In this study, digital frequency filters were applied to the signals measured by the HTS-SQUID magnetometer with a rotating sample container in order to improve the SNR of the system, and then the magnetic properties of ethanol and water were studied. As a result, the SNR of the system improved about twice the level obtained by conventional signal processing method.