Heart rate variability (HRV) is the heart beat-to-beat variation under control of the cardiovascular function of animals. Under stressed conditions, cardiac activity is generally regulated with an upregulated sympathetic tone and withdrawal of vagal tone; thus, HRV monitoring can be a non-invasive technique to assess stress level in animals especially related to animal welfare. Among several stress-induced factors, heat stress is one of the most serious causes of physiological damage to animals. The aim of this study was to assess the effects of heat stress on HRV in small ruminants under free-moving conditions. In three experimental periods (June, August, and October), inter-beat intervals in sheep and goats (three for each) in two consecutive days were measured. HRV parameters were calculated from the inter-beat interval data by three types of analyses: time domain, frequency domain, and non-linear analyses. The temperature–humidity index (THI) was used as an indicator of heat stress, and vectorial dynamic body acceleration (VeDBA) was calculated to quantify the physical activity of the animals tested. First, we investigated correlations of THI and VeDBA with HRV parameters; subsequently, THI was divided into five categories according to the values obtained (≤ 65, 65–70, 70–75, 75–80, and >80), and the effects of the THI categories on HRV parameters were investigated with and without correcting for the effects of physical activity based on the VeDBA. The results indicated that HRV significantly decreased with increasing THI and VeDBA. For non-linear HRV parameters that were corrected for the effects of physical activity, it was suggested that there would be a threshold of THI around 80 that strongly affected HRV; high heat stress can affect the autonomic balance of animals non-linearly by inducing the sympathetic nervous system. In conclusion, to assess psychophysiological conditions of unrestrained animals by HRV analysis, the confounding effect of physical activity on HRV should be minimized for a more precise interpretation of the results.