Objective: To compare a wearable device, the Fitbit Versa (FV), to a validated portable single-channel EEG system across multiple nights in a naturalistic environment. Methods: Twenty participants (10 men and 10 women) aged 25–67 years were recruited for the present study. Study duration was 14 days during which participants were asked to wear the FV daily and nightly. The study intended to reproduce free-living conditions; thus, no guidelines for sleep or activity were imposed on the participants. A total of 138 person-nights, equivalent to 76,539 epochs, were used in the validation process. Sleep measures were compared between the FV and portable EEG using Bland-Altman plots, paired t-tests and epoch-by-epoch (EBE) analyses. Results: The FV showed no significant bias with the EEG for the global sleep measures time in bed (TIB) and total sleep time (TST), and for calculated sleep efficiency (cSE = [TST/TIB] x 100). The FV had 92.1% sensitivity, 54.1% specificity, and 88.5% accuracy with a Cohen's kappa of 0.41, but a prevalence- and bias adjusted kappa of 0.77. The predictive values for sleep (PVS; positive predictive value) and wakefulness (PVW; negative predictive value) were 95.0% and 42.0%, respectively. The FV showed significant bias compared to the portable EEG for time spent in specific sleep stages, for SE as provided by FV, for sleep onset latency, sleep period time, and wake after sleep onset. Conclusions: The consumer sleep tracker could be a useful tool for measuring sleep duration in longitudinal epidemiologic naturalistic studies albeit with some limitations in specificity.