With the goal of mitigating the disasters that are associated with large earthquakes, various geophysical phenomena have been suggested as potential earthquake precursors. The theoretical explanations underlying these proposals are lacking in most cases, and these proposals are often supported by observational evidence in which the precursors are discovered through trial and error. This heuristic approach requires researchers to simultaneously find precursory anomalies and the thresholds used to detect them in a single study. This approach requires the repeated use of statistical hypothesis tests. The theory of multiple hypothesis testing predicts that the repeated application of statistical tests increases the chance that we will obtain false positives. Several measures that can help avoid the identification of false positives have been recommended, but these measures are frequently not applied in studies of earthquake precursors. We argue that the inappropriate use of multiple hypothesis testing may have resulted in apparently successful discoveries of earthquake precursors. We recommend the reevaluation of earthquake precursors that have been identified using heuristic search methods.