MutT homologue 1 (MTH1) protects the nucleotide pool from oxidative stress by hydrolyzing oxidized nucleoside triphosphates and prevents their incorporation into DNA. Cancer cells are dependent on the MTH1 activity for survival due to the high-level of reactive oxygen species in cancer cells; therefore, MTH1 is considered to be a novel target for treatment of various cancers. Here, we show by X-ray crystallographic screening using an in-house cocktail library that α-mangostin, a natural xanthone from mangosteen pericarp, binds to the active site of MTH1. A subsequent inhibition assay revealed that 3-isomangostin, a cyclized derivative of α-mangostin, was the most potent MTH1 inhibitor, with an IC50 value of 0.052 μM. Detailed structural analyses of the MTH1-3-isomangostin complex showed the novel binding mode of 3-isomangostin. Our results demonstrate that X-ray crystallographic screening is useful for the lead discovery for MTH1, and suggest that 3-isomangostin would be an attractive chemical tool for the development of anticancer agents.