Discovering target and off-target effects of specific compounds is critical to drug discovery and development. We generated a compendium of "chemical-genetic interaction" profiles by testing the collection of viable yeast haploid deletion mutants for hypersensitivity to 82 compounds and natural product extracts. To cluster compounds with a similar mode-of-action and to reveal insights into the cellular pathways and proteins affected, we applied both a hierarchical clustering and a factorgram method, which allows a gene or compound to be associated with more than one group. In particular, tamoxifen, a breast cancer therapeutic, was found to disrupt calcium homeostasis and phosphatidylserine (PS) was recognized as a target for papuamide B, a cytotoxic lipopeptide with anti-HIV activity. Further, the profile of crude extracts resembled that of its constituent purified natural product, enabling detailed classification of extract activity prior to purification. This compendium should serve as a valuable key for interpreting cellular effects of novel compounds with similar activities.