© 2020, The Author(s). This paper describes computer vision-based quantitative microscopy and its application toward better understanding species specificity. An image dataset of the Lauraceae family that consists of nine species across six genera was investigated, and structural features were quantified using encoded local features implemented in a bag-of-features framework. Of the algorithms used for feature detection, the scale-invariant feature transform (SIFT) achieved the best performance in species discrimination. In the bag-of-features framework with the SIFT features, each image is represented by a histogram of codewords. The codewords were further analyzed by mapping them to each image to visualize the corresponding anatomical elements. From this analysis, we were able to classify and quantify the modes of aggregation of different combinations of cell elements based on clustered codewords. An analysis of the term frequency–inverse document frequency weights revealed that blob-based codewords are generally shared by all species, whereas corner-based codewords are more species specific.