Raman spectra are molecular structure-specific and hence are employed in applications requiring chemical identification. The advent of efficient handheld and smartphone-based Raman instruments is promoting widespread applications of the technique, which often involve less trained end users. Software modules that enable spectral library searches based on spectral pattern matching is an essential part of such applications. The Raman spectrum recorded by end users will naturally have varying levels of signal to noise (SN), baseline fluctuations, etc., depending on the sample environment. Further, in biological, forensic, food, pharmaceuticals, etc., fields where a vast amount of Raman spectral data is generated, careful removal of background is often impossible. In other words, a 100% match between the library spectrum and user input cannot be often guaranteed or expected. Often, such influences are discounted upon developing mathematical methods for general applications. In this manuscript, we carefully examine how such effects would determine the results of spectral similarity-based library search. We show that several popular mathematical spectral matching approaches give incorrect results under the influence of small changes in the baseline and/or the noise. We also discuss the points to be carefully considered while generating a spectral library. We believe our results will be a guiding note for developing applications of Raman spectroscopy that uses a standard spectral library and mathematical spectral matching.