We have probed the crystalline electric-field ground states of pure vertical bar J = 7/2, J(z) = +/- 5/2 > as well as the anisotropic c-f hybridization in both valence fluctuating systems alpha- and beta-YbAlB4 by linear polarization dependence of angle-resolved core level photoemission spectroscopy. Interestingly, the small but distinct difference between alpha- and beta-YbAlB4 was found in the polar angle dependence of linear dichroism, indicating the difference in the anisotropy of c-f hybridization, which may be a key to understanding a heavy Fermi liquid state in alpha-YbAlB4 and a quantum critical state in beta-YbAlB4