We show that the current extragalactic gamma-ray background (EGB) measurement below 100 GeV sets an upper limit on EGB itself at very high energy (VHE) above 100 GeV. The limit is conservative for the electromagnetic cascade emission from VHE EGB interacting with the cosmic microwave-to-optical background radiation not to exceed the current EGB measurement. The cascade component fits the measured VHE EGB spectrum rather well. However, once we add the contribution from known source classes, the Fermi VHE EGB observation exceeds or even violates the limit, which is approximated as E(2)dN/dE < 4.5 x 10(-5) (E/100 GeV)(-0.7) MeV/cm(2)/s/sr. The upper limit above 100 GeV is useful in the future to probe the EGB origin and the new physics like axionlike particles and Lorentz-invariance violation.