We performed a fluctuation analysis of the Hubble Extremely Deep Field (XDF) at four optical wavelength bands and found large fluctuations that are significantly brighter than those expected for ordinary galaxies. Good cross-correlations with flat spectra are found down to ${0{^{\prime\prime}_{. } }2}$, indicating the existence of a spatial structure even at the ${0{^{\prime\prime}_{. } }2}$ scale. The detected auto- and cross-correlations provide a lower limit of 24 nW m−2 sr−1 for the absolute sky brightness at 700–900 nm, which is consistent with previous observations. We searched for candidate objects to explain the detected large fluctuation using the catalog of the Hubble Ultra Deep Field (UDF), and found that the surface number density of faint compact objects (FCOs) rapidly increases toward the faint end. Radial profiles of FCOs are indistinguishable from the point spread function (PSF), and the effective radius based on de Vaucouleur’s law is estimated to be smaller than ${0{^{\prime\prime}_{. } }02}$. The spectral energy densities (SEDs) of FCOs follow a power law at optical wavelengths, but show greater emission and structure at $\lambda \gt 1\, \mu$m. Assuming that the FCOs are the cause of the excess brightness and fluctuations, the faint magnitude limit is 34.9 mag for the F775W band, and the surface number density reaches 2.6 × 103 arcsec−2. Recent γ-ray observations require that the redshift of FCOs must be less than 0.1, if FCOs are the origin of the excess optical and infrared background. Assuming that FCOs consist of missing baryons, the mass and luminosity of a single FCO range from 102 to 103 solar units, and the mass-to-luminosity ratio is significantly lower than 1.0 solar unit. The maximum effective radius of an FCO is 4.7 pc. These results and the good correlation between the near-infrared and X-ray background indicate that FCOs could be powered by the gravitational energy associated with black holes.