© 2019 Elsevier B.V. Magnets are important for studying magnetism and are commonly used as sample environment devices of neutron scattering instruments. However, the use of magnets was prevented at the time-of-flight direct-geometry neutron spectrometer 4SEASONS in Materials and Life Science Experimental Facility (MLF) at J-PARC, because the instrument is surrounded by iron components, and it is equipped with a piece of equipment operated with magnetic bearings. In the present work, we investigate the influence of stray magnetic fields from a superconducting magnet which is shared among neutron scattering instruments in MLF. Based on the investigation, we modified 4SEASONS such that it can be used with the magnet. Finally, we successfully performed the first inelastic neutron scattering experiments with applied magnetic fields using this sample environment.