Light-driven atropisomeric polymers containing azo and binaphthyl units with a conjugated structure were designed and synthesized. The polymers exhibit a glass transition temperature higher than 75 °C with thermal stability above 280 °C and form uniform and smooth thin films without grain boundaries. The trans-cis isomerization is efficiently reversible upon alternating photoirradiation with UV and visible light or with heat. The photoisomerization of the chiral polymer results in photomodulation of the ellipticity and optical rotation ([α] D 25), owing to a chiroptical switching behavior. The maximum changes in the dihedral angle and [α]D25 are 21% and 700°, respectively, from the initial state owing to a photoinduced molecular twisting motion of the binaphthyl moiety. The polymer also displays photoswitchable fluorescence with a maximum at 409 nm. A photoinduced change in the refractive index of the formed film is also observed after irradiation with linearly polarized light at 532 nm according to the anisotropic molecular orientation. These results suggest that atropisomeric polymers are potential candidates for light-driven chiroptical switches.