For D = 4 theories of a single U(1) gauge field strength coupled to gravity and matters, we show that the electric-magnetic duality can be formulated as an invariance of the actions. The symmetry is associated with duality rotation acting directly on the gauge field. The rotation is constructed in flat space, and an extension to curved spaces is also given. It is non-local and non-covariant, yet generates off-shell extended transformation of the field strength. The algebraic condition of Gaillard and Zumino turns out to be a necessary and sufficient condition for the invariance of actions. It may be used as a guiding principle in constructing self-dual actions in string and field theories. © 1999 Elsevier Science B.V.