We build a privacy-preserving system of linear regression protecting both input data secrecy and output privacy. Our system achieves those goals simultaneously via a novel combination of homomorphic encryption and differential privacy dedicated to linear regression and its variants (ridge, LASSO). Our system is proved scalable over cloud servers, and its efficiency is extensively checked by careful experiments.