Citation:
P. Dhillon, D. P. Foster, S. M. Kakade, and L. Ungar, A Risk Comparison of Ordinary Least Squares vs Ridge Regression. JMLR: ArXiv Report, 2013.
Abstract:
We compare the risk of ridge regression to a simple variant of ordinary least squares, in which one simply projects the data onto a finite dimensional subspace (as specified by a Principal Component Analysis) and then performs an ordinary (un-regularized) least squares regression in this subspace. This note shows that the risk of this ordinary least squares method is within a constant factor (namely 4) of the risk of ridge regression.See also: 2013