An evaluation of Ridge regression as a possible solution to multicollinearity problems

Abstract

The most popular regression technique used to estimate regression coefficients is the Ordinary Least Squares (OLS} method. However, in cases when multicollinearity is present, the Ordinary Least Squares is said to perform badly and is often characterised by instabilities. In this dissertation, multicollinearity and a number of diagnostics for identifying its presence and its degree are examined. An alternative method to the OLS which is suggested in literature, namely the Ridge Regression technique, is studied. Its performance will be compared to that of the Ordinary Least Squares, in cases of linear dependencies. Three datasets obtained from online data libraries will are analysed. Firstly, the degree of multicollinearity present in each dataset is investigated. Both Ordinary Least Squares and Ridge Regression methods are fit to each dataset, and comparisons are carried out in order to choose the best regression method for each dataset, using established criteria. Weakness of the Ridge Regression technique are identified and discussed. Although Ridge Regression offers a better interpretation of the parameter estimates and reduces the Mean Square Error in the presence of multicollinearity, the estimation of the ridge parameter estimate is very subjective, which makes this technique somewhat an inefficient regression method.