Regularisation in regression : the partial least squares approach

Abstract

Ordinary Least Squares (OLS) regression spearheaded by Gauss in the 18th century is a technique that is widely used to estimate parameter coefficients in regression. Throughout the years, data sets started getting larger in size, both in terms of observations and variables. In particular, areas such as spectrometry and gene studies tend to have data sets that consist of a large number of variables which often outnumber the number of observations. Such data sets are known as high-dimensional and estimation techniques such as the OLS tend to perform poorly and the results are either ill-conditioned or undefined. Thus, this paved way for regularisation. Amongst the many regularisation methods that exist, is the Partial Least Squares (PLS) regression method. In this dissertation, we will explain the statistical interpretation of the PLS model based on the "Krylov hypothesis" and explain how the latent variables, loadings and weights can be obtained from various agorithms. A very important component that needs be determined is the number of PLS components k. In view of this, a number of validation techniques will be discussed.