A study on sparse methods for PLS-DA

Abstract

The term Discriminant Analysis (DA) refers to a collection of multivariate statistical techniques used to classify entities into a number of pre-defined groups. DA techniques follow two main steps being the discrimination step and the classification step. The former step involves the formation of a boundary which maximizes separation between the groups considered. The latter step then uses the information obtained from the discrimination step to predict the group membership of any new entities. The main focus will be on Fisher's Linear Discriminant Analysis (LDA), which considers a linear boundary for separation between groups. LDA encounters a number of issues such as the presence of multicollinearity in the attributes when dealing with high-dimensional data, where the sample size n is smaller than the number of attributes p. A possible solution for this problem is to introduce regularization techniques such as Dimension Reduction methods (DR) that reduce the p-dimensional attributes to a lower q dimension, where q < p. Amongst the most popular of this group of methods is the Partial Least Squares (PLS) method, which extends LDA to a high dimensional setting. This hybrid method is known as PLSDA and it is the main protagonist of this study. Further modification on PLS-DA is considered through a concept known as sparsity. Sparsity in PLS-DA involves the application of penalization methods such as LASSO and Ridge Regression to shrink and select the most influential attributes, producing a technique known as Sparse Partial Least Squares Discriminant Analysis. There are two different sparse PLS-DA methods known as SPLSDA and sPLS-DA, which differ in the order of variable selection, dimension reduction and classification. We refer to them as Sparse Method 1 and Sparse Method 2, respectively. Various measures of the classification ability and parameter estimates chosen are discussed and applied to two real data sets to determine if sparsity improves classification ability and interpretability, and whether there is a difference in performance for both Sparse Methods.