Ridge logistic regression for classification : a comparison study

Abstract

Logistic regression (LR) is one of the most widely used multivariate statistical techniques for modelling dichotomous response variables. Although LR was originally used to determine a parsimonious model which best describes the potential relationship that may exist between the response variable and the set of explanatory variables, nowadays it has been found to be useful for classification purposes. The maximum likelihood estimation procedure in LR is known to be negatively affected when the data is characterized by any of the following scenarios: (i) multicollinearity, (ii) high-dimensionality, where the number of explanatory variables p is greater than the sample size n, and (iii) complete or quasi-complete separation, giving rise to unreliable or undefined parameter estimates. The Ridge estimator can be used in LR as a possible solution to the aforementioned scenarios. One of the greatest challenges in Ridge logistic regression (RLR) is that of finding the optimal shrinkage parameter. The use of cross-validation techniques to evaluate the value of the shrinkage parameter are explored in some detail. The focus in this dissertation is on the use of RLR as a classification method, and thus it will be compared to a popular classification technique, namely linear discriminant analysis (LDA). Unfortunately, classical LDA also fails in scenarios (i) and (ii) mentioned above. One possible solution is to apply dimension reduction techniques, such as Partial Least Squares. The performance of the classification methods will be explored by applying them on four real-life datasets having different characteristics.