Measuring change in Maltese education : an impact evaluation of large-scale policy introduction on learning outcomes and the quality of education

Abstract

This research investigated the impact on the quality of education across the Maltese Islands of two broad-scale educational policy changes — the National Minimum Curriculum (1999) and For All Children to Succeed (2004). It did so by investigating change in student outcomes on a set of five benchmark tests held across the same period as the policy introductions. The initial stages considered a quality education framework (input – process – output – context) to underpin the key processes of the study and help define the educational functions being investigated. The main part of the analysis then established a timescale based on the NMC and FACTS polices before analysing changes in examination constructs and forms. Once the inputs, processes and contexts were structured, the research proceeded to investigate outputs using Junior Lyceum Entrance Examinations results data. The overlaying of time series analysis for each of these analytical processes from before and after the policy introductions allowed the determination of impact points and direction of effect. The research indicates that the general effect of the two policies had an indirect positive influence on student outcomes. More specifically, there was/were: i. an overall rate of improvement in student achievement scores before and after the introduction of the NMC and FACTS policies. ii. discrepancies between application of conjunctive criteria (used by the EAU) and a parallel analysis of the data using compensatory criteria suggest that the criteria applied to the Junior Lyceum Examinations would have skewed signs of progress in teaching and learning at classroom level. iii. an increase in the rate of change of achievement when comparing the decade before and after the introduction of the NMC. The main changes took place at the beginning and end of the decade with the period 2001 – 2008, showing very slight change when using the compensatory data sets. iv. English, and mathematics, had a disproportionately higher impact on student success rates that made the inclusion of the other three exams practically irrelevant. v. The Junior Lyceum Examination set as a whole lacked independent objectivity and as such could not be considered an effective benchmarking tool. The aim of the study was to evaluate the effectiveness of the reforms on student learning, develop tools for monitoring change outcomes and identifying key factors driving the change. By analysing changes in student achievement and identifying the contributing factors, the study also sought to provide insights for future policy decisions and educational reforms.