Systematic reviews play a vital role in the updating of clinical practice guidelines (CPGs), legal and regulatory processes, like drug approvals as well as implementing and evaluating health care policies, informed decision-making, and addressing knowledge gaps.
It is important that systematic reviews are replicable, and their authors keep a record of decisions about what to include, how data was gathered and evaluated and any other decisions taken during the overall process. These reviews can be quite complicated to carry out, often requiring forming a diverse research team.
To ensure the reliability of the systematic review, the authors must follow strict protocols and reporting guidelines, such as PRISMA.
A systematic review protocol is a detailed plan that outlines the methods and procedures that will be followed during the systematic review process. It serves as a roadmap for conducting the review in a structured and transparent manner. The protocol includes information about the research question, the criteria for including and excluding studies, the search strategy, data extraction methods, quality assessment criteria, and plans for data synthesis and analysis. One of the most popular systematic review protocol registers is PROSPERO.
If you are conducting the research alone and you are incorporating only some elements of a systematic review process in your literature search and evaluation, most probably you are conducting a systemised review. This type of review is usually undertaken when working on dissertations and does not require forming a research team and registering the protocol.
Each step of the systematic review process requires careful planning and time for execution, hence conducting systematic reviews can be time consuming.
Before starting your review, assess whether a systematic review is the best method to answer your research question, perhaps there's another review type which may be more suitable for you.
Find out if your review will address a significant research gap and check if similar reviews have been conducted before or if any review protocols on similar subjects were registered, this will help you avoid duplicating previous work. You can check by searching PROSPERO, PubMed and Google Scholar.
If you find an existing review on your topic of choice, the next step is to evaluate its quality. You can do so by using AMSTAR-II, a tool that is used to critically appraise the methodology of systematic reviews, which can include randomised and/or non-randomized studies of healthcare interventions.
If a high-quality review exists but it is outdated (for example its search was conducted more than 5 years ago), perhaps you could follow up on it and update it. Additionally, evaluate whether you have sufficient time and resources for conducting the systematic review.
If you are conducting a systematic review search as part of your undergraduate or Master's research project or dissertation, the above steps may not be necessary.
Every systematic review begins with a well-defined research question, which should be clear, feasible, and relevant. Researchers often use frameworks like PICO (Population, Intervention, Comparison, Outcome), PEO (Population, Exposure, Outcome) or SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) to structure their questions. Following a framework helps in constructing an appropriate search query which will be later used in literature search.
Find out more about framing the research question.
Defining criteria for including or excluding articles is crucial. This step ensures that only relevant studies make it into the review. Clear guidelines prevent bias and ensure the quality of the findings. This can be presented visually in tabular format as shown below:
Typically inclusion criteria include demographic, clinical and/or geographic characteristics.
Exclusion criteria pertain to characteristics that are outside of the scope and aims of the review, for example persons with additional or multiple comorbidities.
Read more on inclusion and exclusion criteria.
Designing an effective search strategy is essential for the whole process. Researchers must search multiple databases, registers of trials and repositories using carefully chosen keywords to find published and unpublished studies. This step aims to capture a wide range of material related to the research question. It requires collecting all findings and deduplication of retrieved records prior to the initial screening.
If multiple databases are being searched, there will be a high percentage of overlapping studies, however you can use RefWorks to easily remove all duplicates.
Using a PRISMA diagram will come in handy to document all the steps within your review. Asking a librarian for assistance can help you make this process much smoother.
Once you gathered all findings and removed the duplicates it’s time to screen and select appropriate studies. Screening is typically a two-phase process. The first phase involves reviewing titles and abstracts. Studies which perfectly match your exclusion criteria can be eliminated, while others should be retained for further examination. The next step involves obtaining the full texts of the remaining studies and once again thoroughly evaluating them against the criteria previously established.
To minimise bias, it is recommended to involve at least two team members in this task. However as a student conducting your thesis, you will likely be working as a solo researcher for the literature review. Please remember that if you are handling the selection process alone, you may introduce your own bias into the decision-making process. In such cases fulfilling protocol requirements for a systematic review is not possible.
When excluding studies, it's important to maintain a detailed record of the decision-making process and display those studies which were excluded in the screening by full text stage in a table at the beginning of the results section.
Data extraction involves presenting all significant information about each study included in your review through the use of a standardised data extraction sheet. Cochrane provides an example of such a sheet which you can download as a Word document.
Data fields may include general study information like author, title, and publication year, as well as characteristics of the population/participants, details of the applied methodology, intervention descriptions, and outcome summaries.
Data included in the extraction sheet may vary from one systematic review to another and should be adjusted to a particular research question.
A risk of bias assessment, also known as quality assessment, is typically conducted for each study within the review to reduce biases and demonstrate the strength of the underlying evidence. This process also determines whether the included studies are thorough enough to inform decision making. Certain studies might exhibit biases in their findings due to design weaknesses, raising questions about their credibility. Quality/Risk of bias assessment is an essential part of the systematic review, which is rarely included in other types of reviews. It is recommended to use specific risk of bias assessment tools such as Cochrane Risk of Bias tool version 1 or version 2 or by using the CASP checklists for completing this step. Should you have any doubts on how to use these tools, please ask your lecturer/supervisor for assistance.
Find out more about other quality assessment tools
Meta-analysis is a statistical method that combines results from multiple separate studies. It may be included as a part of a systematic review, however it cannot be conducted as an individual study. When deciding on conducting this type of analysis, carefully following the steps of the systematic review process is essential. When planning on including the meta-analysis in the study it is important to decide which data is worth comparing and could produce meaningful outcomes.
Conducting a meta-analysis can improve precision of the overall study by combining data from multiple smaller studies. It can also answer questions which were not explored by individual studies and help in resolving disputes stemming from conflicting study results. Additionally, meta-analyses can generate new hypotheses for further research.
How to conduct a meta-analysis in eight steps: a practical guide.
Analysing data and undertaking meta-analyses - Cochrane Guidelines.
Meta-Analysis 001 - Video Guide.
Once the analysis is complete, it's time to complete the last step of the process and communicate the findings. This is usually done in the form of a comprehensive report detailing the review's process, findings, and conclusions. Then, the manuscript is submitted to a journal for publication where it undergoes a peer review process. Once accepted for publication, the manuscript becomes part of the scientific knowledge in the field.
Ariel de Lima, D., Helito, C. P., de Lima, L. L., Clazzer, R., Gonçalves, R. K., & de Camargo, O. P. (2022). How to perform a meta-analysis: a practical guide using R software and Rstudio. Acta Ortopedica Brasileira; Acta Ortop Bras, 30(3), e248775. https://doi.org/10.1590/1413-785220223003e248775
Chandler, T. (2023, Oct 31,). Steps to conducting a systematic review. University of Maryland. Retrieved Nov 7, 2023, from https://lib.guides.umd.edu/SR/steps
Choi, A. R., Cheng, D. L., & Greenberg, P. B. (2019). Twelve tips for medical students to conduct a systematic review. Medical Teacher; Med Teach, 41(4), 471-475. https://doi.org/10.1080/0142159X.2018.1426847
Deeks, J. J., Higgins, J. P., Altman, D. G., & Cochrane Statistical Methods Group. (2023). Chapter 10: Analysing data and undertaking meta-analyses. In J. Higgins, J. Thomas, J. Chandler, M. Cumpston, T. Li, M. J. Page & V. A. Welch (Eds.), Cochrane Handbook for Systematic Reviews of Interventions version 6.4. https://doi.org/https://training.cochrane.org/handbook/current/chapter-10
Emily Jones, M. (2023, Oct 17,). Systematic Reviews: Step 2: Develop a Protocol. University of North Carolina. Retrieved Nov 7, 2023, from https://guides.lib.unc.edu/systematic-reviews/protocol
Fu, Y. (2023, Feb 1,). Systematic Review Service: Steps in a Systematic Review. University of Maryland. Retrieved Nov 7, 2023, from https://guides.hshsl.umaryland.edu/c.php?g=94045&p=1595805
Krause, K. (2023, Jun 28,). Research Guides: Systematic Reviews: Introduction. MD Anderson Cancer Center. Retrieved Nov 7, 2023, from https://mdanderson.libguides.com/c.php?g=384755&p=3096433
Muka, T., Glisic, M., Milic, J., Verhoog, S., Bohlius, J., Bramer, W., Chowdhury, R., & Franco, O. H. (2020). A 24-step guide on how to design, conduct, and successfully publish a systematic review and meta-analysis in medical research. European Journal of Epidemiology; Eur J Epidemiol, 35(1), 49-60. https://doi.org/10.1007/s10654-019-00576-5
Patole, S. (2021). Principles and Practice of Systematic Reviews and Meta-Analysis (1st Edition 2021 ed.). Cham: Springer International Publishing AG. https://doi.org/10.1007/978-3-030-71921-0
Randa Lopez, M. (2023, Sep 11,). Systematic Reviews: Steps in a Systematic Review. Louisiana State University. Retrieved Nov 7, 2023, from https://guides.lib.lsu.edu/c.php?g=872965&p=6268540
Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., & Henry, D. A. (2017). AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ (Online); BMJ, 358, j4008. https://doi.org/10.1136/bmj.j4008
Tawfik, G. M., Dila, K. A. S., Mohamed, M. Y. F., Tam, D. N. H., Kien, N. D., Ahmed, A. M., & Huy, N. T. (2019). A step by step guide for conducting a systematic review and meta-analysis with simulation data. Tropical Medicine and Health; Trop Med Health, 47(1), 46. https://doi.org/10.1186/s41182-019-0165-6
Compiled by:
Agata Scicluna Derkowska - Senior Assistant Librarian, University of Malta.
Emanuel Schembri - Visiting lecturer at the Department of Physiotherapy, Faculty of Health Sciences, University of Malta.
Last updated: 17 Nov, 2023