Systematic Review

 

The key features of Systematic Review

Systematic review is particularly popular in the field of medical research, but has seen increasing use in other fields in recent years. It is the most complete form of literature review and aims to provide an objective summary of all publications to date that are relevant to answering a specific research question. That research question usually includes the PICO elements: Patient, Intervention, Control and Outcome. Sometimes Study design (S) is added to the research question. A different framework may be used for non-medical research questions. See here for more information on different types of research questions. 

 

A systematic review is also characterized by the use of a thorough and transparent method. It can be said that doing a systematic review is a study in itself, in which the publications found serve as research data.  Furthermore, the search strategy and choices for including or excluding publications are transparently recorded and justified in the research protocol and in the methods section of the paper, and the publications used are analyzed and reported in a meta-analysis. Not surprisingly, systematic review is also the most time-consuming type of literature review of the four discussed in this guide: you may need 1-3 years to complete a systematic review paper.

 

Before you begin

Before you dive into your systematic review, it is advisable to do an initial search in a large database (e.g. PubMed) for the topic you want to cover. First, to check whether enough has been published on the topic to write a review on it; if not, another type of literature search or another topic might be more appropriate. Second, it is important to know whether a systematic review on the topic already exists, or whether anyone is currently working on a systematic review on your topic but has not yet published it. Therefore, check platforms for pre-registering Systematic Reviews, for example INPLASY or PROSPERO. 

 

Does a systematic review on your topic already exist, or is one in the making? Then consider whether writing a new systematic review on this topic will add new insights: is the review recently published, but have new relevant studies been published in the meantime that were not included? Or is the topic the same but you want to focus on a different study population, for example? These could be reasons to go ahead with the systematic review anyway. If you are unsure of if you have questions, contact the Information Specialist of your research center. 

 

Also consider possible journals in which you would like to publish your systematic review: what are the author's guidelines, what standards are required? Does the journal have templates, checklists, etc.? It is advisable that you look into this early on in the process. 

 

Systematic review: what do you need?

  • Time: 1-3 years;
  • Team: writing a systematic review requires a collaboration between researchers, information specialists, at least 2 reviewers for screening the abstracts and papers, statisticians for analyzing the (meta)results, and a project leader to manage it all and write the final paper. 
  • A clearly formulated research question;
  • Selection criteria for your search strategy;
  • Systematic review protocol;
  • Guest access account (GVO) for access to University of Groningen databases;
  • Consultation with an Information Specialist (please complete this form for search strategy support);
  • Log for keeping track of your search strategy;
  • PRISMA Flow Diagram;
  • Reference tool such as RefWorks;

 

Determine your research question and associated search terms

You have probably already drafted a research question for your systematic review. It is important that this research question is clearly formulated and focused, and does not contain any ambiguities or vague terms. For suggestions, check out the page on composing the research question and the different frameworks you can use for this. 

 

Revisit all the terms in your question to make sure they are really specific enough. A thesaurus can help you with this, see also the page on finding keywords. It may also be useful to conduct a preliminary search in a large database to catch subject-specific synonyms or related terms. These subject-specific terms can give the right focus to the research question. 

 

Focusing the research question, finding synonyms and building the search string are obviously related. In practice you may find out that there are search terms missing or that your results contain a lot of  entries that are not very useful, while performing the initial searches. At this stage you can still adjust your research question and search string. You record the final version in the research protocol.

 

Determine which databases you want to consult

At its core, a Systematic Review is the most comprehensive form of literature search. This means that the goal is to include all works published to date in the search strategy - that is, regardless of the language in which it was written, the location where the study was conducted, whether it is a scientific article or, for example, a clinical trial registry, and regardless of whether it is a recent or old publication. As a result, the databases you choose to conduct your search should cover this entire spectrum. As a researcher at Hanze, you have access to about 160 databases; it is recommended that you check them all and make a top 5 or 10 of most relevant ones. Unfortunately, there is no magical formula for deciding which databases to include, but a combination of more general and subject-specific databases is often a sensible choice. 

 

A characteristic of conducting a systematic review is to include gray literature in your search. These are works that are not distributed in the "traditional way" (i.e., not through a publisher). Examples include dissertations, conference contributions, articles in non-scientific journals, clinical trial registrations, reports from government agencies, and newspaper articles. So consider carefully how to include gray literature in your search strategy as well. Suggestions for finding gray literature can be found on this page. 

 

Write the research protocol

The review protocol first describes the context and rationale for the review: why is it important to conduct this review? What do we already know about the topic and in what areas do we need more clarity? What is the exact research question? In addition, the protocol describes the choices made regarding the search strategy, the selection process, the extraction of the data, how the critical appraisal is conducted, and how the data is analyzed. The questions to ask here are: 

  • what are the selection criteria when conducting the search?
  • how is it determined whether a study is relevant or not?
  • How to proceed if researchers disagree on whether or not to include a study.

All in all, the protocol documents how you will proceed during the review. The protocol is established in advance; you will have to justify any deviation from the protocol in your final review paper. Note that writing up the protocol usually takes a long time, up to 18 months on average.


PRISMA has a checklist for preparing a protocol, PRISMA-P, found on the PRISMA Web site. The checklist provides an indication of how to format a protocol and what information it should contain. 

 

Pre-register your protocol

It is highly recommended that you register your protocol; this way, other researchers will know that you are already working on this topic and you will prevent others from writing a systematic review paper. PRISMA has a checklist for preparing a protocol, PRISMA-P, found on the PRISMA Web site. The checklist provides an indication of how to format a protocol and what information it should contain. 

 

Register your protocol

It is highly recommended that you register your protocol; this way, other researchers will know that you are already working on this topic and you will prevent others from writing a systematic review on your topic. Moreover, by doing so, you indicate exactly what you are going to do a priori, which increases the transparency of your research. Well-known platforms for registering systematic review protocols are PROSPERO, INPLASY and OSF. You can also use these platforms to find examples of registered protocols, for example on COVID-19.

 

 

Conduct your search

Once you have worked out your search strategy, it is time to conduct the search in the selected databases. Tips for customizing the search string based on the syntax of the particular database can be found here. Make sure that you keep track of the results of the searches in a log, and use a reference manager such as RefWorks. A user manual on RefWorks can be found here. 

 

Don't forget to fill in the PRISMA flow chart. Use it to document your final research results, the reasons for deleting results at the different stages of the review process (e.g. deduplication), and the number of results that are included in the final analysis. 

 

De-duplication

Expect to find duplicates in your results: articles that are retrieved as unique results from different databases. Before you start screening your results, you should remove all duplicates to avoid unnecessary extra work. It is advisable to use a tool such as RefWorks for this purpose. You can specify how rigid the tool should be in determining whether something is a duplication or not. If it is too strict, you will have to screen more articles and remove false negatives yourself; if it is not strict enough, some results may be removed as duplicates when they are not (false positives). 

 

Again, remember to record in your PRISMA flow chart how many duplicates you have removed and how many items remain for screening.

 

Screening the results

Screening the results is done in two steps: first, you will assess all results by title and abstract. What remains will be screened based on the full text. It is necessary to have at least two people carry out the screening process independently, to ensure the objectivity of the review. The inclusion and exclusion criteria are already defined in the review protocol. These criteria form the basis for the screening process. 

 

It is recommended to use a tool for this part of the literature search. Rayyan is an approved application that eases the screening process somewhat by allowing you to screen easily and independently. You indicate for each result whether it should, should not, or perhaps should be included, and for what reason. Once all screeners have finished screening, Rayyan checks which items you agree on. The items you do not (yet) agree on can be discussed, or are automatically included in the full-text screening (depending on what is included in the protocol). With three or more people, the "majority vote" principle can be more easily applied. Record the number of results removed in the PRISMA flow chart. 

 

Critical appraisal of the articles

During the critical appraisal you go through the article in a structured manner, carefully assessing the quality of the paper in question: what are the strengths and weaknesses of the paper: were the researchers thorough, is the analysis correct and is the conclusion based on the results? You do this by using a questionnaire or other critical appraisal tool. Note that a critical appraisal can be done at different levels depending on the type of study you are reviewing, for example, a primary research article, a review article, or a review or reviews article. Examples of critical appraisal tools can be found on the Joanna Briggs Institute website. 

 

A regular part of the critical appraisal is the risk of bias assessment, in which you assess the studies for the possibility that certain features of the research method lead to misleading results, for example by choosing a particular target population or outcome variable. Always use a thorough risk of bias tool for this component, such as the one provided by Cochrane Methods. 

 

Data extraction

Data extraction is the part in which the researcher meticulously maps out the characteristics of the included studies. Such an overview provides insight into the differences between the studies, for example, differences in effect size and/or what choices were made regarding method and analysis and can serve as justification for whether or not a study was included in the final analysis. The data extraction is often reported in a data matrix, see also under tab reporting.

 

 

Reporting your research
If you already have a (scientific) journal in mind where you would like to publish, then make sure to read the author's guide before you start writing. This way, you know in advance what the requirements are for a systematic review publication in the relevant journal (e.g. maximum number of words, structure).

 

For your inspiration, we have added some systematic reviews published by Hanze researchers in the list of resources below. All publications are open access available.

 

Research method and PRISMA flow chart
A systematic review is characterized by transparently reporting the search strategy. Sharing your search strategy enhances the transparency and reproducibility of your research.  Among other things, you report the databases, and the date(s) on which the final searches were conducted, as well as how many people performed the screening, the inclusion and exclusion criteria that were used, and how the quality of the papers was determined, including which instrument was used for quality assessment. 

 

The PRISMA flow chart is a standard visualization of your research method and may be included as a figure or as an appendix in your paper. 

 

Data matrix (reporting of data extraction)
A data matrix is an overview of the papers included in the analysis and clearly presents the key characteristics of the studies. The data matrix gives the reader insight into the degree of variability among the studies included in your review. See here for an example of a data matrix in a systematic review paper (Table 1).  

 

Results
Reporting the results can be done in several ways. You may choose to provide an overview of the number of studies, the quality of the studies and degree of agreement between the results for each theme. Ideally, you provide an overview of the relevant studies for each theme, summarizing the results for each study and presenting effect estimates and confidence intervals, for example in a forest plot.


A quantitative approach to reporting results is to do a meta-analysis, or analysis of the analyses of the included studies. An online guide on conducting meta-analyses can be found here.

Useful links and resources

 

Useful websites and library guides :

 

Video clips about systematic review:

 

Tools:

 

 

Open Access publications by Hanze researchers

  • Lakke, Sandra, et al. "The added value of therapist communication on the effect of physical therapy treatment in older adults; a systematic review and meta-analysis." Patient Education and Counseling 102.2 (2019): 253-265. DOI: 10.1016/j.pec.2018.09.020
  • Wagenaar, M. C., van der Putten, A. A. J., Douma, J. G., van der Schans, C., & Waninge, A. (2022). Definitions, signs, and symptoms of constipation in people with severe or profound Intellectual Disabilities: a systematic review. Heliyon 8(5):e09479. DOI: 10.1016/j.heliyon.2022.e09479
  • van der Wal-Huisman, H., Dons, K. S., Smilde, R., Heineman, E., & van Leeuwen, B. L. (2018). The effect of music on postoperative recovery in older patients: A systematic review. Journal of geriatric oncology, 9(6), 550-559. DOI: 10.1016/j.jgo.2018.03.010

 

If you have questions, please contact the Information Specialist Research of your research center, or go to support & contact for more information and advice. 

[anchornavigation]