As anyone who has worked on a systematic review (SR) knows, screening references for the study selection stage of the SR process can be quite time consuming and labor intensive. Ideally, the screening should be done by two people working independently, so it is a lot of work – times two! It’s not surprising, therefore, that many researchers wonder:
- if they can get away with single screening
- if there exists some way to automate part, or all, of the screening stage
Single Screening vs. Double Screening
An August 2020 paper by Mahtani et al. explores the latest evidence on this topic (see some examples listed below) and summarizes the guidance from leading evidence synthesis organizations/producers like the Cochrane Collaboration, the Joanna Briggs Institute, the Campbell Collaboration, and the Institute of Medicine (US) Committee on Standards for Systematic Reviews of Comparative Effectiveness Research – all of whom recommend (in their handbooks and documentation) that at least two people working independently be involved in the screening process.
Mahtani KR, Heneghan C, Aronson J. Single screening or double screening for study selection in systematic reviews? BMJ Evid Based Med. 2020 Aug;25(4):149-150. doi: 10.1136/bmjebm-2019-111269. Epub 2019 Nov 13. PMID: 31722997.
Waffenschmidt S, Knelangen M, Sieben W, Bühn S, Pieper D. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Med Res Methodol. 2019 Jun 28;19(1):132. doi: 10.1186/s12874-019-0782-0. PMID: 31253092; PMCID: PMC6599339.
Edwards P, Clarke M, DiGuiseppi C, Pratap S, Roberts I, Wentz R. Identification of randomized controlled trials in systematic reviews: accuracy and reliability of screening records. Stat Med. 2002 Jun 15;21(11):1635-40. doi: 10.1002/sim.1190. PMID: 12111924.
Conventional vs. Automated or Semi-Automated Screening
Quite a bit of research is currently being done on automating steps of the systematic review process, particularly investigating using AI/machine learning or text mining/natural language processing to replace the second reviewer (ie. semi-automated screening) and/or to reduce the number of records needed to be screened. There are already software tools in existence that have introduced relevance prediction/screening prioritization capabilities (for example, Abstrackr, DistillerSR/DistillerAI, EPPI-Reviewer, RobotAnalyst, etc.) but their performance is largely still under evaluation.
As technology improves, it’s highly likely that we will someday soon see acceptance of automated screening tool use for study selection in systematic reviews by leaders in the evidence synthesis field, but we are still far from there yet. Progress in this area is already being made, however, as demonstrated by the creation and efforts of the International Collaboration for the Automation of Systematic Reviews (ICASR):
Beller E, Clark J, Tsafnat G, Adams C, Diehl H, Lund H, Ouzzani M, Thayer K, Thomas J, Turner T, Xia J, Robinson K, Glasziou P; founding members of the ICASR group. Making progress with the automation of systematic reviews: principles of the International Collaboration for the Automation of Systematic Reviews (ICASR). Syst Rev. 2018 May 19;7(1):77. doi: 10.1186/s13643-018-0740-7. PMID: 29778096; PMCID: PMC5960503.
O’Connor AM, Glasziou P, Taylor M, Thomas J, Spijker R, Wolfe MS. A focus on cross-purpose tools, automated recognition of study design in multiple disciplines, and evaluation of automation tools: a summary of significant discussions at the fourth meeting of the International Collaboration for Automation of Systematic Reviews (ICASR). Syst Rev. 2020 May 4;9(1):100. doi: 10.1186/s13643-020-01351-4. PMID: 32366302; PMCID: PMC7199360.
Be sure to check out the MSK Library’s Systematic Review Service LibGuide or Ask Us for more information if you are thinking about embarking on a systematic review project.