In the last few months, the Covidence systematic review software (Veritas Health Innovation, Melbourne, Australia) has updated its data extraction module, offering users additional flexibility in the case of many of its features in Extraction 2.0. At least for the time being, users still have the ability to revert from the now 2.0 default back to Extraction 1.0 in the Settings if they so choose.

What’s involved in Systematic Review data extraction, in general?

Before discussing the Covidence Extraction 2.0 changes, it’s good to review some best practice standards for data extraction. The standards related to data extraction are mainly  intended to provide quality control to the extraction process by creating an environment where errors of omission and errors of inaccuracy can be minimized. From a PCORI presentation:

Relevant IOM Standards for Data Extraction
Standard 3.5: Manage data collection

• At a minimum, use two or more researchers, working independently to extract quantitative and other critical data from each study

• For other types of data, one person could extract data, while a second person independently checks for accuracy and completeness

• Establish a fair process for resolving discrepancies (do not give final decision making to the senior reviewer)

• Link publications from the same study to avoid including data more than once

• Use standard data extraction forms developed for the specific review

• Pilot-test the data extraction forms and process

Added flexibility and customization in Extraction 2.0 

Covidence Extraction 1.0 was best suited for systematic reviews where quantitative data would be extracted. As such, the default setting for the number of extractors was automatically locked in to two reviewers and could not be changed (in the way that the Title/Abstract Screen and Full Text Screen stages could be set to just one reviewer). Extraction 2.0 has made it possible to proceed with just one person carrying out the extraction. If two people are set to extract, consensus checking is still required. The comparison step, however, can be carried out as soon as both reviewers are done, at the individual study level.

Another noteworthy difference with Extraction 2.0 is that users now have the ability to begin designing the extraction form as soon as the Covidence review is created, even before any references have been added to the project. In other words, the extraction templates can be set up separate from the studies, something which benefits the process of pilot-testing the form/template. Also, whereas previously the addition of quantitative intervention outcome data automatically was configured into a table format and strictly followed a PICO framework, Extraction 2.0 recognizes that systematic reviews can vary and is less table intensive and more customizable in terms of the broader range of information that can be captured.

There are a couple of areas worth mentioning that have been intentionally left inflexible (most likely in the interest of better data quality/integrity). There is currently no way to restrict who on the team can be an extractor or do consensus – it is done by whoever gets there first. And once the data extraction has begun in Extraction 2.0, assigned roles cannot be re-assigned (so basically, only two people from the team can be involved in the data extraction stage). Also, once data extraction has begun, publications from the same study can be merged, however, this step cannot be undone.

Clearly, the Covidence Extraction 2.0 updates are still very much in line with the Standards for Data Extraction set out by the IOM (listed above). To learn more about the systematic review process and how to use Covidence, be sure to check out the MSK Library’s upcoming workshop schedule or Ask Us!

As preprints become more pervasive in biomedical research, many of us may be wondering:

“How will bibliographic indexes and biomedical literature databases like PubMed be handling the version control issue presented by preprints?”

Although the introduction of preprints into PubMed is still in its pilot test phase, enough time has now passed since it began in June 2020 for some of those preprint publications to have been officially published as peer-reviewed journal articles.

How does PubMed indexing work in general?

To get a better understanding of what you can expect to see in PubMed, it’s useful to know how PubMed indexing in general works (and has worked for many years). More than any other bibliographic index, PubMed does a terrific job of quality control. The reason for this can be attributed to their strict policy of only ever assigning one PMID (unique identifier) to an individual published item.

In other words, as a publication evolves from version to version, going from its “Online ahead of print” or “prepub” version that the publisher might make immediately available to readers on their website to the eventual “final published version”, only one PubMed record is created and one PMID is assigned for that item. The PubMed record actually tracks (in the case of many but not all publishers) the history of how this one published item is processed from the point of manuscript submission to its release into the public scientific record.

Take for example this item:

Robilotti EV, Babady NE, Mead PA, Rolling T, Perez-Johnston R, Bernardes M, Bogler Y, Caldararo M, Figueroa CJ, Glickman MS, Joanow A, Kaltsas A, Lee YJ, Lucca A, Mariano A, Morjaria S, Nawar T, Papanicolaou GA, Predmore J, Redelman-Sidi G, Schmidt E, Seo SK, Sepkowitz K, Shah MK, Wolchok JD, Hohl TM, Taur Y, Kamboj M. Determinants of COVID-19 disease severity in patients with cancer. Nat Med. 2020 Aug;26(8):1218-1223. doi: 10.1038/s41591-020-0979-0. Epub 2020 Jun 24. PMID: 32581323.

In the full PubMed catalog citation record for this item dates are included related to its interaction history with the publisher (i.e., the date the manuscript was received and accepted), plus dates for when the item first entered the NLM system and was indexed in PubMed and processed for inclusion in Medline. See:

EDAT- 2020/06/26 06:00

MHDA- 2020/08/28 06:00

CRDT- 2020/06/26 06:00

PHST- 2020/04/30 00:00 [received]

PHST- 2020/06/11 00:00 [accepted]

PHST- 2020/06/26 06:00 [pubmed]

PHST- 2020/08/28 06:00 [medline]

PHST- 2020/06/26 06:00 [entrez]

Managing the indexing in this way ensures better version and quality control, as all steps are tracked and applied to the same record (i.e., only ONE record is ever created for one published item).

Note: When things are not handled in this way – as is the case with some other database vendors – you often end up with two database records for the same item, particularly if the two versions appeared in different calendar years (for example if the prepub ahead-of-print appeared in December 2019 and the final published version appeared in March 2020) and the two records are “missed” (i.e., not identified as duplicates and purged) by the database producer.   

How will PubMed indexing work in the case of preprints?

Keeping in mind that dealing with preprints is still a work in progress for the National Library of Medicine (NLM) and that their cataloging policies may likely evolve as lessons are learned from their pilot – below is an overview of what PubMed has been doing so far with preprints.

PubMed is essentially handling preprints like other database vendors (that index conference proceedings) handle meeting abstracts. In the same way that there is no guarantee that research presented as a conference abstract will not be added to (data or otherwise) if and when it appears as a published peer-reviewed journal article, there is no way of ensuring that the preprint will be exactly the same informationally once it appears as a final, peer-reviewed article. And so, logically, one should assume that the preprint (which by definition has not yet undergone peer-review) will very likely undergo considerable improvement/change as it undergoes the peer-review process and is confirmed as such.

The folks at PubMed, therefore, are creating a separate database record for the preprint and a separate record for the related journal article, each record with its own unique PMID. And because the research reported in each may not be identical (even if they may have identical titles, one could be reporting on preliminary or partial data, etc.), the two records are not being connected via citation record linking, for example, in the way that a Retraction or a Comment might be. (The preprint record, however, will appear in the results of the “Similar Articles” search algorithm and so may be brought to the readers’ attention in that way.)

The preprint citation record related to the citation above, therefore, is a unique one and looks like this in PubMed:

For more information on preprints and preprint citation records – be sure to Ask Us at the MSK Library! 

As anyone who has worked on a systematic review (SR) knows, screening references for the study selection stage of the SR process can be quite time consuming and labor intensive. Ideally, the screening should be done by two people working independently, so it is a lot of work – times two! It’s not surprising, therefore, that many researchers wonder:

• if they can get away with single screening
• if there exists some way to automate part, or all, of the screening stage

Single Screening vs. Double Screening

An August 2020 paper by Mahtani et al. explores the latest evidence on this topic (see some examples listed below) and summarizes the guidance from leading evidence synthesis organizations/producers like the Cochrane Collaboration, the Joanna Briggs Institute, the Campbell Collaboration, and the Institute of Medicine (US) Committee on Standards for Systematic Reviews of Comparative Effectiveness Research – all of whom recommend (in their handbooks and documentation) that at least two people working independently be involved in the screening process.

Mahtani KR, Heneghan C, Aronson J. Single screening or double screening for study selection in systematic reviews? BMJ Evid Based Med. 2020 Aug;25(4):149-150. doi: 10.1136/bmjebm-2019-111269. Epub 2019 Nov 13. PMID: 31722997. 

Waffenschmidt S, Knelangen M, Sieben W, Bühn S, Pieper D. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Med Res Methodol. 2019 Jun 28;19(1):132. doi: 10.1186/s12874-019-0782-0. PMID: 31253092; PMCID: PMC6599339. 

Edwards P, Clarke M, DiGuiseppi C, Pratap S, Roberts I, Wentz R. Identification of randomized controlled trials in systematic reviews: accuracy and reliability of screening records. Stat Med. 2002 Jun 15;21(11):1635-40. doi: 10.1002/sim.1190. PMID: 12111924. 

Conventional vs. Automated or Semi-Automated Screening

Quite a bit of research is currently being done on automating steps of the systematic review process, particularly investigating using AI/machine learning or text mining/natural language processing to replace the second reviewer (ie. semi-automated screening) and/or to reduce the number of records needed to be screened. There are already software tools in existence that have introduced relevance prediction/screening prioritization capabilities (for example, Abstrackr, DistillerSR/DistillerAI, EPPI-Reviewer, RobotAnalyst, etc.) but their performance is largely still under evaluation.

As technology improves, it’s highly likely that we will someday soon see acceptance of automated screening tool use for study selection in systematic reviews by leaders in the evidence synthesis field, but we are still far from there yet.  Progress in this area is already being made, however, as demonstrated by the creation and efforts of the International Collaboration for the Automation of Systematic Reviews (ICASR):

Beller E, Clark J, Tsafnat G, Adams C, Diehl H, Lund H, Ouzzani M, Thayer K, Thomas J, Turner T, Xia J, Robinson K, Glasziou P; founding members of the ICASR group. Making progress with the automation of systematic reviews: principles of the International Collaboration for the Automation of Systematic Reviews (ICASR). Syst Rev. 2018 May 19;7(1):77. doi: 10.1186/s13643-018-0740-7. PMID: 29778096; PMCID: PMC5960503.

O’Connor AM, Glasziou P, Taylor M, Thomas J, Spijker R, Wolfe MS. A focus on cross-purpose tools, automated recognition of study design in multiple disciplines, and evaluation of automation tools: a summary of significant discussions at the fourth meeting of the International Collaboration for Automation of Systematic Reviews (ICASR). Syst Rev. 2020 May 4;9(1):100. doi: 10.1186/s13643-020-01351-4. PMID: 32366302; PMCID: PMC7199360.

Be sure to check out the MSK Library’s Systematic Review Service LibGuide or Ask Us for more information if you are thinking about embarking on a systematic review project.