Background

Systematic review methods are rigorous and transparent approaches used to answer research questions by identifying and critically appraising studies, synthesizing the body of evidence, and reporting results using a pre-specified process. These methods were originally developed in the field of evidence-based medicine and have since been adapted for conducting literature-based assessments in numerous fields of study and multiple review types, such as scoping reviews and systematic evidence maps. Increased objectivity and transparency are the key principles that have contributed to the widespread adoption of these methods. Literature flow diagrams are one of the critical reporting elements of frameworks and reporting standards [the PRISMA statement [1]] for conducting these reviews because they efficiently convey the literature search and screening process, document how many studies were excluded and why, and show the extent of the body of evidence that was identified for addressing the review’s specific research question.

Literature flow diagrams have historically been manually generated visual representations of the literature screening results, and while they add transparency in the reporting, this traditionally static format presents a number of limitations. These diagrams can be time-intensive for authors to generate and maintain, especially when developing them manually and when a review extends over a period of years and includes literature search updates. We have found that manually generated diagrams have greater potential to introduce errors into the study counts throughout the literature assessment, requiring additional time and effort from authors to quality control, correct, and maintain clear documentation to verify accurate study counts. These diagrams are limited to high-level information in the form of summary study counts which are not typically linked to the underlying studies. While literature flow diagrams should cite included studies (i.e., those relevant to addressing the research question), it can be difficult to match study counts to the actual underlying studies recorded at any particular step. Consequently, readers often cannot easily identify an individual study and track its specific screening results contributing to a lack of transparency that is contrary to the goals of systematic review methods. Although some recent tools and templates have been developed to aid in the creation of literature flow diagrams and reduce the effort required to draft them (e.g., Haddaway et al. [2]), resulting diagrams that link studies to underlying data (e.g., citation information) require coding, and Shiny app applications allow for manual entry of the information without interactive capabilities. In addition, DistillerSR provides a PRISMA flow diagram that tracks references that move through the various stages of the screening workflow, but does not provide an interactive display of reference information. In our work, we outline an approach that incorporates screening data to develop an interactive reference flow (I-REFF) diagram that is compliant with PRISMA reporting and does not require coding.

Literature flow diagrams can be better

Software and tools currently used to screen literature and visualize data have led to increased efficiency in screening steps in the past several years, and these tools have the potential to revolutionize the way literature flow diagrams are developed. We propose the Interactive REFerence Flow (I-REFF) approach (described in the supplemental material) to leverage these new and increasingly powerful tools in developing interactive literature flow diagrams that are populated from screening data and linked to underlying screening results. The I-REFF approach can increase efficiency during diagram development, help to minimize the potential for errors, and enhance transparency and accessibility. To demonstrate I-REFF, we converted a standard literature flow diagram to the I-REFF format using an example from the field of toxicology for which we had access to the underlying data. We used the literature flow diagram from the 2020 National Toxicology Program scoping review of Potential Human Health Effects Associated with Exposures to Neonicotinoid Pesticides to create an interactive diagram, linked to the screening results (original, static literature flow diagram: Fig. 1 in Boyd et al. [3]; updated, I-REFF diagram).

An interactive reference flow diagram has several advantages over static literature flow diagrams. Linking screening data to the literature flow diagram allows summary counts to be automatically calculated. Therefore, when a change in the summary counts occurs, for example, following a search and screening update, minimal effort is required to update the literature flow diagram. Furthermore, because the screening results are already linked to elements such as reference citations and URLs, greater detail and interactivity can be achieved without additional effort. These elements make the new flow diagram more transparent and much more informative for readers. While the structure and summary-level information of the visual are unchanged, interactive elements now allow readers to quickly and easily identify studies considered in the review. The I-REFF approach provides both the capability to develop a static format that is normally seen in publications, but also an interactive format via a link in the publication for the reader to interact, and search the references in an evaluation. With details of the review readily available, readers have a greater ability to check and confirm, re-create, or build upon the review.

Conclusions

It is time to move beyond manually generated, static literature flow diagrams as the standard in systematic review methods. The I-REFF approach for generating interactive literature flow diagrams is applicable across literature screening platforms and can be achieved with several visualization software programs. An intermediate data transformation step may be required to ensure screening data exists in a format or structure that meets the requirements of visualization software. Widely available tools, such as Microsoft Power Query for Excel or a KNIME workflow, can minimize the effort required for data transformation.

The impact and potential for these diagrams extend beyond what we have demonstrated with our example and include the following:

  1. 1.

    The approach of linking literature screening data to literature flow diagrams sets the stage for true automation in generating these diagrams in the future. By connecting visualization tools directly to literature screening platforms and databases, literature screening results could be viewed in a literature flow diagram in real time and without a separate data transformation step. This integration of tools would require collaboration with the tool developers.

  2. 2.

    The enhanced transparency of reporting reference information for every study considered in the review strengthens the merits of the systematic review methods. Readers can more efficiently examine, replicate, and expand upon a review.

The richness of information conveyed in interactive elements establishes a new way to explore systematic review results. In a single literature flow diagram, authors have the potential to communicate information about individual studies, ranging from study characteristics to study quality to URLs for in-depth data visualizations and more.