1.1 Why a Review Team Needs to Understand AI Tools

By the middle of a review, a team has a question, a protocol, a database search and a plan for grey literature, and it has probably been told that an artificial intelligence (AI) tool could do much of the remaining work. Some of these tools help with parts of a review, and some of their outputs are wrong in ways that are hard to see. A reviewer who understands how the tools produce their answers can decide where they help and how to check them.

The running example in this course is the Cedar Valley evidence review, a fictional project. Before launching a community connector (social prescribing) program for older adults, the planning team of the fictional Cedar Valley Health Authority in British Columbia asked its small evidence team, made up of an evidence officer, a university librarian and a student intern, for a rapid scoping review and environmental scan within twelve weeks. The review asks which community-based interventions have been evaluated for reducing loneliness or social isolation among adults aged 65 and older, and with what outcomes. By this lesson the team has searched MEDLINE, Embase, CINAHL, PsycINFO and Web of Science (Lesson 4), retrieving 2,480 records, of which 1,870 remain after 610 duplicates are removed, and it has planned its grey-literature and web searching (Lesson 5).

Artificial intelligence is a broad label for computer systems that perform tasks usually associated with human judgement. Machine learning is the part of AI in which a system learns patterns from examples, as the screening tools in Section 4 do. A large language model is a generative AI system that produces text. HSCI 841 Lesson 12 complements this lesson by examining the use of large language models to code qualitative data.

1.2 How a Large Language Model Produces Text

Tokens and next-token prediction

A large language model works with tokens, which are short pieces of text such as a whole word, part of a word or a punctuation mark. A long or unusual word, such as a surname or a technical term, is usually split into several tokens. In training, the model reads a very large collection of text and learns to predict the next token in a passage from the tokens before it, storing what it learns as billions of numerical settings called parameters. Most current models use the transformer architecture introduced by Vaswani and colleagues (2017), which lets the model weigh every earlier token when it predicts the next one.

Developers then train the model further on examples of instructions and on human ratings of its answers, so that it follows requests and holds a conversation. When it answers, it repeats one step many times: it calculates a probability for every possible next token, selects one, adds it to the text and calculates again. A setting often called temperature controls how much randomness enters the selection, which is one reason the same prompt can produce different answers on different occasions.

What next-token prediction means for a reviewer

Four consequences follow for evidence work. First, a language model on its own has no catalogue of sources. What it knows about the literature is spread across its parameters as patterns, so when it writes a reference it produces text that resembles references it has seen. Second, the model's information stops at its knowledge cutoff, so recent studies are missing unless the tool can search. Third, because selection involves chance, a repeated prompt can give a different list of studies, which makes the output difficult to reproduce. Fourth, fluency and confidence in the text carry no information about accuracy, because the model produces fluent text whether or not the content is correct.

Bender and colleagues (2021) described large language models as systems that assemble plausible sequences of language without grounding in meaning, and argued that this creates risks when people treat the output as reliable. Reviewers need only recognize that a model's output is shaped by its training text and by the wording of the prompt. A prompt that asks for "studies showing that befriending reduces loneliness" invites a list of supportive studies, whereas a neutral prompt that asks what has been evaluated and with what results gives the model less to agree with.

1.3 Retrieval-Augmented Generation

Many AI research tools add a search step to the language model. Lewis and colleagues (2020) introduced the term retrieval-augmented generation for systems that combine a retriever, which finds relevant passages in a collection of documents, with a generator, which writes an answer using those passages. In a typical tool, the user's question is turned into a search, the retriever returns the passages it ranks highest from its index (the web, or a database of scholarly abstracts), those passages are placed in the model's context window, and the model writes an answer with links to the passages it drew on. Many conversational assistants available in 2026 can also run a web search during a conversation, which turns them into retrieval-augmented systems for that response.

What retrieval changes and what it leaves in place

Because each citation points to a document the retriever actually returned, a retrieval-augmented tool is much less likely to invent a reference outright. Other errors remain. The model can attach a claim to a source that does not support it, for example by turning a cautious conclusion into a confident one or by combining findings from two passages into a statement neither makes. The retriever can return weak, outdated or off-topic documents, and the model will summarize them with the same fluency as strong ones.

That last limit connects directly to Lesson 3. A bibliographic database running a Boolean search returns every record that matches the search, which is why reviews depend on it for high recall, the proportion of all relevant records that a search finds. A retrieval-augmented tool usually converts the question into an embedding, a list of numbers that represents its meaning, and returns the documents whose embeddings are closest. This approach, often called semantic search, can find relevant documents that use different words from the question, which is useful. It is still relevance ranking: the tool returns a top set of perhaps ten to a few dozen sources and has no notion of retrieving every eligible study. Its index also limits what it can find: many scholarly AI tools index abstracts from open sources and hold little of the content of Embase or CINAHL and little Canadian grey literature.

1.4 Why Fabricated and Mis-Attributed Citations Occur

References are among the most patterned text that exists. A reference list follows a predictable shape: surnames and initials, a year, a title with a colon, a journal name, a volume, an issue, a page range and a digital object identifier (DOI). A model trained on millions of reference lists learns that shape very well. Asked for sources without a retrieval step, it produces text in that shape from patterns associated with the topic. The surnames may belong to real researchers, the journal may be real and the title may read like a typical title, while the combination corresponds to no published work. Ji and colleagues (2023), in a survey of the problem across text-generation systems, describe this kind of output as hallucination: content that is fluent but unsupported by the source or by the facts.

Studies that tested chatbots in 2023 documented the problem. Walters and Wilder (2023) asked ChatGPT to write short literature reviews on 42 topics and checked the 636 references in the 84 reviews it produced. They found that 55 percent of the references produced by GPT-3.5 and 18 percent of those produced by GPT-4 were fabricated, and that 43 percent and 24 percent of the real references, respectively, contained substantive errors. Chelli and colleagues (2024) gave ChatGPT and Bard (later renamed Gemini) the inclusion criteria of eleven published systematic reviews on shoulder rotator cuff conditions and compared the references the tools returned with the references in the reviews. They classified 39.6 percent of the GPT-3.5 references, 28.6 percent of the GPT-4 references and 91.4 percent of the Bard references as hallucinated, and the GPT models found fewer than one in seven of the studies the human reviews had included. These figures describe the model versions tested at that time, and newer tools may perform differently. They show that fabrication fell with a newer model without disappearing.

Two features of these errors make them hard to see. They look exactly like correct references, so a reader cannot tell them apart by inspection, and they are often mixed in with real references, so a list that contains several familiar studies earns trust it has not earned. The only reliable check is to look each source up in an independent system and to read what it says, which is the procedure Section 3 sets out.

1.5 From Mechanism to Practice

The mechanism explains the rules the rest of the lesson applies. A generated reference is a lead to be checked. Retrieval-augmented tools rank and summarize a small top set, so they can supplement a documented database search without replacing it (Section 2). Outputs vary with the prompt, the model version and the date, so every use has to be recorded (Section 3). Machine learning can rank records well without making inclusion decisions, which is why ordering records for screening is its most established use in reviews (Section 4).

2.1 Categories Last Longer Than Products

AI research tools change quickly. Products are launched, renamed, merged, given new features and withdrawn within the span of a single degree program, and a guide that ranked individual products would be out of date before the term ended. This section therefore organizes tools into four categories by how they work. Product names appear only as examples available in 2026, and their presence in the text is no endorsement. A student who understands the categories can place a new tool into one of them, predict its strengths and weaknesses, and ask the right questions about it.

Two questions sort most tools. The first asks where the answer comes from: from patterns the model learned in training, from a search of an index of documents, or from the review team's own records and decisions. The second asks what the tool produces: written text, such as an answer or a summary, or a ranking or classification of records. Figure 6.2 places the four categories on these two dimensions.

2.2 The Four Categories

The tabs below describe each category in turn: how it works, examples available in 2026, the review tasks it can support and the limits that matter in a review.

The categories side by side

2.3 Why an AI Search Cannot Replace the Database Search

The difference between these tools and the search built in Lesson 4 is a difference of purpose. A systematic or scoping review promises its readers that the team has made a documented, reproducible attempt to find all of the eligible evidence. The database search keeps that promise because it is exhaustive within each database, because its full strategy can be published, and because another team can run it again and obtain essentially the same records from the same databases on the same date. An AI search engine is designed to give a busy user a good answer quickly, so it ranks, selects and summarizes. Each of those steps reduces recall and reproducibility. For this reason the AI tools in this lesson can add to a review's search and can check it, and the database search remains the backbone of the review.

The same reasoning explains where AI-found studies belong in reporting. PRISMA 2020 separates records identified from databases and registers from records identified by other methods, such as websites, organizations and citation searching. A study that an AI search engine finds, and that the team then confirms and screens, enters the review through the other-methods route, and the search that found it is documented in the same way as a web search (Lesson 5 and Section 3).

2.4 Evaluating a Tool You Have Not Seen Before

A new tool will appear during this term, and another will appear next year. The questions below turn the categories into an evaluation that a student can complete in a few minutes using the tool's own documentation. They echo the expectations that the 2025 joint position statement on AI in evidence synthesis, discussed in Section 3, sets for tool developers, which include publishing how a tool works, making evaluations of it available and stating its limitations and potential biases.

3.1 The Reviewer Remains Responsible

Every use of AI in a review rests on one principle: the authors of the review are responsible for its content, whatever tools they used. The 2025 position statement on AI use in evidence synthesis issued jointly by Cochrane, the Campbell Collaboration, JBI and the Collaboration for Environmental Evidence (Flemyng et al., 2025) states that evidence synthesists remain ultimately responsible for their work, including the decision to use AI, and that AI should be used with human oversight. The Committee on Publication Ethics (COPE) stated in 2023 that AI tools cannot be authors because they cannot take responsibility for the work. In practice, every source an AI tool suggests is checked against a primary source, and every use is recorded and reported.

3.2 A Four-Step Verification Procedure

A verification log is a table in which a reviewer records, for each AI-suggested source, the checks performed, the result and the action taken. The procedure below fills one row of the log for each source and takes a few minutes per reference.

Step 1, existence. Search for the exact title, in quotation marks, in a bibliographic database such as MEDLINE, in Google Scholar and in the library's discovery tool, and enter any DOI at doi.org or look it up in Crossref, the agency that registers most journal DOIs. A source that none of these systems can find is treated as fabricated. Step 2, details. Compare the authors, year, title, journal, volume, issue, pages and DOI with the publisher's record. Many articles appear online a year or more before they are assigned to an issue, so a difference between the two dates is common. Step 3, claim. Read the abstract and the relevant part of the full text, and decide whether the source says what the AI output says it says. Step 4, fit. Compare the source with the eligibility criteria, and check for a retraction or correction on the publisher's page or in the Retraction Watch database, which Crossref now makes openly available.

3.3 Worked Verification Exercise: The Cedar Valley Citation Check

This exercise uses AI output invented for teaching, imitating what a conversational assistant without web search can produce. It mixes real publications with fabricated and mis-attributed entries, which the answers identify. Do not cite the fabricated entries.

The team's verification log

Six of the eight sources exist, only three were correct in every detail and claim as given, and none is an eligible primary study for the scoping review. The tool did name several well-cited reviews that are useful for citation chasing. A reviewer who copied the list unchecked would have cited two articles that do not exist, attributed a confident finding to a review that judged the evidence insufficient, and given wrong details for two more.

3.4 Reproducibility and the AI-Assisted Search Log

Lesson 3 showed that web search engines personalize and re-rank results. AI tools add further variation: the model samples its output, providers update and retire models, the index behind a retrieval tool changes, and small changes in a prompt's wording can change the results. Because an AI-assisted search cannot be repeated exactly, the goal is a record full enough that a reader can see what was done, judge its influence and repeat it approximately.

This log sits with the web search documentation from Lesson 5 and the PRISMA-S log from Lesson 4. PRISMA 2020 also asks authors to report automation tools used in selecting studies and collecting data, which covers the tools in Section 4.

3.5 Privacy, Confidentiality and Copyright

Whatever is entered into an AI tool leaves the team's control to some degree. Depending on the provider's terms and the account, inputs may be stored, reviewed or used to train future models, and may be held in another country. Public bodies in British Columbia, including health authorities and universities, are subject to the Freedom of Information and Protection of Privacy Act and have policies on which AI tools staff and students may use. The team checks both institutions' policies before entering anything other than published material.

3.6 Disclosure

A disclosure statement tells readers which AI tools were used, for what purpose, at which stages, and how their output was checked. The RAISE recommendations (Responsible use of AI in evidence SynthEsis), developed through work involving Cochrane, the Campbell Collaboration, JBI and the Collaboration for Environmental Evidence, are emerging guidance on responsible AI use in evidence synthesis that was still being revised in 2025. The joint position statement that endorses them (Flemyng et al., 2025) asks authors to report fully any AI use that makes or suggests judgements, such as eligibility, risk of bias, extraction or synthesis decisions, giving each tool's name, version and dates of use, its purpose and the stages affected, the justification for using it, any related interests, and its limitations. Editing limited to spelling and grammar generally need not be reported, although journal policies vary. The template below covers these elements.

4.1 The Screening Workload

Title and abstract screening is usually the most time-consuming stage of a review. In the fictional Cedar Valley review, 1,870 records remain after de-duplication, and Lesson 7 describes how the evidence officer and the intern screen each one independently, which amounts to 3,740 decisions before any full text is read. Machine learning has been applied to this stage for longer than to any other, and its use here has the most evidence behind it.

Screening tools can use machine learning in three ways, and the distinction matters more than the choice of product. In prioritized screening, the tool changes the order in which records are shown, and the team still screens every record. With a model acting as a second screener, the model's prediction takes the place of one of two human reviewers. In screening truncation, the team stops screening before it has seen every record, because the model's ranking suggests that few relevant records remain. O'Mara-Eves and colleagues (2015), in a systematic review of text mining for identifying studies, concluded that using text mining to prioritize the order of screening should be considered safe and ready for use in live reviews, that its use as a second screener could be considered with caution, and that using it to eliminate studies automatically was promising but not yet fully proven.

4.2 How Active Learning Works

Active learning is a form of machine learning in which the model chooses which unlabelled record a human should label next, learns from that label, and chooses again. In screening, the reviewer first labels a few records as relevant or irrelevant. The tool converts each title and abstract into numerical features, such as word counts weighted by how distinctive the words are, or embeddings of the kind described in Section 1. A classifier, such as naive Bayes or logistic regression, learns from the labelled records which features predict relevance, and then scores every unscreened record. The tool shows the reviewer the record with the highest score. The reviewer decides, the model retrains on the larger set of decisions, and the cycle repeats.

ASReview, described by van de Schoot and colleagues (2021), is an open-source tool developed at Utrecht University that implements this cycle and lets the user choose among several feature extractors and classifiers. Its simulation mode replays a completed review whose decisions are known, showing how early the relevant records would have appeared. Screening platforms such as Rayyan, EPPI-Reviewer and DistillerSR offer comparable machine-learning features within their own workflows. A different kind of tool, the classifier trained once on a large labelled collection, is used for specific tasks. Thomas and colleagues (2021), for example, developed and evaluated a classifier that identifies reports of randomized controlled trials for Cochrane Reviews.

4.3 Measuring Performance: A Worked Cedar Valley Simulation

Three measures describe how well a screening tool performs. Recall, also called sensitivity, is the proportion of all relevant records that have been found at a given point. The proportion screened is the share of all records the reviewer has read at that point. Work saved over sampling, introduced by Cohen and colleagues (2006), compares the work needed to reach a target recall with the work needed to reach the same recall by screening in random order.

The work saved over sampling at 95 percent recall is (1,870 − 540) ÷ 1,870 − 0.05 = 0.711 − 0.05 = 0.661, or about 66 percent. In words, prioritized screening reached 95 percent recall after the reviewer had read 540 records, whereas random order would have required about 1,777 records (95 percent of 1,870) to reach the same recall on average. Figure 6.5 shows the same simulation as a curve.

4.4 The Stopping Problem

The simulation measures are calculated after the fact, when the team knows that 142 records are relevant. In a live review nobody knows the total, so nobody knows the recall at any moment. Prioritized screening that continues to the last record is safe, and the time saving from stopping early is where the risk lies. A stopping rule is a pre-specified criterion for ending screening before every record has been read.

Heuristic stopping rules are simple to apply. The most common stops after a fixed number of consecutive irrelevant records, such as 50 or 100, on the reasoning that a long run without a relevant record means few remain. Boetje and van de Schoot (2024) proposed the SAFE procedure, a practical and deliberately conservative combination of stopping heuristics for active-learning screening in tools such as ASReview. Statistical stopping rules take a different approach. Callaghan and Müller-Hansen (2020) proposed drawing a random sample from the records that remain unscreened and using the number of relevant records in the sample to test whether a recall target, such as 95 percent, has been reached with a stated level of confidence. Callaghan and colleagues (2024) later argued that stopping criteria for computer-assisted screening need to be well evaluated and transparently reported before reviews rely on them.

In the Cedar Valley simulation, a heuristic of 100 consecutive irrelevant records would have stopped screening at record 710, after the 138th relevant record had appeared at record 610. Stopping there would have saved reading 1,160 records, or 62.0 percent of the total, and would have missed 4 of the 142 relevant records (recall 97.2 percent). Suppose that one of the four missed records was a qualitative evaluation of a men's shed program whose abstract never used the words loneliness or isolation, and that it was later among the included studies. A missed record can matter even at high recall. Hard-to-find records usually describe the concept in unexpected language or have weak abstracts, and a scoping review of a varied public health literature most needs to catch them.

4.5 What the Evidence Shows

The evidence on active learning comes mainly from simulations on completed reviews. O'Mara-Eves and colleagues (2015) found that most of the studies in their systematic review suggested workload savings of between 30 and 70 percent, sometimes accompanied by the loss of 5 percent of relevant studies, which is 95 percent recall. They noted that studies rarely replicated each other, and that more evaluation was needed outside highly technical and clinical areas. Van de Schoot and colleagues (2021) reported simulation studies with ASReview in which relevant records were found after screening a fraction of the total. Hamel and colleagues (2021) evaluated active machine learning across ten completed systematic reviews and developed a seven-step framework for teams adopting it, which ends with guidance on truncating screening.

Several conditions limit how far these findings transfer to a new review. Performance varies with the topic, the share of records that are relevant, the clarity of the abstracts and the size of the dataset. Simulations treat the original human decisions as the truth, although human screeners also miss studies; Gartlehner and colleagues (2020) found in a randomized trial that single-reviewer abstract screening missed 13 percent of relevant studies. The joint position statement on AI in evidence synthesis (Flemyng et al., 2025) cites that finding as an example of the trade-offs a team should weigh, since in a rapid review an AI tool used as a second reviewer may catch studies that a single human reviewer would miss.

Evaluations of large language models for screening are newer and less settled. Khraisha and colleagues (2024) tested GPT-4 on title and abstract screening, full-text screening and data extraction across several literature types and languages. GPT-4's accuracy matched human performance on some tasks, but performance fell across all stages once the authors adjusted for chance agreement and for the imbalance between included and excluded records. The authors concluded that substantial caution was warranted, while noting that for certain tasks under specific conditions the model approached human performance.

4.6 When AI Assistance Is Acceptable: A Decision Table

The table below brings the lesson together. It is a teaching tool for this course that applies three principles found in the joint position statement and in the earlier sections: human oversight of every judgement, verification of every output against primary sources, and full disclosure of any AI use that makes or suggests judgements. Journals, funders, the university and the health authority may set stricter rules, and those rules take precedence.

Bringing It All Together

This lesson examined how artificial intelligence tools produce answers and what that means for a review team. Section 1 showed that a large language model writes text, including references, by predicting one token after another from patterns learned in training, which explains why fabricated and conflated references occur. Retrieval-augmented generation ties citations to retrieved documents and reduces fabrication, but its claims can still misstate their sources, and its relevance ranking returns a top set with unknown recall. Section 2 sorted AI research tools into conversational assistants, AI search engines and research assistants, citation-context tools and AI-assisted screening tools, and set out questions for evaluating any new tool.

Section 3 turned these ideas into practice. The fictional Cedar Valley team verified eight AI-suggested references with a four-step procedure and found that six existed, three were correct as given and two had been fabricated. The section introduced the AI-assisted search log, the privacy, confidentiality and copyright limits on what may be entered into a tool, and a disclosure statement built on the RAISE recommendations and the 2025 joint position statement. Section 4 explained active-learning screening, the measures used to evaluate it, the stopping problem and the evidence on performance, and ended with a decision table for judging AI use at each stage of a review.

The final reflection asks you to write an AI-use plan and disclosure statement for a new rapid review by the Cedar Valley evidence team.

Final Knowledge Assessment

Definitions of the terms, tools, frameworks and people introduced in this lesson.