# Lesson 10: Scoping Reviews, Rapid Reviews and Qualitative Evidence Synthesis

*Companion-podcast transcript, Sarah and Kiffer*

---

**Sarah:** Welcome back to Office Hours. I'm Sarah.

**Kiffer:** And I'm Kiffer. This is the episode for Lesson ten of Health Sciences two forty-one. We are covering three kinds of review that students meet all the time: the scoping review, the rapid review and the qualitative evidence synthesis. Then we finish with the evidence and gap map, which is a way of drawing the whole body of evidence on one page.

**Sarah:** Let's start with the running case. Where is the Cedar Valley team now?

**Kiffer:** The Cedar Valley evidence review is our fictional project. A health authority in British Columbia is planning a community connector program, a form of social prescribing, for adults aged sixty-five and older, and it asked a small team for a rapid scoping review and an environmental scan within twelve weeks. The team now has forty-two included studies. Fourteen are randomized trials, ten are non-randomized controlled studies, nine are before-and-after studies without a control group, six are qualitative studies and three are mixed-methods studies.

**Sarah:** So section one. What exactly is a scoping review for?

**Kiffer:** A scoping review maps the extent, range and nature of the evidence on a topic. It asks what kinds of evidence exist, how the topic has been studied, and where the gaps are. Munn and colleagues listed six purposes, including identifying the types of evidence in a field, clarifying how a concept is defined, and identifying gaps in knowledge. A systematic review of effects usually asks a narrower question, whether an intervention works and by how much. The Cedar Valley planning team mainly wants to know which programs have been evaluated, with which measures and in which groups, which is a mapping question.

**Sarah:** I have heard people describe scoping reviews as a lighter version of a systematic review. Is that fair?

**Kiffer:** I would push back on that. A scoping review still needs a protocol, a documented search and explicit criteria applied by more than one person. What it usually leaves out is appraisal of each study and any synthesis of effects. The search is just as careful, because a map with a hole in it is misleading.

**Sarah:** Where did the method come from?

**Kiffer:** Hilary Arksey and Lisa O'Malley published the first framework in two thousand five. They described five stages. You identify the research question, identify relevant studies, select studies, chart the data, and then collate, summarize and report the results. There is also an optional sixth stage, a consultation exercise, in which people with practical knowledge of the topic suggest sources and comment on the findings.

**Sarah:** Arksey and O'Malley made that stage optional. Did anyone disagree?

**Kiffer:** Danielle Levac, Heather Colquhoun and Kelly O'Brien did, in two thousand ten. They recommended refinements at every stage. They argued that consultation should be essential, with a clear purpose. They also asked teams to link the purpose of the review to the question, to have two reviewers select studies independently, and to build the charting form together, with two people charting the first five to ten studies.

**Sarah:** Did consultation make any difference in Cedar Valley?

**Kiffer:** It did. The team consulted twice, in week two to settle the scope and in week ten to show a draft summary. At the second meeting, the connectors pointed out that the charting form had no field for whether a program offered transport. The team added that field and went back to chart it from the program descriptions.

**Sarah:** The reading also talks about JBI. What is that?

**Kiffer:** JBI is an evidence synthesis organization based at the University of Adelaide. It used to be called the Joanna Briggs Institute. Micah Peters and colleagues wrote its guidance for scoping reviews, and JBI has since added guidance on protocols and on how to extract, analyze and present results.

**Sarah:** What does the JBI method add to the earlier frameworks?

**Kiffer:** It is more specific. The team registers a protocol before searching, and the question is framed with Population, Concept and Context, which students know as PCC from Lesson two. The search runs in three steps: a limited search of at least two databases, an analysis of the words and index terms in the relevant records, and then a full search with all of those terms, followed by checking reference lists.

**Sarah:** And appraisal?

**Kiffer:** JBI does not generally require it, because the purpose is to describe what evidence exists. The Cedar Valley team chose to appraise anyway, because the planning team wanted to know how strong the designs were. The key is to say so in the protocol and to avoid presenting the scoping review as a test of whether programs work.

**Sarah:** Let's talk about charting. Is that just another word for data extraction?

**Kiffer:** Charting is the JBI word for extraction in a scoping review, and it differs in two ways. It is more descriptive, because the questions are about what was done, with whom and how it was measured. And it is more iterative. Teams often discover only while charting which categories they need, so the form is piloted, revised and the changes are reported.

**Sarah:** What does the finished product look like?

**Kiffer:** The completed forms become the charting table, with one row per source and one column per field. One row describes a fictional cluster-randomized trial in twelve seniors' centres in Canada, which tested weekly volunteer-led group walks for sixteen weeks and measured loneliness with the six-item De Jong Gierveld scale. The row also records its other outcomes, its attendance figures, and that it reported gender and living alone but not income.

**Sarah:** And from all those rows, the team writes the results?

**Kiffer:** First it summarizes them. JBI describes the analysis in most scoping reviews as descriptive. A descriptive numerical summary counts the sources by the characteristics charted, and basic qualitative content analysis sorts text into categories, for example the referral pathways in connector programs.

**Sarah:** What did the counts show?

**Kiffer:** There were fourteen group-activity studies, eight of them randomized. There were eight one-to-one befriending or telephone studies, six randomized. There were twelve community connector studies, and none of them was randomized. The rest were intergenerational and technology-based programs. The counts also showed that few studies described participants' ethnicity, language or disability. Ethnicity was reported in eleven of the forty-two, and language in only three.

**Sarah:** How is a scoping review reported?

**Kiffer:** With the PRISMA extension for scoping reviews. PRISMA stands for Preferred Reporting Items for Systematic reviews and Meta-Analyses, and Andrea Tricco and colleagues published the extension in two thousand eighteen. It has twenty essential items and two optional items, and the optional ones are about critical appraisal. It also speaks of sources of evidence, because a scoping review may include documents that are not research studies.

**Sarah:** So to summarize section one, a scoping review maps evidence, it still needs a protocol and a careful search, it charts the data iteratively, and it reports with its own checklist.

**Kiffer:** That is it. And the Cedar Valley team managed all of that in twelve weeks only because it took some shortcuts, which brings us to section two.

**Sarah:** Rapid reviews. Why do they exist at all?

**Kiffer:** Because a full systematic review commonly takes a year or more, and decision-makers often need an answer in weeks. Hamel and colleagues looked at the many published definitions and proposed that a rapid review speeds up a systematic review by streamlining or leaving out some of its methods, so that it uses fewer resources.

**Sarah:** Isn't that just a systematic review done badly?

**Kiffer:** That is the worry, and it is why the definition matters. A rapid review keeps the logic of a systematic review. It has a protocol, an explicit search, stated criteria and a structured synthesis. It departs from the standard methods at specific points, which the team chooses in advance and reports. A review that hides its shortcuts is a problem. A review that states them lets the reader judge.

**Sarah:** Who decides which shortcuts are acceptable?

**Kiffer:** The team decides, ideally with its knowledge users. That is the term for the people who will act on the findings, such as managers, clinicians, policy-makers and community members. For Cedar Valley, they are the planning team and, through the advisory group, the older adults the program will serve.

**Sarah:** Is there guidance on the shortcuts?

**Kiffer:** The main guidance comes from the Cochrane Rapid Reviews Methods Group. It published interim guidance in two thousand twenty, with twenty-six recommendations, and the journal version by Garritty and colleagues appeared in two thousand twenty-one. It covers every stage of a review, from the question to the certainty of evidence.

**Sarah:** Give me a few examples.

**Kiffer:** Knowledge users help refine the question, and the team limits the interventions and outcomes to those that matter most. The search covers a core set of databases, with date and language limits only when justified. Two reviewers screen at least twenty percent of the titles and abstracts, then one screens the rest, and a second screens everything excluded. For extraction and risk of bias, one reviewer does the work and a second verifies it.

**Sarah:** Has the guidance changed since then?

**Kiffer:** The group published an update in two thousand twenty-four with twenty-four recommendations. It keeps the same structure. Among other changes, it allows single screening after a dual-screened sample only when the two reviewers agree closely, and it asks every rapid review to describe its restricted methods and discuss how they might affect the findings.

**Sarah:** The reading sorts shortcuts into two kinds. What are they?

**Kiffer:** Some shortcuts narrow what the review can find. A narrower question, fewer databases, date and language limits and less grey literature all belong here. The risk is that relevant studies never enter the review. The other kind reduces independent checking. Single screening, single extraction and single appraisal belong here. The risk is that the team makes errors in handling the studies it did find.

**Sarah:** Why does the distinction matter?

**Kiffer:** Because the remedies differ. If you narrow the search, you protect yourself with supplementary searching, such as citation chasing. If you reduce checking, you protect yourself with verification by a second person. Naming the kind of shortcut tells you which protection to use.

**Sarah:** Do we know how much these shortcuts actually cost?

**Kiffer:** For some of them. Nussbaumer-Streit and colleagues looked at a sample of Cochrane reviews and asked which of their included studies shortened searches would have found. Most conclusions would have stayed the same, though not all. The same group found that leaving out publications in languages other than English rarely changed conclusions. Screening is riskier. In a randomized trial, Gartlehner and colleagues found that single reviewers missed thirteen percent of relevant studies at the abstract stage, which is why the guidance keeps a second reviewer involved.

**Sarah:** The worked example in this section has some arithmetic. Can you talk us through it?

**Kiffer:** Sure. The Cedar Valley protocol planned the Cochrane screening approach. Two reviewers would screen twenty percent of the one thousand eight hundred and seventy records, which is three hundred and seventy-four records. One reviewer would screen the remaining one thousand four hundred and ninety-six, and a second reviewer would check every exclusion.

**Sarah:** And how much work does that save?

**Kiffer:** Let's count decisions, where one decision is one person reading one record. Screening everything twice takes three thousand seven hundred and forty decisions. In the first three hundred and seventy-four records, which both reviewers screened, about ninety-two percent were excluded at the title and abstract stage. So under the protocol plan, the second reviewer would still read about one thousand three hundred and eighty excluded records. The total comes to about three thousand six hundred and twenty-four decisions.

**Sarah:** That is barely less.

**Kiffer:** It saves about three percent. When nine records in ten are excluded, checking every exclusion costs almost as much as screening everything twice. A bigger saving would need single screening with no check, which is where the thirteen percent figure comes in. So the team amended its protocol, dated the change, and dual screened all the records.

**Sarah:** Is the arithmetic different for full texts?

**Kiffer:** It is. At full text, a larger share of reports is included. Of the one hundred and forty-two full-text reports, forty were included, so checking only the one hundred and two exclusions would save about twenty-eight percent of the second readings. The lesson is to calculate the saving for your own review before adopting a shortcut.

**Sarah:** And how should all of that be reported?

**Kiffer:** The title or abstract says the review is rapid. The methods state each shortcut where it occurs. The flow diagram keeps its standard boxes, as Lesson seven showed. And the limitations explain the likely effect of each shortcut. A sentence like one reviewer screened eighty percent of titles and abstracts after dual screening of a twenty percent sample tells the reader exactly what happened.

**Sarah:** Let me summarize section two. A rapid review keeps the logic of a systematic review, takes shortcuts that either narrow what can be found or reduce checking, should calculate what each shortcut saves, and must report every one.

**Kiffer:** That is a good summary. Now, nine of the Cedar Valley sources describe what people experienced, and section three is about how to synthesize them.

**Sarah:** What is a qualitative evidence synthesis?

**Kiffer:** It brings together the findings of several qualitative studies, which analyze data such as interviews, focus groups and observation, to give an account that no single study could give. Decision-makers need it because trials tell them whether a program can work, and qualitative studies tell them how people experience it, why some take part and others do not, and what makes it feasible to deliver.

**Sarah:** But Cedar Valley is a scoping review. Do scoping reviews do this?

**Kiffer:** Usually they stop at basic content analysis. In week eight, the planning team asked what older adults and the people who run programs say helps or hinders connection. The team amended its protocol, with the date and reason recorded, to add a brief thematic synthesis of nine sources: the six qualitative studies and the qualitative parts of the three mixed-methods studies. It labelled that part as going beyond a standard scoping review.

**Sarah:** The lesson names three methods. Start with thematic synthesis.

**Kiffer:** James Thomas and Angela Harden described thematic synthesis in two thousand eight, from their work at the Evidence for Policy and Practice Information and Co-ordinating Centre in London, usually called the EPPI-Centre. It has three stages. The reviewers code the findings of each study line by line. They group related codes into descriptive themes that stay close to what the studies reported. Then they develop analytical themes that go beyond the studies to answer the review question.

**Sarah:** What did that look like for Cedar Valley?

**Kiffer:** The intern and the evidence officer coded two sources together, agreed on an approach, and then divided the rest. They produced seventy-one codes, such as same volunteer every week, no bus on Sundays and did not want to be called lonely. Those codes became six descriptive themes, for example a consistent and trusted person, practical barriers to getting there, and interests before labels.

**Sarah:** And the analytical themes?

**Kiffer:** There were three. Connection grows through relationships that last. Programs work when they remove the effort of getting there. And people take part as contributors with shared interests. Each points to a design choice, such as keeping the same connector with each person, budgeting for transport and accompaniment, and offering roles in which older adults help others. These are hypotheses for the program to test, because they rest on experiences in other programs.

**Sarah:** What about framework synthesis?

**Kiffer:** Framework synthesis starts from a framework chosen before coding. It comes from framework analysis, a method for applied policy research described by Ritchie and Spencer. Reviewers index each study's findings against the framework, chart them in a matrix and interpret the patterns. Carroll, Booth and Cooper described a version called best fit framework synthesis, in which you start from the published model closest to your question and create new themes for anything that does not fit.

**Sarah:** Did the team try it?

**Kiffer:** The intern coded the same findings against a classification from a meta-analysis by Masi and colleagues, which grouped loneliness interventions by four strategies: improving social skills, enhancing social support, increasing opportunities for social contact, and addressing maladaptive social cognition. Social support and social contact fitted well, but about a third of the codes did not fit. They concerned access and identity, such as transport and not wanting to be called lonely, which a classification of intervention strategies was never designed to capture. That is why the best fit approach insists on analyzing what does not fit.

**Sarah:** And meta-ethnography?

**Kiffer:** George Noblit and Dwight Hare developed meta-ethnography in education research, in a book published in nineteen eighty-eight. It is the most interpretive of the three. Its central task is to translate the concepts of one study into the terms of another. It has seven phases, from getting started and deciding what is relevant, through reading the studies and working out how they relate, to translating them, synthesizing the translations and expressing the synthesis.

**Sarah:** The reading mentions three forms of synthesis.

**Kiffer:** A reciprocal translation applies when studies describe similar things in different words, so a familiar face and someone who remembers my week could both become continuity of a known person. A refutational synthesis applies when studies contradict each other, as when one study found people welcomed a group openly about loneliness and three found reluctance to be labelled lonely. A lines-of-argument synthesis assembles studies of different parts of a phenomenon into a whole.

**Sarah:** So why didn't Cedar Valley use it?

**Kiffer:** Meta-ethnography needs rich studies, an experienced team and time. Most Cedar Valley sources were short evaluations, the team had general qualitative training, and the planning team needed implications within four weeks. Booth and colleagues proposed seven criteria for choosing a method, known by the acronym RETREAT, covering the question, the view of knowledge, time, resources, expertise, audience and type of data. On those criteria, thematic synthesis was the better fit.

**Sarah:** Once you have findings, how do you know how much to trust them?

**Kiffer:** That is the job of an approach called Confidence in the Evidence from Reviews of Qualitative research, known as CERQual. It comes from the working group behind GRADE, the Grading of Recommendations Assessment, Development and Evaluation, and Simon Lewin and colleagues led its development. It looks at four components. Methodological limitations asks about how the contributing studies were done. Coherence asks how well the finding fits the data. Adequacy asks how rich and plentiful the data are. Relevance asks how well the studies match the review's context.

**Sarah:** And then a rating?

**Kiffer:** For each component, the reviewers judge whether there are no or very minor concerns, minor, moderate or serious concerns. Then they make an overall judgement, starting from high confidence and moving down as concerns accumulate. High means it is highly likely that the finding is a reasonable representation of the phenomenon. Moderate means likely, low means possible, and very low means it is not clear.

**Sarah:** What did Cedar Valley get?

**Kiffer:** Four findings. Valuing the same volunteer or connector over time was rated moderate. Practical barriers such as transport and mobility, eased by being accompanied to a first session, was also moderate. Reluctance to be called lonely was low, because the data were thin and one source pointed the other way. And connectors' heavy caseloads in small towns was low, because only two sources supported it.

**Sarah:** How does that fit with the effect estimates from Lesson eight?

**Kiffer:** Lesson eight found, with low certainty, that group programs may reduce loneliness slightly and one-to-one programs may reduce it. The qualitative findings suggest, with moderate confidence, that continuity and practical access shape whether people take part and stay. Together they tell the planning team more than either does alone.

**Sarah:** Summary of section three, then. Thematic synthesis for practical questions on a deadline, framework synthesis when a good model exists, meta-ethnography for building new theory, and CERQual to rate each finding separately.

**Kiffer:** Exactly. And section four pulls the whole review together in one picture.

**Sarah:** What is an evidence and gap map?

**Kiffer:** It is usually a grid with interventions as rows and outcomes as columns, and each cell shows how many studies examined that combination. The International Initiative for Impact Evaluation developed the idea, and the Campbell Collaboration has published guidance for producing maps. A systematic review says what the evidence shows. A map shows where evidence exists and leaves the findings to reviews, so a big bubble means many studies, whatever they found.

**Sarah:** The reading mentions two kinds of gap.

**Kiffer:** An absolute gap is a cell with no studies, or very few. A synthesis gap is a cell with several primary studies but no recent, trustworthy review of them. Many Campbell maps include reviews as well as primary studies, so they can show both. The Cedar Valley map includes primary studies only, so it can show absolute gaps, and the team listed the existing reviews beside it instead.

**Sarah:** How do you build one?

**Kiffer:** You agree the framework with the people who will use the map, then search and select as in any scoping review. You code each study for its row and for every outcome it reports. You appraise if you planned to. Then you count, draw, and check the counts before anyone interprets them.

**Sarah:** Check them how?

**Kiffer:** Every study had to report loneliness or social isolation. The loneliness column holds thirty-nine studies, the thirty-three that measured loneliness plus the six qualitative studies that describe it, and the other three all appear in the social isolation column, so all forty-two are accounted for. The team also expected the counts within a row to exceed the row total, because studies report several outcomes. The connector row has twelve studies and forty-one entries.

**Sarah:** What does the map show?

**Kiffer:** Randomized evidence on loneliness clusters in group activities, with eight trials, and in one-to-one programs, with six. Community connector programs, the model the health authority plans to launch, have twelve studies and no randomized trial. Connector studies measured health service use more often than the others, which reflects primary care's interest in visits. And physical health and health service use are empty for intergenerational and technology-based programs.

**Sarah:** So should the authority conclude that technology programs don't help physical health?

**Kiffer:** No, and this is the most common misreading. An empty cell means no included study measured that outcome, which is a gap in research and no evidence either way about the effect. A big bubble counts studies and says nothing about their quality or results. And the framework decides which gaps can appear, so this map cannot show that few studies reported income, ethnicity, language or disability.

**Sarah:** What did the team tell the planning team, in the end?

**Kiffer:** Launch the connector program with an evaluation built in, which matches the very low certainty rating from Lesson eight. Measure loneliness with a widely used scale, such as the De Jong Gierveld scale, so the results can be compared with earlier studies. And record participants' equity characteristics, uptake, experience and health service use, because those are where the evidence is weakest and what primary care partners will ask about.

**Sarah:** Let's put it all together. What does a small team produce at this stage of a rapid scoping review?

**Kiffer:** Five things. A complete charting table, with not reported wherever information is missing. A descriptive summary, with a cross-tabulation by design and category. And a table of the rapid review shortcuts that compares the protocol with what the team did and states the risk of each one.

**Sarah:** And the qualitative part?

**Kiffer:** If the review has three or more qualitative sources, the team writes findings from a brief thematic synthesis and rates each with the four CERQual components. If it doesn't, it explains why a qualitative synthesis does not apply. Finally, it draws a simple evidence and gap map of its sources, checks the counts against the charting table, and names the gaps, with a sentence on whether each one matters for the decision-maker.

**Sarah:** And next week?

**Kiffer:** Lesson eleven turns to the other half of the Cedar Valley project, the environmental scan. We will look at document reviews, program inventories, surveys and key informant interviews, and at how a scan is combined with a literature review.

**Sarah:** Thanks, Kiffer. That's it for this episode of Office Hours.

**Kiffer:** Thanks, Sarah. See you next week.