# Lesson 2: Review Questions, Eligibility Criteria and Protocols

*Companion-podcast transcript, Sarah and Kiffer*

---

**Sarah:** Welcome back to Office Hours. I'm Sarah.

**Kiffer:** And I'm Kiffer. This week we are in Lesson two of Finding and Synthesizing Health Evidence, which covers review questions, eligibility criteria and protocols.

**Sarah:** Last week ended with the Cedar Valley team choosing what kind of review to do. Remind me where we are with that case.

**Kiffer:** Cedar Valley is our fictional health authority in British Columbia. Its planning team wants to launch a community connector program, a form of social prescribing, for older adults who are lonely or socially isolated. They have asked a small evidence team for a rapid scoping review of what has been evaluated and an environmental scan of programs already running in Canada. The team is an evidence officer, a university librarian and a student intern, and they have twelve weeks.

**Sarah:** And this lesson is everything that happens before they search.

**Kiffer:** Yes. It covers the question, the rules for what counts, the written plan, and the conversations that settle the boundaries. Students often want to skip ahead to searching. In my experience, most of the trouble in a review traces back to decisions left vague at this stage.

**Sarah:** Then let's start with the question. Students have already met question frameworks in the research methods course, HSCI two zero seven. What is different here?

**Kiffer:** In that course, the question was for a study you would design yourself. A review question is asked of studies other people have already done, and each part of it gets used three times. It becomes an eligibility criterion, which is a rule that decides whether a study is in or out. It becomes a block of terms in the search. And it becomes a heading on the form where you record information from each study.

**Sarah:** So if the question is fuzzy, the fuzziness shows up three times.

**Kiffer:** Exactly. Take "older adults" with no age attached. One reviewer includes a study of people aged fifty-five and older and the other excludes it. The librarian does not know whether to search for middle-aged people. The charting form has no rule for mixed-age samples. Each problem appears weeks later, when it is expensive to fix.

**Sarah:** What makes a good review question, then?

**Kiffer:** I look for four things. It can be answered from studies or documents that exist. It is specific enough to set boundaries. It fits the type of review, so a scoping review asks what evidence exists, while a systematic review of effectiveness asks whether something works. And it matches the decision the people who asked for it need to make.

**Sarah:** Let's go through the frameworks. Start with the one everyone knows.

**Kiffer:** That is PICO, which stands for population, intervention, comparison and outcome. It is usually traced to a nineteen ninety-five paper by Richardson and colleagues on the well-built clinical question, and it is the standard frame for systematic reviews of whether an intervention works. Its extensions add letters. PICOS adds the eligible study designs, and PICOTS adds timing and setting. Each extra letter is simply another eligibility rule.

**Sarah:** And PECO?

**Kiffer:** PECO swaps the intervention for an exposure, something people experience without a researcher assigning it. Morgan and colleagues described it in two thousand eighteen. It suits questions about risk factors, such as whether living alone is associated with more emergency department visits among older adults.

**Sarah:** The reading has a box about outcomes that I had to read twice. Can you explain it?

**Kiffer:** The Cochrane Handbook, the main methods guide for reviews of interventions, advises that in most reviews the outcome should not decide whether a study gets in. Studies often measure an outcome and then leave it out of the paper, sometimes because the result was disappointing. If you exclude studies that do not report your outcome, the review leans toward favourable results.

**Sarah:** But Cedar Valley uses the outcome as a criterion.

**Kiffer:** On purpose. Their question is about interventions evaluated for reducing loneliness or social isolation, and an evaluation that measured neither cannot answer it. So the team made the outcome a criterion and recorded the decision and its consequence in the protocol. You can depart from standard advice if you have a reason and you write it down.

**Sarah:** Now the framework designed for scoping reviews.

**Kiffer:** PCC, which stands for population, concept and context. It is recommended by JBI, the evidence synthesis organization formerly called the Joanna Briggs Institute, in guidance by Peters and colleagues. A scoping review maps the extent, range and nature of evidence on a topic. It is rarely asking whether one thing beats another, so a comparison is often unnecessary, and the outcomes are frequently part of what is being mapped.

**Sarah:** Walk me through the three letters with Cedar Valley.

**Kiffer:** The population is adults aged sixty-five and older who live in the community. The concept is community-based interventions evaluated for their effects on loneliness or isolation, with the outcomes those evaluations report. The context is community settings in high-income countries, chosen so the findings transfer reasonably well to British Columbia. JBI also asks you to state the types of evidence source, such as studies of any design and evaluation reports.

**Sarah:** Read us the question.

**Kiffer:** Which community-based interventions have been evaluated for reducing loneliness or social isolation among adults aged sixty-five and older who live in the community in high-income countries, and what outcomes have those evaluations reported?

**Sarah:** That's broad.

**Kiffer:** It is, which is why scoping reviews add sub-questions. Cedar Valley has five. They ask about the types of intervention and how they are delivered, the designs and measures used, the other outcomes reported, how well studies describe equity-relevant characteristics such as income, rurality and language, and what studies say about implementation. Those sub-questions become the headings of the charting form, so they need to be settled before the search.

**Sarah:** The reading also writes a question for the environmental scan.

**Kiffer:** It uses the same framework, but the population is organizations. It asks which organizations currently run community programs for socially isolated older adults, how those programs are designed, funded and evaluated, in British Columbia and elsewhere in Canada. Writing the two questions in parallel means the review's charting form and the scan's data template line up later.

**Sarah:** What about qualitative questions?

**Kiffer:** For experiences or views, there is SPIDER, from Cooke, Smith and Booth in two thousand twelve. The letters stand for sample, phenomenon of interest, design, evaluation and research type. The last two let you say you want qualitative or mixed-methods studies, which PICO cannot express. A Cedar Valley example would ask how older adults experience referral to a community connector, as reported in interview studies.

**Sarah:** Is SPIDER good for searching too?

**Kiffer:** Methley and colleagues tested that in two thousand fourteen. The SPIDER search returned fewer irrelevant records than the PICO search, but it also missed relevant studies. So use SPIDER to frame a qualitative question and then build a broader search. Booth's earlier framework, SPICE, adds the setting and a perspective, which helps with questions about how one group sees a service.

**Sarah:** What mistakes do you see most when students frame review questions?

**Kiffer:** Four, mainly. The first is forcing PICO onto a mapping question, inventing a comparison like "compared with nothing" that then excludes the uncontrolled evaluations the team wanted. The second is the compound question, such as which interventions exist and which is most cost-effective, which need different evidence. The third is an element nobody can judge from a report, like "programs that were well implemented". The fourth is assuming the words in the question are the search terms, when Lesson four will add many synonyms and subject headings.

**Sarah:** Let's move to Section two, eligibility criteria.

**Kiffer:** These are the rules that decide, record by record, whether a study is included. Inclusion criteria state what a study must have. Exclusion criteria remove a study that would otherwise qualify.

**Sarah:** The reading says exclusion criteria should do different work. What does that mean?

**Kiffer:** Many draft tables have exclusions that are the inclusions turned around, like "exclude people under sixty-five". That line adds length and no information. A useful exclusion carves out a case the inclusion criteria would otherwise let in. A study of older adults living in long-term care homes meets the age rule, so Cedar Valley needs a specific exclusion for long-term care.

**Sarah:** There's also a distinction between study characteristics and report characteristics.

**Kiffer:** Study characteristics describe the research: who took part, what was done, where, with what design and outcomes. Report characteristics describe the document: its date, language and type of publication. Limits on report characteristics are usually practical, and each can throw away relevant evidence, so each needs a stated reason and should be treated as a limitation.

**Sarah:** How do you know if criteria are applied consistently?

**Kiffer:** When two trained reviewers, working on their own, usually reach the same decision about the same record. The reading lists six properties that help. A criterion should be specific, observable in what authors report, operationally defined, justified, applied in a fixed order at full text, and tested before screening.

**Sarah:** Say more about operational definitions.

**Kiffer:** An operational definition says how you will recognize a concept in a study report. Words like "older adults" and "community" feel clear until two people disagree about a particular paper. For older adults, Cedar Valley's rule is that at least eighty percent of participants are sixty-five or older, or results are reported separately for that group. If no age distribution is given, a mean or median age of sixty-five or more is accepted. Many programs in British Columbia accept people from fifty-five, so this rule gets used constantly.

**Sarah:** And loneliness versus social isolation?

**Kiffer:** The team used Perlman and Peplau's definition from nineteen eighty-one: loneliness is the unpleasant feeling that arises when your relationships fall short, in number or quality, of what you want. It is subjective and measured with self-report scales. Social isolation is an objective lack of contacts, measured by network size or how often you see people. A connector program might reduce isolation without changing loneliness, so the review reports them separately.

**Sarah:** And community-based?

**Kiffer:** Delivered to people living in their own homes or in independent or assisted living, in places like community centres, libraries, primary care offices or people's homes, or by telephone or online. Hospital wards and long-term care homes fall outside it.

**Sarah:** The table has a column I haven't seen before, decision notes.

**Kiffer:** The table has eight rows: population, intervention, outcome, setting, country, types of evidence source, date and language. Each has an include column, an exclude column and a decision note for borderline cases, most of which came from piloting. One note says that a record mentioning "seniors" without ages goes forward to full text. Another says that a program with a different main aim, like an exercise class, is eligible if it measured loneliness or isolation. There is no comparison row, because the review maps evaluations of any design.

**Sarah:** What about qualitative studies? They don't usually have a loneliness score.

**Kiffer:** The outcome criterion has two versions. Quantitative studies need at least one measure of loneliness or isolation. Qualitative studies qualify if they report participants' or providers' accounts of effects on loneliness or isolation. Studies that report only perceived social support or general well-being are excluded.

**Sarah:** And the date limit? Twenty ten seems arbitrary.

**Kiffer:** Every date limit looks arbitrary until you state the reason. The protocol says the planning team needs evidence on current service models and technologies, and that earlier evaluations are summarized in existing reviews. You might disagree, and because it is written down, you can.

**Sarah:** Then the section turns to screening in two stages.

**Kiffer:** At title and abstract, reviewers exclude only records that clearly fail, and anything uncertain goes forward. At full text, they apply every criterion strictly and record one reason for each exclusion, taken from a fixed list. Cedar Valley's list follows its screening questions, starting with population, then intervention and setting, outcome and country, and it ends with publication type, date and language. If a report fails two criteria, everyone records the first, so the flow diagram counts can be reproduced. Lesson seven teaches screening in depth.

**Sarah:** Let me test that. A German report from two thousand eight about a befriending program in Austria.

**Kiffer:** It fails on date and on language. Date comes first in the order, so the reason recorded is "published before twenty ten".

**Sarah:** Tell me about the pilot.

**Kiffer:** The intern and the evidence officer each screened the same twenty records from the librarian's preliminary search and agreed on fifteen, which is seventy-five percent. The disagreements came from three causes: programs for people aged sixty and older, studies reporting "social connectedness", and a program in assisted living. The team added the age rule, listed which measures count as isolation, and said that assisted living counts as the community.

**Sarah:** And the second round?

**Kiffer:** On a fresh set of twenty, they agreed on nineteen, or ninety-five percent. That shows the criteria are now applied more consistently. It does not prove they capture the right studies. And plain percent agreement ignores agreement by chance, which is why Lesson seven teaches Cohen's kappa.

**Sarah:** Section three is the protocol. Why write one at all for a twelve-week rapid review?

**Kiffer:** Because many decisions in a review could be made after you have seen the studies, such as which outcome to emphasize or how to word a criterion. If you write the plan first, any later change shows up as a difference between plan and review. Shamseer and colleagues add practical benefits: you estimate the work, other teams can see the review is under way, and readers can check the review against its promises. Rapid reviews take shortcuts, and the protocol is where readers see them.

**Sarah:** The checklist is PRISMA-P. What does that stand for?

**Kiffer:** The Preferred Reporting Items for Systematic review and Meta-Analysis Protocols. Moher and colleagues published it in two thousand fifteen, with an explanation paper by Shamseer and colleagues. It has seventeen items, or twenty-six elements once you count the sub-items.

**Sarah:** How are they organized?

**Kiffer:** In three parts. Administrative information covers the title, registration, authors, how amendments will be handled, and funding, including the funder's role. The introduction has the rationale and objectives. Everything else is methods: eligibility, sources, search, how records are managed and selected and data collected, data items, outcomes, risk of bias, synthesis, meta-bias such as publication bias, and confidence in the body of evidence.

**Sarah:** PRISMA-P was built for systematic reviews. Does it fit a scoping review?

**Kiffer:** Mostly, with adaptations you state openly. The checklist for reporting a finished scoping review is PRISMA-ScR, from Tricco and colleagues in two thousand eighteen, and it treats critical appraisal as optional. Cedar Valley registers on the Open Science Framework registry, because PROSPERO, the international register for systematic reviews, does not accept scoping reviews. Its objectives use the population, concept and context question, and its later methods items describe mapping in place of pooling.

**Sarah:** Let's go into the annotated protocol. What stood out to you?

**Kiffer:** The sub-item on the role of the sponsor. The planning team wants to launch a program, so it could prefer favourable findings. The protocol says the planning team agreed the question and scope and will comment on the draft, but has no role in screening, charting or appraisal. That sentence protects the credibility of the review.

**Sarah:** What about the rationale?

**Kiffer:** A rationale says what is known, what is missing and what decision the review informs. Cedar Valley's notes that loneliness and isolation are common in later life and associated with poorer health and higher mortality, citing a meta-analysis by Holt-Lunstad and colleagues and a National Academies report. It names earlier reviews, including Dickens and colleagues, Gardiner and colleagues, and a review of reviews by Fakoya and colleagues, and says what Cedar Valley needs that they were not designed to provide. It avoids claiming that no relevant review exists, because a preliminary search rarely proves that.

**Sarah:** The search strategy item asks for a draft. Isn't that premature?

**Kiffer:** PRISMA-P asks for a draft for at least one database so readers can judge its scope. Cedar Valley's combines three concepts: older adults, loneliness or isolation, and interventions or evaluation, each with a few subject headings and text words. Lesson four builds and peer reviews the full version. The protocol also says an artificial intelligence research assistant may suggest extra studies, and that every suggested citation will be checked against its source, which anticipates Lesson six.

**Sarah:** Item eleven is the selection process. That's where the rapid shortcuts are.

**Kiffer:** Two reviewers independently screen the first twenty percent of titles and abstracts. Then one screens the rest, and a second checks every exclusion. Full texts are screened by two reviewers, with the librarian as tiebreaker. The Cochrane Rapid Reviews Methods Group discusses streamlined approaches like this in guidance by Garritty and colleagues.

**Sarah:** What's the cost of that shortcut?

**Kiffer:** Some risk of missing eligible studies early on, in return for time, and checking every exclusion reduces that risk. The team also decided not to contact study authors for missing information, which saves time and must be reported as a limitation. Lesson ten looks at these trade-offs.

**Sarah:** What goes on the charting form?

**Kiffer:** Citation, country, setting, design, sample size, participant characteristics, the intervention, outcomes and measures, findings as the authors report them, implementation information and funding. Two published tools give it structure. PROGRESS-Plus, from O'Neill and colleagues, lists equity-relevant characteristics such as place of residence, ethnicity, gender and socioeconomic status. TIDieR, the Template for Intervention Description and Replication from Hoffmann and colleagues, structures the description of each intervention. Every data item should answer one of the sub-questions.

**Sarah:** And the last items, appraisal and synthesis?

**Kiffer:** Appraisal is optional in a scoping review, but the team chose to do it, because the planning team needs to see how strong the designs are behind each type of intervention. Appraisal will describe the evidence and will not exclude studies. There is no meta-analysis. The plan is counts, tables, an evidence map of intervention types against outcomes, and a narrative summary giving the direction of effect as authors reported it, without counting studies by statistical significance.

**Sarah:** And items sixteen and seventeen?

**Kiffer:** Meta-bias and confidence in the evidence. Publication bias will not be formally assessed, because nothing is pooled, and certainty will not be rated with GRADE, which stands for Grading of Recommendations Assessment, Development and Evaluation. What students should notice is that each item is still answered, with a reason. If you write "not applicable", a reader cannot tell a considered decision from an oversight.

**Sarah:** The section ends with amendments.

**Kiffer:** Protocols change. What matters is that a reader can see each change, when it happened, why, and whether it came before or after the team saw the studies. The reading gives two hypothetical log entries. Switching screening platforms before screening carries little risk. Relaxing the age rule partway through full-text screening, after reading many studies that report mean ages in the early sixties, is a different matter.

**Sarah:** Because it changes which studies get in.

**Kiffer:** Yes, and the studies themselves prompted it. So the change has to be applied to every record already screened, explained, reported as a deviation, and the studies it admits flagged. The final review then describes it, as the PRISMA two thousand twenty reporting guideline asks.

**Sarah:** Section four is about people. Let's start with the librarian.

**Kiffer:** A health sciences librarian, sometimes called an information specialist, knows how databases and their controlled vocabularies are organized. The Cochrane Handbook recommends working with one, and the evidence supports that. Rethlefsen and colleagues found that systematic reviews in general internal medicine with a librarian co-author reported higher-quality search strategies.

**Sarah:** What does a librarian actually do for a review like Cedar Valley?

**Kiffer:** They test whether each part of the question can be searched. The Cedar Valley librarian pointed out that community connector is a local term, and that authors also write link worker, navigator and social prescribing. They check for existing reviews and registered protocols, run the preliminary search, build and translate the full search, arrange peer review using PRESS, which stands for Peer Review of Electronic Search Strategies, document the search to PRISMA-S, the PRISMA extension for reporting literature searches, and often remove duplicate records before screening.

**Sarah:** Is the librarian an author?

**Kiffer:** Often, and you should raise it at the first meeting. The International Committee of Medical Journal Editors sets out the criteria: a substantial contribution, drafting or critically revising the work, approving the final version and being accountable for it. A librarian who designs and documents the search usually meets them. A librarian who gives brief advice is acknowledged.

**Sarah:** What should students bring to a consultation?

**Kiffer:** Drafts the librarian can test: your question and sub-questions, draft criteria, the deadline and your available time, and a few studies you already know are relevant. Those are called benchmark studies, and if the search misses them, it needs work. Choose them before the search is built, because picking them afterwards weakens the test.

**Sarah:** And what did the preliminary search show in Cedar Valley?

**Kiffer:** It found existing reviews, which the team read to see how others defined interventions and outcomes, and no registered protocol matching the question. It suggested the full search would return a few thousand records, which, with three part-time people and twelve weeks, supported the streamlined screening plan. And it supplied the forty records used in the two pilot rounds.

**Sarah:** Now interest holders. Who are they in Cedar Valley?

**Kiffer:** People and organizations with an interest in the review or the decisions it informs. For Cedar Valley, that means the planning team, primary care clinicians who would refer patients, community organizations and connectors who would run the program, older adults who might take part, and the health authority's Indigenous health team.

**Sarah:** Is involving them standard in scoping reviews?

**Kiffer:** Arksey and O'Malley, in their two thousand five framework, included consultation as an optional sixth stage, and Levac, Colquhoun and O'Brien argued in two thousand ten that it should be required. The International Association for Public Participation describes a spectrum running from informing through consulting, involving and collaborating to empowering. In Cedar Valley, clinicians are informed, community organizations are consulted, the older adults' advisory group and the Indigenous health team are involved, and the planning team collaborates on the question and scope.

**Sarah:** What can interest holders change, and what can't they?

**Kiffer:** They can say which decisions the review must inform, name outcomes that matter to people, supply local vocabulary and sources, and comment on how findings are presented. What stays with the review team is whether individual studies meet the criteria, how they are appraised and what the evidence shows. That separation lets the findings be trusted, especially when the people commissioning the review hope for a particular answer.

**Sarah:** Tell me about the scoping meeting.

**Kiffer:** In week two, the evidence team met for ninety minutes with members of the planning team, two community connectors, two members of the older adults' advisory group and a member of the Indigenous health team. The intern had circulated the draft question and criteria a week before, and during the meeting kept a decision log recording each request, who raised it, the decision, the reason and where it appears in the protocol.

**Sarah:** Give me a request that was declined.

**Kiffer:** The planning team asked for Canadian studies only. The team declined for the review, because they expected very few Canadian evaluations, and explained that the environmental scan would cover Canadian programs. A physician asked for randomized trials only, which was declined because a scoping review maps all designs. The planned appraisal answers the underlying concern.

**Sarah:** And accepted ones?

**Kiffer:** The older adults' advisory group asked that the review capture outcomes such as having someone to call and feeling useful, which was accepted, with a charting category for outcomes participants describe in qualitative studies. The Indigenous health team asked that the scan identify Indigenous-led programs and follow those organizations' protocols when contacting them, and the scan's outreach will be planned with them. The connectors pointed out that many seniors' programs start at fifty-five, and the eighty percent rule means those programs still qualify when most participants are older.

**Sarah:** And the cost-effectiveness question?

**Kiffer:** That went on the parking-lot list. It is a good question that needs a different review and economic evidence. The list keeps good questions from being lost and from quietly expanding the review.

**Sarah:** Which brings us to scope creep.

**Kiffer:** Scope creep is the gradual growth of a review through requests that each seem small, and with twelve weeks Cedar Valley had little room for it. After registration, any scope change goes through the amendment log. The planning team sees progress at agreed points: after screening, after charting and at the draft evidence brief. Anything that comes up in between waits on the parking-lot list.

**Sarah:** So for someone planning a rapid scoping review and scan of their own, what does this stage produce?

**Kiffer:** A question, eligibility criteria and a short protocol for the rapid scoping review and environmental scan. Write the review question with the framework that fits, which for a mapping question will usually be population, concept and context, and add three to five sub-questions. Write the parallel scan question too. Then draft a criteria table with include, exclude and decision-note columns, run a short preliminary search, choose three benchmark studies, and pilot the criteria on ten records with a second reviewer.

**Sarah:** And the protocol and the people?

**Kiffer:** Outline a short protocol that answers each PRISMA-P item, saying which items you adapt for a scoping review and why. Consult a health sciences librarian early, and identify one interest holder whose view could change your scope. Keep a short decision log of anything that changes.

**Sarah:** Any last advice?

**Kiffer:** Take the time now. Every hour spent making your question and criteria precise saves several hours of disagreement during screening, and it makes the final report much easier to write.

**Sarah:** Next week, Lesson three looks at how databases and search engines find information, which sets up the search building in Lesson four. Thanks for listening to Office Hours.

**Kiffer:** Thanks, everyone. See you next week.