Introduction

By the end of Lesson 7, a review team has a list of included studies and a PRISMA flow diagram showing how it got there. The next task is to take the information the review needs out of each study and record it in a form that can be compared, checked and synthesized. This task is called data extraction: the systematic recording of the same set of details from every included study onto a standard form. Much of the work is clerical, but the form decides what the review can say later. A characteristic that nobody extracted cannot be used to group studies, explain differences between them or describe whom the evidence applies to.

The lesson follows the order of the work. Section 1 designs and pilots the extraction form. Section 2 introduces the tools that assess the risk of bias in each study, matched to its design. Section 3 shows how two reviewers apply a tool consistently. Section 4 rates the certainty of the whole body of evidence for an outcome with GRADE. These tools rest on a few concepts of bias, chiefly selection bias, information bias and confounding. Section 2 summarizes them, and HSCI 230 Lessons 7 to 11 teach them in depth for students who take that course.

In a scoping review, the JBI method calls this step data charting (Peters et al., 2020). Charting tends to be more descriptive and more iterative than extraction for a systematic review of effects, and Lesson 10 returns to it. The design principles in this section apply to both. The Cedar Valley team uses one form for its 42 studies, because the planning team asked about outcomes as well as about the range of programs studied, and a shorter descriptive form for the 26 grey-literature documents, most of which are short program evaluation reports whose methods are too briefly described to classify by design, and which also feed the environmental scan in Lesson 11.

1.1 What an Extraction Form Is For

An extraction form does four jobs. It describes the included studies, which become the characteristics table that readers use to judge whom the evidence covers (Lesson 12). It collects the information the synthesis needs, such as intervention categories and outcome results (Lesson 9). It records the details on which risk-of-bias assessment depends, such as how participants were allocated and how many were lost to follow-up (Sections 2 and 3). And it leaves an audit trail, so that a reader or a later update can see where every number came from.

The unit of extraction is the study. A single study often produces several reports, such as a registry entry, a protocol, a main results paper and a later follow-up paper. These are called companion reports, and the Cochrane Handbook advises collating all reports of a study so that each study is counted once and its information is complete (Li, Higgins and Deeks, 2019). The reverse problem also occurs, when two papers that look like separate studies report overlapping samples. The Cedar Valley form therefore begins with a study identifier and a field that lists every report linked to it.

1.2 What to Collect: The Domains of a Form

Most extraction forms are organized into domains that follow the elements of the review question. The table shows the domains of the Cedar Valley form.

The intervention fields need the most planning, because community programs for loneliness vary in ways that matter for synthesis. Hoffmann and colleagues (2014) published the Template for Intervention Description and Replication, known as TIDieR, a 12-item checklist for reporting interventions. Reviewers can use its items to structure extraction, which also shows when a study left out something important. The accordion shows how the Cedar Valley team turned TIDieR items into fields.

1.3 Writing Fields That Two People Fill In the Same Way

A field that two people interpret differently produces data that cannot be trusted. The main safeguard is a data dictionary, sometimes called a codebook, which defines every field: its name, a definition, the allowed values, instructions for unusual cases and an example. Several principles make fields reliable. Each field should hold one fact, so "age and gender of participants" becomes two fields. Closed fields with allowed values suit anything that will be used to group or count studies, and free text suits descriptions that readers will need in the authors' own terms. Every field should allow "not reported" and "not applicable" as answers. Extractors should record the page or table where each number was found, and results should be copied exactly as reported, with the type of statistic, the time point and the group sizes; any conversion is done later and documented.

1.4 Piloting the Form

Piloting means that two or more people use the draft form on a small, varied set of included studies, compare what they recorded, and revise the form and the data dictionary where they differed. The figure shows where piloting fits in the extraction workflow.

The Cedar Valley team piloted its form with the student intern and the evidence officer. Each extracted the same five studies independently: an individually randomized trial, a cluster-randomized trial, a non-randomized controlled study, a qualitative study and a mixed-methods study. The draft form had 24 fields that could be compared directly, giving 5 × 24 = 120 paired entries, and the extractors disagreed on 17 of them.

The pattern of disagreement mattered more than the total. Six of the 17 disagreements concerned setting, because two studies recruited people living at home and in assisted living without separating their results. Five concerned the time point, because studies reported several follow-ups and the extractors chose different ones. Four concerned the intervention category, because one program combined group activities with one-to-one support, and two were copying errors. The team added the "mixed" setting value, the time-point rule and the secondary category field shown in the data dictionary, then piloted the revised form on three new studies (3 × 24 = 72 paired entries), with 5 disagreements. There is no universal threshold for acceptable agreement, so a team should decide in advance what it will accept and should look at which fields disagree, since a high overall percentage can hide one field that is often wrong.

1.5 Managing Extraction: Who, How and With What

Errors in extraction are common. Buscemi and colleagues (2006) found that single extraction produced more errors than double extraction, although it took less time. Cochrane guidance therefore recommends that two people extract data independently, particularly outcome data, and resolve differences by discussion. A common compromise is for one person to extract and a second to check every entry against the source, and the Cochrane Rapid Reviews Methods Group recommends this approach for rapid reviews (Garritty et al., 2021). Lesson 10 discusses the trade-offs that rapid-review shortcuts make.

The Cedar Valley protocol, as amended in week 7 (Lesson 10), uses both approaches. For the 24 comparative studies (the 14 randomized trials and the 10 non-randomized controlled studies), two people extract the results fields independently, because errors in effect estimates would change what the brief says about effectiveness. One person extracts the descriptive fields for those studies, and all fields for the other 18 studies, and the other person checks them in full. A discrepancy log records each disagreement, its cause and its resolution, and the team emails authors when key information is missing, recording the request and any reply on the form.

The software matters less than the form. Spreadsheets work for small reviews if the columns are restricted to the allowed values. Screening platforms such as Covidence (Lesson 7) include extraction modules that support dual extraction, and other tools available in 2026 include SRDR+ (the Systematic Review Data Repository Plus, hosted by the United States Agency for Healthcare Research and Quality) and REDCap, a data capture platform that many universities provide. The cards describe errors that the Cedar Valley team watched for.

What Carries Forward

The form now holds, for each of the 42 studies, its design, its methods and the details that bear on bias, such as allocation, blinding, attrition and the outcome measure. The design field decides which appraisal tool applies to each study, and Section 2 introduces those tools.

Introduction

A study can report its methods clearly and enrol thousands of people and still give a misleading answer, because its design or conduct pushed the result away from the truth in a consistent direction. That systematic error is bias, and the risk of bias of a result is the likelihood that features of the study's design or conduct have made it systematically too large or too small. HSCI 230 Lessons 7 to 11 teach the sources of bias: measurement (Lesson 7), selection (Lesson 8), information bias (Lesson 9), design-specific and temporal biases (Lesson 10) and confounding (Lesson 11). The tools in this section turn those concepts into structured questions that a reviewer answers for each included study.

Risk of bias should be kept separate from three other properties of a study. Reporting quality describes how completely a paper reports its methods, and a poorly reported trial may have been conducted well. Imprecision is random error, shown by a wide confidence interval, and a small unbiased trial is imprecise without being biased. Applicability asks whether the study's population, intervention and setting match the review question. Risk-of-bias tools address bias alone, and GRADE in Section 4 considers imprecision and applicability (which it calls indirectness) across the body of evidence.

Earlier instruments often added points into a quality score. Jüni and colleagues (1999) applied 25 quality scales to the same set of trials and found that conclusions about whether high-quality trials showed different effects depended on which scale was used. Current tools are domain-based: the reviewer judges each type of bias separately, explains each judgement and combines them with explicit rules. The broader term critical appraisal, used particularly by JBI, covers assessments of conduct and relevance as well as bias.

2.1 Choosing a Tool by Design

The ways in which a randomized trial can go wrong differ from the ways a cohort study or a qualitative study can go wrong, so each design has its own tools. The figure maps common designs to the tools in this section.

Authors' design labels are sometimes wrong, for example when a "pilot trial" allocated participants by alternation, so the extraction form records the design as the reviewers judge it.

2.2 RoB 2 for Randomized Trials

The revised Cochrane risk-of-bias tool for randomized trials, RoB 2 (Sterne et al., 2019), replaced the original Cochrane tool (Higgins et al., 2011), which rated items such as sequence generation and blinding as low, high or unclear risk. RoB 2 assesses a specific result, such as loneliness at twelve weeks, so one trial can receive different judgements for different outcomes. The reviewer states the effect of interest first: the effect of assignment to the intervention (the intention-to-treat effect, which most reviews of public health programs estimate) or the per-protocol effect of adhering to it. Judgements are reached through signalling questions, factual questions answered "yes", "probably yes", "probably no", "no" or "no information". An algorithm maps the answers in each domain to a proposed judgement of low risk of bias, some concerns or high risk of bias, which the reviewer can override with a written reason. The tabs describe the five domains.

The overall judgement follows explicit rules. A result is at low risk of bias overall when all five domains are at low risk. It has some concerns when at least one domain raises some concerns and none is at high risk. It is at high risk when at least one domain is at high risk, or when several domains raise some concerns in a way that substantially lowers confidence in the result. A variant for cluster-randomized trials adds a domain on bias arising from the identification or recruitment of participants into clusters, and a variant for crossover trials addresses carry-over and period effects. The Cedar Valley team used the cluster variant for its 4 cluster-randomized trials. Because the developers revise these tools from time to time, a protocol should name the version used.

2.3 ROBINS-I for Non-randomized Studies of Interventions

The Risk Of Bias In Non-randomised Studies of Interventions tool, ROBINS-I (Sterne et al., 2016), assesses studies that compare people who received a program with people who did not, using as a reference a hypothetical randomized trial that would answer the same question without bias. This target trial is a reference point, and it may be hypothetical even when such a trial would be impractical or unethical (Hernán and Robins, 2016). Describing it makes the reviewer specify the population, intervention, comparator and outcome that the study is trying to estimate. The seven domains are arranged by the stage at which bias arises.

Each domain and the overall result are judged at low risk (comparable to a well-performed randomized trial), moderate risk (sound for a non-randomized study but not comparable to a well-performed randomized trial), serious risk (some important problems) or critical risk (too problematic to provide useful evidence on the effect), with a further option of no information. The overall judgement is generally the most severe domain judgement. Because confounding can rarely be ruled out without randomization, moderate risk is a good result for a non-randomized study.

2.4 ROBINS-E for Studies of Exposures

Some questions concern exposures that nobody assigns, such as air pollution, shift work or living alone. The Risk Of Bias In Non-randomized Studies of Exposures tool, ROBINS-E (Higgins et al., 2024), was developed for follow-up (cohort) studies of such exposures. It adapts the ROBINS-I domains to bias due to confounding, bias arising from measurement of the exposure, bias in selection of participants into the study or analysis, bias due to post-exposure interventions, bias due to missing data, bias arising from measurement of the outcome, and bias in selection of the reported result. Judgements range from low risk through some concerns and high risk to very high risk, with an additional category of low risk except for concerns about uncontrolled confounding, which recognizes that residual confounding can almost never be excluded in exposure studies. The Cedar Valley question concerns interventions, so the team does not use ROBINS-E.

2.5 The Newcastle-Ottawa Scale

The Newcastle-Ottawa Scale (NOS), developed by Wells and colleagues at the Universities of Newcastle (Australia) and Ottawa, appraises cohort and case-control studies and appears in many published reviews. The cohort version has three categories: selection, with four items (representativeness of the exposed cohort, selection of the non-exposed cohort, ascertainment of exposure, and absence of the outcome at the start); comparability, with one item worth up to two stars for control of the most important factor and any additional factor; and outcome, with three items (assessment of the outcome, adequate length of follow-up and adequacy of follow-up). The case-control version replaces outcome with an exposure category. The maximum is nine stars.

Stang (2010) criticized the scale for unclear items and cut-offs with no stated basis, and Hartling and colleagues (2013) found low agreement between individual reviewers using it. Many reviews convert stars into "good", "fair" or "poor" ratings with thresholds that vary between reviews, which reproduces the problem with summary scores that Jüni and colleagues (1999) described. A review that uses the NOS should report the stars awarded on each item and explain what earns each star for its question. For non-randomized studies of interventions, ROBINS-I is the more structured choice.

2.6 The JBI Checklists

JBI (formerly the Joanna Briggs Institute, at the University of Adelaide) publishes a family of critical appraisal checklists for many designs, including randomized trials, quasi-experimental studies, cohort, case-control and analytical cross-sectional studies, prevalence studies, case series, case reports, qualitative research, economic evaluations, and text and opinion papers. Questions are answered "yes", "no", "unclear" or "not applicable". The analytical cross-sectional checklist has eight questions, including questions on valid measurement and on identifying and dealing with confounders. The qualitative checklist has ten, most of which ask about congruity between the study's philosophical perspective, methodology, methods, analysis and interpretation, along with the researcher's influence and the representation of participants' voices. JBI has revised its checklists for randomized and quasi-experimental studies so that their questions are grouped by the type of bias they address (Barker et al., 2023).

The family covers designs, such as prevalence studies and case series, that the Cochrane tools do not. JBI guidance asks reviewers to decide in advance how appraisal results will be used, for example which answers would lead to exclusion or to reporting with caution (Aromataris and Munn, 2020), and reporting the answer to each question is more informative than a count of "yes" answers. The Cedar Valley team uses the JBI quasi-experimental checklist for its 9 uncontrolled before-and-after studies.

The CASP checklists, from the Critical Appraisal Skills Programme, are a widely used alternative family, with checklists for randomized trials, cohort, case-control and qualitative studies, among others; HSCI 230 and HSCI 841 Lesson 2 use them. Whichever family a review uses, it names the checklist it applies to each design in its protocol.

2.7 The Mixed Methods Appraisal Tool

A review that includes qualitative, quantitative and mixed-methods studies may prefer one tool for all of them. The Mixed Methods Appraisal Tool (MMAT), first developed by Pluye and colleagues (2009) and revised as version 2018 by Hong and colleagues (2018), begins with two screening questions: whether the study has clear research questions, and whether the collected data allow those questions to be addressed. It then offers five design categories, each with five criteria answered "yes", "no" or "can't tell": qualitative studies, quantitative randomized controlled trials, quantitative non-randomized studies, quantitative descriptive studies and mixed-methods studies. A mixed-methods study is appraised against the mixed-methods criteria, which ask about the rationale for combining methods and the integration of components, and against the criteria for its qualitative and quantitative components. The developers discourage an overall score and recommend reporting the rating for each criterion. The Cedar Valley team uses the MMAT for its 6 qualitative and 3 mixed-methods studies. The table summarizes the allocation of all 42 studies.

What Carries Forward

Each of these tools asks for judgement, and two careful reviewers can answer the same question differently. Section 3 describes the procedures that make appraisal consistent.

Introduction

The tools in Section 2 structure judgement without removing it. A signalling question such as "Is it likely that assessment of the outcome was influenced by knowledge of intervention received?" asks a reviewer to weigh the outcome, the comparator, the setting and what participants were told. Studies of the tools show that reviewers often disagree. Hartling and colleagues (2013) reported low agreement between individual reviewers using the Newcastle-Ottawa Scale, and Minozzi and colleagues (2020) reported low agreement between reviewers applying RoB 2 and described the difficulties they met in applying it. Much of this disagreement comes from reviewers reading different parts of a report or making different assumptions that can be written down and agreed. The procedures in this section are designed to remove those avoidable sources of disagreement and to make the remaining judgements transparent.

The standard approach is for two reviewers to assess each study independently, compare their answers and reach consensus, with a third person available to arbitrate. The Cedar Valley protocol uses this approach for the 24 comparative studies, which are the studies whose risk of bias most affects what the evidence brief will say about effectiveness, and uses one assessor with full checking by a second for the other 18 studies (Section 2).

3.1 Preparing to Appraise

Preparation prevents most disagreements before they happen. Both reviewers should read the full guidance document for each tool, since the short signalling questions depend on definitions given in the guidance. For RoB 2, the team agrees on the effect of interest before starting, and the Cedar Valley team chose the effect of assignment to the intervention because the planning team wants to know what happens when a program is offered to older adults, including those who attend rarely. For ROBINS-I, the team lists the important confounding domains and co-interventions in the protocol, as Section 2 described. For every study, the reviewers gather all available sources: the main paper, its supplementary files, the trial registry entry, any published protocol or statistical analysis plan, and companion reports identified during extraction. A judgement about selective reporting made without the registry entry is likely to differ from one made with it.

The team then runs a calibration exercise, in which both reviewers independently appraise the same small set of studies, usually two or three per tool, and discuss every difference before appraisal begins. Calibration serves the same purpose as piloting the extraction form in Section 1, and it usually produces the first decision rules. The figure shows the full process.

3.2 Writing Decision Rules

A decision rule is a written agreement about how the team will answer a signalling question in a situation that recurs across studies. Decision rules leave the tool itself unchanged and record how the team interprets the tool's guidance for its own topic, so that the same situation receives the same answer in every study and so that readers can see the reasoning. The rules belong in an appendix to the review or in a protocol amendment. The table shows the rules the Cedar Valley team wrote during calibration and the first round of comparison.

The first rule came from the most common disagreement in calibration. In two trials whose comparison groups attended a different social activity of similar length, the student intern had answered 4.4 "probably no", reasoning that both groups expected some benefit, and so had judged domain 4 at low risk. The evidence officer had answered "probably yes", because a participant who knows that she is in the "new program" group may still report her loneliness differently. The team agreed that the active comparator is better considered at question 4.5, where it reduces the likelihood of influence without removing the possibility, and wrote the rule accordingly.

3.3 Recording the Support for Each Judgement

Every answer should be supported by a record that another reader could check: a short quotation or summary from the source, its location, and the reviewer's reason when the answer involves judgement. RoB 2 and ROBINS-I provide a free-text box for this purpose beside each signalling question. The support record turns an opinion into an argument, and it makes consensus meetings faster because reviewers can see where their sources differed. The example shows the support record for domain 3 of one Cedar Valley trial, labelled Trial G.

3.4 Comparing Judgements and Reaching Consensus

After independent assessment, the two reviewers compare their answers question by question and their judgements domain by domain. It helps to classify each disagreement. A disagreement of fact occurs when one reviewer found information that the other missed, such as a registry entry or a table in a supplementary file, and it is resolved by looking at the source together. A disagreement of interpretation occurs when both reviewers saw the same information and answered differently, and it is resolved by discussion with reference to the guidance. If the same interpretive disagreement recurs, the team writes a decision rule and rechecks the studies already assessed. If the reviewers cannot agree, a third person with methodological experience arbitrates, and the record notes that arbitration was used. When the report lacks information that would change a judgement, the team can write to the authors, using the same log as in Section 1.

Many reviews report agreement between the reviewers' independent judgements, before consensus, as an indicator of how difficult the appraisal was. Lesson 7 introduced percent agreement and Cohen's kappa for screening decisions, and the same statistics apply here. In the Cedar Valley review, the two reviewers made 5 domain judgements for each of 14 trials, giving 70 judgements. They disagreed on 13, so they agreed on 57 of 70 (81.4 percent), and 7 of the 13 disagreements were in domain 4. Their independent overall judgements for the 14 trials are cross-tabulated below.

A kappa of 0.33 is lower than the 81.4 percent agreement on domain judgements might suggest, for two reasons. The overall judgement takes the most severe domain judgement into account, so a single disagreement in any of five domains can change it. And because most trials fell into one category, chance agreement is high, which lowers kappa for a given level of observed agreement. After consensus, the 14 trials were judged at some concerns (9 trials) or high risk of bias (5 trials) for loneliness, and none reached low risk overall. This pattern is common in reviews of psychosocial programs, because participants report their own loneliness and know which program they received, so domain 4 rarely reaches low risk.

3.5 Common Errors in Applying the Tools

Several errors recur in published appraisals, and the Cedar Valley team reviewed them during calibration.

3.6 Presenting Risk of Bias and Using It in Synthesis

Risk-of-bias results are usually presented in two figures. A traffic-light plot shows each study as a row and each domain as a column, with a coloured symbol for each judgement. A summary plot shows, for each domain, the proportion of studies at each level of risk. Reviewers can draw these figures by hand, in a spreadsheet, or with robvis, a free tool released as an R package and a web application (McGuinness and Higgins, 2021). The plot below shows the consensus judgements for the 8 Cedar Valley trials of group-based programs, labelled Trial A to Trial H.

The review then puts the assessment to use. It should present each study's judgement beside its results, so that readers can see whether the studies at higher risk show different effects. It can stratify or order studies by risk of bias in tables and figures (Lesson 9 shows how), and a review with a meta-analysis can run a sensitivity analysis restricted to studies at lower risk (HSCI 230 Lesson 2). Excluding studies because of their risk of bias is defensible only when the protocol specified the rule in advance, since an exclusion decided after seeing the results can be used to steer the conclusions. Finally, the judgements feed the risk-of-bias domain of GRADE, which Section 4 describes.

What Carries Forward

The team now has a consensus judgement, with its support, for every comparative study. Those judgements describe individual studies. Section 4 asks a different question: taking all the studies of one intervention and one outcome together, how confident can the team be in what they show? GRADE answers that question for each outcome.

Introduction

Risk-of-bias tools describe single studies. A decision-maker asks a different question: taking all the relevant studies together, how confident can we be about the effect of this intervention on this outcome? The Grading of Recommendations Assessment, Development and Evaluation approach, known as GRADE, answers that question with a rating of the certainty of evidence for each outcome. GRADE was developed by an international working group (Guyatt et al., 2008) and is used by Cochrane, the World Health Organization and many guideline developers. Students who take HSCI 230 Lesson 2, before or after this course, meet GRADE there in a short call-out; this section shows how a review team applies it.

Three features of GRADE shape its use. Certainty is rated for a body of evidence, which means all the studies that address one comparison and one outcome, so the same review can have high certainty for one outcome and very low certainty for another. Certainty is rated for outcomes that matter to the people who will use the review, which for the Cedar Valley planning team means loneliness first. And GRADE separates the certainty of evidence from the strength of any recommendation that follows from it. Guideline panels and decision-makers make recommendations after weighing certainty together with benefits, harms, costs, values and feasibility, so a recommendation can be strong even when certainty is low.

4.1 Four Levels of Certainty

GRADE uses four levels. The descriptions below paraphrase those given by Balshem and colleagues (2011), and the symbols are those used in GRADE tables.

GRADE originally called these levels the quality of evidence. The current term, certainty, makes clear that the rating describes confidence in an estimate of effect, which can be low even when every study was carefully conducted.

4.2 Where the Rating Starts

The rating begins from the design of the studies. A body of evidence from randomized trials starts at high certainty, because randomization protects against confounding. A body of evidence from non-randomized studies, including cohort studies, controlled before-and-after studies and other observational designs, starts at low certainty, because confounding and selection can rarely be excluded. The team then considers reasons to rate the evidence down and, less often, reasons to rate it up, as the figure shows.

A second starting point exists for teams that use ROBINS-I. Because ROBINS-I compares each study with a target trial, the GRADE Working Group has described an approach in which non-randomized evidence assessed with ROBINS-I starts at high certainty and is then rated down for risk of bias by as many levels as the ROBINS-I judgements warrant (Schünemann et al., 2019). Studies at serious risk of bias usually bring the evidence to low certainty or below under this approach, so the two starting points tend to reach similar results. The Cedar Valley protocol states that it uses the conventional low starting point, which is simpler to explain to the planning team.

Two kinds of evidence in the Cedar Valley review sit outside this profile. The 9 uncontrolled before-and-after studies have no comparison group, so they cannot separate the program's effect from change over time, and the team reports them as supporting information. The 6 qualitative studies and the qualitative parts of the 3 mixed-methods studies describe participants' experiences and are assessed with GRADE-CERQual, the companion approach for qualitative evidence that Lesson 10 introduces.

4.3 Five Reasons to Rate Down

For each domain, the team judges whether there is no serious concern, a serious concern (rate down one level) or a very serious concern (rate down two levels). Certainty cannot fall below very low, and the same problem should be counted in only one domain. The accordion describes each domain.

4.4 Three Reasons to Rate Up

GRADE allows the certainty of non-randomized evidence to be rated up in three circumstances (Guyatt et al., 2011). Rating up is usually considered only when no domain has been rated down, since a large effect in studies at serious risk of bias may itself be a product of bias.

4.5 Rating Certainty Without a Pooled Estimate

GRADE is often described in terms of a meta-analysis, with a pooled estimate and its confidence interval. Many reviews, including the Cedar Valley review, synthesize results without pooling them, using the methods and the SWiM reporting guideline that Lesson 9 teaches. Murad and colleagues (2017) described how to rate certainty in this situation. The same five domains apply, but the team judges them from the pattern of results across studies: the spread of effects and their direction for inconsistency, the precision of individual studies and the total number of participants for imprecision, and so on. The team explains each judgement in a footnote, since a reader cannot check it against a single number. Rating certainty in a scoping review is unusual, because scoping reviews are designed to map evidence and seldom estimate effects. The Cedar Valley protocol planned no certainty ratings, and a dated amendment in week 8 added them, limited to loneliness for three intervention categories, because the planning team asked directly how confident it could be; the brief labels the ratings as provisional.

4.6 Worked Example: The Cedar Valley Evidence Profile

An evidence profile sets out the judgement for each domain, the resulting certainty and a footnote explaining each decision.

^a All 8 trials have some concerns (6) or high risk of bias (2) for loneliness, mainly because participants who know their group report their own loneliness (Section 3). ^b Five trials reported small reductions in loneliness, two reported little or no difference, and one reported a larger reduction; program length and setting did not explain the variation. ^c Fewer than ten studies per comparison, so funnel plots were not used; registry searches found no completed but unreported trials. ^d Three of the 6 trials are at high risk of bias and three have some concerns; their results were similar, so the team rated down one level. ^e The total sample is modest, and most trials' confidence intervals include both no effect and a meaningful reduction. ^f Three of the 5 studies are at serious risk of confounding by ROBINS-I, because clinicians referred people they judged likely to benefit; the other two are at moderate risk.

Group-based programs start at high certainty as randomized evidence. The team rated down one level for risk of bias and one for inconsistency, giving low certainty. It judged indirectness not serious because all trials enrolled community-dwelling adults aged 65 and older, and imprecision not serious because the trials together enrolled more than a thousand participants. One-to-one programs also start at high, and the team rated down one level for risk of bias and one for imprecision, again giving low certainty. Community connector programs start at low certainty as non-randomized evidence and were rated down one level for risk of bias, giving very low certainty. Two of these studies showed larger reductions in loneliness among people with more connector contacts, and the team considered rating up for a dose-response gradient. It did not rate up, because the risk of bias was serious and people whose loneliness improved early may have chosen to keep meeting their connector, which would produce the same gradient.

4.7 Communicating Certainty

Certainty ratings help decision-makers when the review states them in consistent, plain terms. Santesso and colleagues (2020) proposed standard wording for the statements in reviews and summaries, linking each level of certainty to a verb phrase. The Cedar Valley evidence brief uses this wording, as the table shows.

In a full review, these statements appear in a summary of findings table, which presents, for each important outcome, the number of studies and participants, the effect and the certainty. Software such as GRADEpro GDT, available in 2026, helps teams build these tables, and Lesson 12 shows how to present them to decision-makers.

The Cedar Valley ratings have a direct implication for the planning team. The community connector model it intends to launch rests on very uncertain evidence about loneliness, so the true effect could be substantially larger or smaller than existing studies suggest. That rating leaves open whether the program works, and it strengthens the case for launching the program with a built-in evaluation, for example by staggering its introduction across clinics, which the evidence brief in Lesson 12 recommends.

Bringing It All Together

This lesson followed the Cedar Valley evidence team from a list of 42 included studies to a statement about how confident the planning team can be in what those studies show. The extraction form, built on the TIDieR items and a data dictionary and piloted until two people recorded the same information, determined what the review could later compare and appraise. The design recorded on that form decided which appraisal tool applied: RoB 2 for the randomized trials, ROBINS-I for the non-randomized controlled studies, the JBI quasi-experimental checklist for the uncontrolled before-and-after studies, and the MMAT for the qualitative and mixed-methods studies, with ROBINS-E and the Newcastle-Ottawa Scale described for the exposure and observational questions where students will meet them.

The tools structure judgement without removing it, which is why calibration, decision rules, recorded support, independent assessment and consensus matter. In the Cedar Valley review, no trial reached low risk of bias overall for loneliness, because participants report their own loneliness and know which program they received. GRADE then moved the question from single studies to bodies of evidence: group-based and one-to-one programs reached low certainty, and community connector programs, the model the health authority intends to launch, reached very low certainty. That rating leaves the program's effect open, which supports launching it with an evaluation built in.

The final reflection asks you to write the extraction, appraisal and certainty methods for a new rapid scoping review, drawing on all four sections.

Final Knowledge Assessment

These terms, tools and people appear in this lesson on data extraction, risk of bias and certainty of evidence.