1.1 A First Wave Chosen for Readiness

The running example is the fictional Cedar Valley Connector program, which runs through this course. The fictional Cedar Valley Health Authority in British Columbia supports primary care clinicians to refer adults aged 65 and older who screen as socially isolated or lonely to a community connector. The connector meets each person up to six times over twelve weeks, co-develops a plan, and links the person to community groups, volunteer roles, transportation help and services. The program launched in 12 of the region's 24 primary care clinics, and the other 12 clinics are scheduled to join in a second wave a year later. All Cedar Valley figures in this lesson are illustrative.

Lesson 6 imagined a version of this rollout in which the health authority had randomized clinics to the two waves. This lesson treats the first wave as it was actually chosen. The health authority selected the 12 first-wave clinics for their readiness: their managers volunteered, they had a room the connector could use, and their electronic medical records could produce a list of older patients who had screened as lonely. Those are sensible operational reasons to start somewhere, and each of them could also be related to the outcomes the program hopes to change. The 12 second-wave clinics, which continue usual care during the program's first year, are the obvious comparison group, but they were not chosen to be comparable.

The first report to the steering committee illustrates the problem. In the first six months, 312 older adults were referred, 241 attended a first meeting, and 188 had both a baseline and a follow-up loneliness measurement on the three-item UCLA Loneliness Scale, which is scored from 3 to 9 with higher scores indicating greater loneliness. The mean score among those 188 people fell from 7.1 to 6.3. A committee member asks whether the 0.8-point fall shows that the program works. This section gives the vocabulary for answering that question, and Sections 2 to 4 give the designs that answer it better than a pre-post comparison can.

1.2 The Counterfactual Problem Restated

Lesson 6 introduced the potential outcomes framework associated with Rubin (1974). Each older adult has a loneliness score Y(1) that they would have at six months if they took part in the program and a score Y(0) that they would have if they did not. Holland (1986) called the impossibility of observing both for the same person the fundamental problem of causal inference. For a program that people choose or are chosen to join, the quantity of most interest is usually the average treatment effect on the treated (ATT), which is the mean of Y(1) − Y(0) among the people who actually received the program. The first term is observed for participants. The second term, the outcome participants would have had without the program, is the counterfactual, and every design in this lesson is a strategy for estimating it from data that can be observed.

Shadish, Cook and Campbell (2002) define a quasi-experiment as a study that shares the purpose and structure of a randomized experiment, with a manipulable cause that occurs before the outcome is measured, but in which units are not assigned to conditions at random. The designs differ in where they find a stand-in for the missing counterfactual. A one-group pre-post design uses the participants' own earlier scores. A posttest-only comparison uses another group's level at the same time. Difference-in-differences uses another group's change over the same period. Matching and weighting use other people who resemble participants on measured characteristics. A synthetic control uses a weighted combination of untreated units that tracks the treated unit before the program. Each stand-in is valid under a different assumption, and the evaluator's task is to choose the stand-in whose assumption is most credible for the program at hand and then to probe that assumption with data.

1.3 The Validity Typology

The vocabulary of threats to validity comes from a line of work that began with Campbell (1957) and Campbell and Stanley (1963), was extended by Cook and Campbell (1979), and was consolidated in Shadish, Cook and Campbell (2002). Their typology distinguishes four kinds of validity, each concerned with a different inference that an evaluation makes.

The name construct validity also has a narrower use in measurement, where it refers to evidence that an instrument's scores relate to other measures and predictions as theory expects, such as convergent and discriminant validity; HSCI 410 Lesson 7, Section 3 (Measurement and Psychometrics), teaches that sense. The broad aim in measurement is that an instrument measures the construct it is meant to measure. The Shadish, Cook and Campbell sense extends that aim from a single instrument to the whole study, asking whether the program, outcomes, people and settings are correctly labelled.

Shadish, Cook and Campbell give internal validity priority when the question is causal, because an estimate of an effect that may not exist cannot be generalized. The priority is a judgement about sequence; it does not make the other types unimportant. For a health authority deciding whether to extend the program to every clinic, external validity matters a great deal, and the readiness criterion that threatens internal validity also limits generalization. Lesson 9 returns to this tension when it discusses the transfer of programs from early adopters to later sites.

1.4 Threats to Internal Validity

A threat to internal validity is a specific reason why the inference that a program caused an observed change might be wrong. Shadish, Cook and Campbell (2002) list nine: ambiguous temporal precedence, selection, history, maturation, regression, attrition, testing, instrumentation, and the additive and interactive effects of these threats. Ambiguous temporal precedence is rarely the main concern in program evaluation, because the program's start date is usually known, although it can arise when people join a program because their outcome has already begun to change. The cards below describe the seven threats that most often decide the credibility of a program evaluation, each with a Cedar Valley example. Click a card to open it.

The threats also combine. Shadish, Cook and Campbell stress the interactions of selection with the other threats, because they are the threats that survive the addition of a comparison group. A selection-maturation interaction occurs when the program and comparison groups would have changed at different rates without the program, for example if first-wave clinics serve neighbourhoods where emergency department use is falling faster. A selection-history interaction occurs when an event affects one group and not the other, such as an urgent care centre that opens near several first-wave clinics. A selection-instrumentation interaction occurs when measurement differs between groups, and a selection-regression interaction occurs when one group was selected on more extreme scores than the other. Section 2 shows that the key assumption of difference-in-differences is, in effect, the assumption that these interactions are absent.

Regression to the mean in the Cedar Valley pre-post result

Regression to the mean was first described by Galton (1886) in his studies of the heights of parents and children, and Barnett, van der Pols and Dobson (2005) give a clear account of its consequences for health research. Any measure with less than perfect reliability contains a component of chance, and people selected for extreme scores are disproportionately those whose chance component was extreme on the day they were measured. On remeasurement, the chance component is drawn afresh and the group's mean moves back toward the population mean.

Suppose, as illustrative assumptions, that the mean UCLA score among all older adults in the region is 4.6, that the correlation between two administrations of the scale a few months apart is 0.70, and that the baseline score used in the analysis is the screening score on which people were referred. The expected follow-up mean among the 188 participants, with no program effect at all, would be 4.6 + 0.70 × (7.1 − 4.6) = 4.6 + 1.75 = 6.35. The observed follow-up mean was 6.3. Under these assumptions, regression to the mean could account for 0.75 of the observed 0.8-point fall. The table shows how the answer depends on the reliability of the measure.

The calculation does not show that the program had no effect, and it does not show that the program had an effect. It shows that the pre-post change cannot distinguish the two explanations without information that the one-group design does not contain. Three design responses reduce the problem. The evaluator can take a second baseline measurement at the first connector meeting and use it as the pretest, so that the selection measurement and the pretest are separate. The evaluator can select a comparison group by the same rule, for example older adults in second-wave clinics who scored 6 or higher at screening, so that both groups regress by similar amounts. The evaluator can also analyze follow-up scores with adjustment for baseline scores, which removes the part of the change predicted by the baseline score when the groups were selected in the same way.

1.5 Design Notation

Campbell and Stanley (1963) introduced a compact notation for research designs, and Shadish, Cook and Campbell (2002) use an updated version of it. Each row represents a group and time runs from left to right. X denotes the program, O denotes an observation (a measurement of the outcome), and subscripts number the observations in time order. R at the start of a row indicates random assignment and NR indicates nonrandom assignment. Writing a design in this notation forces the evaluator to state exactly what is observed, when, and for whom, and it makes the threats that each design leaves open easier to see.

Which design addresses which threat

The table summarizes the threats that each design rules out by its structure, assuming that the comparison group is measured with the same instrument, over the same period, by the same procedures. The entries describe plausibility; a threat marked as addressed can still operate if those assumptions fail.

Attrition remains open in every design because it arises after assignment. It is handled by keeping follow-up rates high, by comparing the baseline characteristics of those lost and those retained, and by analyses such as multiple imputation or inverse probability of censoring weights under stated assumptions. Administrative outcomes such as emergency department visits are less affected by attrition than survey outcomes, because they are recorded for everyone who remains enrolled at a clinic, which is one reason the Section 2 worked example uses them.

Section 2 of 5

Comparison Group Designs and Difference-in-Differences

Comparing changes, checking parallel trends and handling staggered rollouts.

Three estimates

Same data, three answers

Posttest-only: −17.1

The estimate includes the gap of about 9.8 visits that existed before launch.

Pre-post: −3.7

The estimate ignores the rise of 3.6 visits in the comparison clinics.

DiD: −7.3

The program group's change minus the comparison group's change.

All values are emergency department visits per 1,000 older patients per quarter.

The key assumption

Parallel trends

Untreated changes are equal

\[ E[Y(0)_{post} - Y(0)_{pre} \mid P] = E[Y(0)_{post} - Y(0)_{pre} \mid C] \]

• The groups may differ in level, and they must not differ in their untreated paths.
• No anticipation, stable composition and no spillover are also required.

Worked example

Group means by quarter

Fixed effects and clustering

Same estimate, honest uncertainty

−7.31Two-way fixed effects estimate

2.11Conventional standard error

3.34Clinic-clustered standard error

The 95 percent confidence interval with clustering runs from −14.2 to −0.4.

Event study

The gap, quarter by quarter

Staggered adoption

When two-way fixed effects mislead

−3.83True average effect in the toy rollout

−1.17Two-way fixed effects estimate

+1.50Comparison that uses wave 1 as the control

Estimators that use only not-yet-treated or never-treated comparisons avoid the problem.

Carry forward

From clinics to people

• Difference-in-differences relies on parallel trends, which pre-launch data can probe.
• Standard errors should be clustered by the unit that receives the program.
• Section 3 builds individual comparison groups with propensity scores.

2.1 From Comparison Groups to Difference-in-Differences

Section 1 described the Cedar Valley rollout as a switching replications design with nonrandom assignment. During the program's first year, the 12 first-wave clinics deliver the program and the 12 second-wave clinics do not. The health authority's administrative records give the number of emergency department visits made each quarter by the patients aged 65 and older enrolled at each clinic, so the evaluator can compute a visit rate per 1,000 enrolled older patients for every clinic in every quarter, both before and after launch. This section uses that clinic-by-quarter panel to estimate the program's effect on emergency department use.

Two simple comparisons are available, and each carries a known bias. A posttest-only comparison of first-wave and second-wave clinics after launch estimates the program effect plus the difference that already existed between the groups. A one-group pre-post comparison of first-wave clinics before and after launch estimates the program effect plus whatever change would have occurred anyway, from secular trends, seasonal patterns, maturation and history. Difference-in-differences (DiD) combines the two. It takes the change in the program group and subtracts the change in the comparison group over the same period, so that a stable difference between the groups and a change shared by both groups are both removed.

The logic is old. John Snow's comparison of cholera deaths among households supplied by two London water companies in the 1850s is often cited as an early example. In economics, Card and Krueger (1994) compared employment in fast-food restaurants in New Jersey, which raised its minimum wage, with restaurants in neighbouring eastern Pennsylvania, which did not, before and after the increase. In public health, Wing, Simon and Bello-Gomez (2018) review the design's use for evaluating policies and programs and set out practices for doing it well.

2.2 The Parallel Trends Assumption

DiD replaces the assumption of exchangeable levels with an assumption about changes. The parallel trends assumption states that, in the absence of the program, the average outcome in the program group would have changed by the same amount as the average outcome in the comparison group. In potential outcomes notation, E[Y(0)_post − Y(0)_pre | P] = E[Y(0)_post − Y(0)_pre | C]. The assumption allows the groups to differ in level, which is why DiD suits Cedar Valley, where clinics chosen for readiness had lower emergency visit rates before launch. It does not allow the groups to be on different paths. In the language of Section 1, it assumes away selection-maturation, selection-history, selection-instrumentation and selection-regression interactions.

Four further conditions are usually stated alongside it. No anticipation requires that the program not change outcomes before it starts, as it might if first-wave staff began contacting isolated patients while preparing for launch. Stable composition requires that the population measured in each group not change because of the program, as it would if older adults moved to first-wave clinics to see a connector. No spillover would fail if second-wave patients joined community groups that connectors had strengthened. Finally, parallel trends in rates and parallel trends in the logarithm of rates are different assumptions, and at most one can hold when the groups start at different levels, so the scale should be chosen on substantive grounds before the results are seen.

The assumption fails in a characteristic way when people or units enter a program because of a recent change in their outcome. Ashenfelter (1978) observed that the earnings of people entering job training programs dipped in the year before entry, so that their later recovery would look like a program effect in a pre-post or DiD comparison. A health analogue is a clinic chosen for a program after a bad year, or an older adult referred after an unusual run of emergency visits. Several pretest periods are the best protection, because they show whether the groups were moving together before the program and whether a dip preceded entry.

2.3 Worked Example: Difference-in-Differences for the Cedar Valley Clinics

Worked example: Difference-in-differences estimates for the Cedar Valley clinics

Group means by quarter. The panel is balanced, with 96 pre-launch and 48 post-launch clinic-quarters in each group. The graphical check of parallel trends plots the mean rate in each group in each quarter.

Before launch, first-wave clinics have lower rates in every quarter, with gaps between −8.63 and −12.53 visits per 1,000, and the gap in quarter 8, the last quarter before launch, is −8.75. Both series rise and fall together, peaking in quarters 4 and 8, which are the winter quarters in this dataset. The gap does not drift in one direction over the eight pre-launch quarters, which is what parallel trends predicts. After launch the gap widens to −11.88, −17.08, −22.66 and −16.93 in quarters 9 to 12.

The two-by-two estimate by hand. The table gives the four group means, in emergency department visits per 1,000 enrolled patients aged 65 and older per quarter, and the change in each group.

Clinics	Before launch	After launch	Change
First wave	95.768	92.023	−3.745
Second wave	105.595	109.158	3.564

First-wave clinics fell by 3.745 visits per 1,000 per quarter after launch, while second-wave clinics rose by 3.564, so the DiD estimate is −3.745 − 3.564 = −7.308. The posttest-only comparison of the two groups after launch (−17.135) more than doubles this, because it includes the pre-existing gap of about 9.8 visits. The pre-post comparison within first-wave clinics (−3.745) understates it, because it ignores the rise in rates that the comparison clinics show over the same period. The simulated effect is deliberately large so that the methods are easy to see. Spread over the roughly 500 older adults who attend a first meeting in the first year, a fall of 7.3 visits per 1,000 enrolled older patients per quarter would mean more than one avoided visit per participant each year, far more than the 0.12 visits that Lesson 10 assumes in its economic evaluation, so a real evaluation would check an estimate of this size against the program’s reach.

Background: how a product term reproduces four group means

The regression below contains two indicators, wave1 (1 for first-wave clinics) and post (1 for quarters after launch), and their product, wave1 × post, which equals 1 only for first-wave clinics after launch. Substituting the four combinations of 0 and 1 shows what each coefficient means. For second-wave clinics before launch every indicator is 0, so the prediction is β₀. For second-wave clinics after launch it is β₀ + β₂, for first-wave clinics before launch β₀ + β₁, and for first-wave clinics after launch β₀ + β₁ + β₂ + β₃. With four coefficients and four groups, the model reproduces the four means exactly. The first-wave change is β₂ + β₃ and the second-wave change is β₂, so their difference, the DiD estimate, is β₃, the coefficient on the product term. A product term is how a regression represents statistical interaction: it allows the change over time to differ between the two waves.

HSCI 410 Lesson 3, Section 4 (Linear and Logistic Regression), covers interaction terms more fully and is optional reading.

The same estimate by regression. The regression Y = β₀ + β₁ wave1 + β₂ post + β₃ (wave1 × post) reproduces the four means, and β₃ is the DiD estimate. Standard errors are clustered by clinic, and the t tests use 23 degrees of freedom (the number of clinics minus one). The intercept (105.5948) is the comparison group's pre-launch mean, the wave1 coefficient (−9.8271) is the pre-launch difference between the groups, the post coefficient (3.5635) is the comparison group's change, and the interaction (−7.3083) is the DiD estimate computed by hand. Its clustered standard error is 3.1503, giving t = −2.3199 and p = 0.0296.

Clinic and quarter fixed effects. The two-way fixed effects (TWFE) model replaces the group indicator with a separate intercept for each clinic and the period indicator with a separate intercept for each quarter. Its program indicator equals 1 for first-wave clinics after launch. The table compares conventional and clinic-clustered standard errors for the program effect.

Type of standard error	Estimate	Standard error	t	p
Conventional	−7.3083	2.1087	−3.4658	0.0006
Clustered by clinic (23 degrees of freedom)	−7.3083	3.3444	−2.1853	0.0393

In a balanced panel with a single launch date, the TWFE estimate equals the DiD of group means (−7.3083). The standard errors differ. The conventional standard error (2.1087) treats the 288 clinic-quarters as independent, whereas a clinic's rate in one quarter is correlated with its rate in neighbouring quarters. The clustered standard error (3.3444) allows any pattern of correlation within a clinic and is about 1.59 times larger. The 95 percent confidence interval based on the clustered standard error runs from −14.227 to −0.39 visits per 1,000 per quarter.

Bertrand, Duflo and Mullainathan (2004) showed that ignoring serial correlation in DiD studies with many periods produces standard errors that are far too small and tests that reject a true null hypothesis much more often than the nominal 5 percent. Clustering by the unit at which the program is assigned, here the clinic, is the standard response. Clustered standard errors rely on the number of clusters being reasonably large, and with 24 clinics the worked example uses a t distribution with 23 degrees of freedom as a small-sample correction; Cameron and Miller (2015) describe further options, such as the wild cluster bootstrap, for studies with fewer clusters. The clustered standard error from the fixed-effects model (3.3444) is slightly larger than the one from the two-by-two regression (3.1503) only because the usual small-sample adjustment multiplies the variance by (n − 1)/(n − k), and k is 36 in the fixed-effects model and 4 in the two-by-two model.

2.4 Event-Study Plots

The two-by-two estimate averages over all pre-launch and all post-launch quarters. An event study estimates a separate coefficient for each quarter relative to launch, so that the evaluator can see whether the groups moved together before launch and how the effect develops afterwards. The model is

Worked example: An event study for the Cedar Valley clinics

The event-study model was fitted to the Cedar Valley panel with one indicator for each lead and lag except event time −1, clinic and quarter fixed effects, and clinic-clustered standard errors with 23 degrees of freedom. A joint test checks whether the seven leads all equal zero.

The seven leads lie between −3.77 and 0.12, with standard errors between 5.22 and 5.96, and the joint test gives F = 0.135 on 7 and 23 degrees of freedom (p = 0.994), so the pre-launch data show no sign of diverging trends. The lags are −3.13, −8.32, −13.91 and −8.18, a pattern consistent with an effect that builds as connectors' caseloads grow. Each lag is imprecise on its own, and every 95 percent interval includes zero. In a balanced panel with one launch date, each coefficient equals the raw gap in that quarter minus the gap in quarter 8: for event time 1 (quarter 10), −17.08 − (−8.75) = −8.33, which matches −8.32 up to rounding. The two-by-two estimate is the mean of the four lags minus the mean of the eight pre-launch coefficients, (−8.385) − (−1.076) = −7.309, which matches the two-by-two estimate in Section 2.3 up to rounding.

Flat leads are reassuring, and they are weaker evidence than they appear. The parallel trends assumption concerns the post-launch period, in which the counterfactual trend is unobservable, and pre-launch agreement shows only that the groups moved together when neither had the program. Pre-trend tests also have low power when, as here, each lead has a standard error of about 5.5; a real divergence of a few visits per quarter could easily pass the test. Roth (2022) showed that reporting results only when a pre-trend test passes can make the bias of the selected estimates worse. Rambachan and Roth (2023) propose a sensitivity analysis that asks how large a post-launch violation of parallel trends, relative to the largest pre-launch deviation, would be needed to overturn the conclusion. For Cedar Valley, the evaluator would also check the plausible sources of divergence directly, by asking whether urgent care services, clinic staffing or record systems changed differently in the two groups of clinics during the study period.

2.5 Staggered Adoption and Two-Way Fixed Effects

The clean comparison in the worked example lasts only one year. When the second wave launches in quarter 13, every clinic is receiving the program and the rollout becomes a case of staggered adoption, in which units start at different times. A common approach has been to fit the TWFE model from Section 2.3 to the full period, with the program indicator switched on for each clinic from its own launch date. A body of econometric work, most of it published since 2020, has shown that this estimate can be misleading when program effects change over time or differ between adoption groups (Goodman-Bacon, 2021; de Chaisemartin and D'Haultfœuille, 2020; Callaway and Sant'Anna, 2021; Sun and Abraham, 2021; Roth et al., 2023).

Goodman-Bacon (2021) showed that the TWFE estimate under staggered adoption is a weighted average of every two-group, two-period comparison in the data. Some comparisons use not-yet-treated units as controls, which is legitimate under parallel trends. Others use already-treated units as controls for later adopters. When the earlier group's effect is still growing, its change during the later group's launch includes its own growing effect, and subtracting that change biases the comparison. The worked example below shows this with a noiseless version of the full Cedar Valley rollout.

Several estimators avoid the problem by building each comparison only from units that are not yet treated or never treated, and then averaging the resulting estimates with weights the analyst chooses. The accordion summarizes the main approaches, each of which is available in standard statistical software.

Section 3 of 5

Propensity Scores

Estimation, matching, weighting, balance and the problem of unmeasured confounding.

The propensity score

One number that balances many covariates

Definition

\[ e(X) = \Pr(T = 1 \mid X) \]

No unmeasured confoundingPositivity and overlapConsistencyNo interferenceA model that balances

Worked example

Estimating scores and checking overlap

Matching

Nearest-neighbour matching with a caliper

410Matched pairs

13Participants without a match

0.1474Caliper on the logit scale

All 13 unmatched participants had propensity scores of 0.659 or higher.

Balance diagnostics

Standardized mean differences

The limits of matching

Balanced, precise and still biased

−0.68Matched estimate (95 percent CI −0.82 to −0.54)

−0.42True effect built into the simulation

Balance can be checked only on covariates that were measured.

Carry forward

From individuals to whole jurisdictions

• Propensity scores balance measured covariates and make the estimand explicit.
• No diagnostic can detect an unmeasured confounder, so sensitivity analysis is part of the design.
• Section 4 turns to single treated units and the choice among designs.

3.1 The Individual-Level Question

Section 2 estimated the program's effect on a clinic-level outcome, emergency department visits, using a long series of administrative data. The steering committee of the fictional Cedar Valley Connector program also wants to know whether the program reduces loneliness among the older adults who take part. Loneliness is measured with the three-item UCLA Loneliness Scale (3 to 9, higher scores indicating greater loneliness) at enrolment and six months later, so there is one pre-program measurement per person. A comparison group is available because second-wave clinics screened their older patients for loneliness during the program's first year with the same instrument and followed up a sample of them six months later. The natural comparison is between program participants in first-wave clinics and screened older adults in second-wave clinics.

The two groups differ by design. Participants were referred by clinicians, usually because they scored 6 or higher at screening, and then chose to attend. They are therefore likely to be lonelier at baseline, more likely to live alone, and perhaps more motivated to change their circumstances than the comparison group. Each of these characteristics can also affect loneliness six months later.

3.2 The Propensity Score

The propensity score is the probability of receiving the program given a person's measured pre-program characteristics, e(X) = Pr(T = 1 | X). Rosenbaum and Rubin (1983) showed that it is a balancing score: among people with the same propensity score, the distribution of the measured covariates X is the same in the program and comparison groups. If assignment to the program is unconfounded given X, it is also unconfounded given e(X). The result reduces the problem of comparing people who are alike on many covariates to the problem of comparing people who are alike on one number, which can then be used for matching, weighting, stratification or regression adjustment. In the notation of this lesson, T = 1 indicates receipt of the program and X is the set of measured pre-program covariates.

The propensity score does not remove the need for assumptions. It moves them into a form that the evaluator can state and partly check. The cards below set out the assumptions on which a propensity score analysis rests.

Propensity score methods target a specific estimand. The average treatment effect (ATE) is the average effect in the whole population of participants and comparisons. The average treatment effect on the treated (ATT) is the average effect among the people who actually received the program, and it is usually the estimand of interest for an existing program, because it describes what the program did for the people it served. Matching each participant to one or more comparisons, as below, targets the ATT.

The choice of covariates is a design decision that should be made before the outcomes are examined. Covariates should be measured before the program began, so that the program cannot have affected them; the number of connector meetings or six-month social participation should never enter a propensity model. Brookhart and colleagues (2006) showed in simulations that including variables related to the outcome improves precision without increasing bias, whereas variables related only to receipt of the program increased variance without reducing bias; later work showed that such variables can also amplify bias from unmeasured confounding (Myers et al., 2011). The pre-program measurement of the outcome, here the baseline UCLA score, is usually the most important single covariate, because it captures many unmeasured influences on later scores and because matching on it means that both groups regress toward the mean by similar amounts.

3.3 Worked Example: Propensity Score Matching for Cedar Valley Participants

Worked example: Describing the groups and estimating propensity scores

The groups. The table gives the mean of each covariate and of the six-month outcome in each group.

Variable	Comparisons (n = 877)	Participants (n = 423)
Age (years)	76.029	75.452
Female (proportion)	0.592	0.603
Lives alone (proportion)	0.351	0.518
Baseline UCLA score	5.497	6.345
Chronic conditions	2.152	2.310
Emergency department visits in the prior year	0.491	0.539
Six-month UCLA score	5.696	5.657

Participants were lonelier at baseline (6.345 against 5.497) and more often lived alone (0.518 against 0.351). Their six-month mean (5.657) is almost the same as the comparison group's (5.696), so the unadjusted difference is −0.0383 (standard error 0.0823, p = 0.6414) and suggests no effect. Adjusting for the six covariates by ordinary regression changes the estimate to −0.6983 (standard error 0.0612), because participants started out lonelier and would have been expected to remain lonelier without the program.

The propensity score. A logistic regression of program receipt on the six covariates estimates each person's propensity score. Each additional point on the baseline UCLA score multiplies the odds of being a participant by 1.550 (z = 9.088), living alone multiplies them by 1.775 (z = 4.455), and each additional year of age multiplies them by 0.979 (z = −2.289). The odds ratios for sex, chronic conditions and prior emergency department visits lie between 0.901 and 1.113, with smaller z statistics. Propensity scores range from 0.065 to 0.780 among comparisons and from 0.083 to 0.802 among participants, with means of 0.293 and 0.393. The groups overlap over most of the range, but comparisons become sparse above about 0.65, which is where matching will struggle.

3.4 Matching

Nearest-neighbour matching pairs each participant with the comparison whose propensity score is closest. In the greedy form used here, participants are matched one at a time, each comparison can be used only once (matching without replacement), and each participant receives one match (1:1 matching). A caliper sets the largest acceptable distance between matched scores, and participants with no comparison within the caliper are left unmatched. Austin (2011) recommended a caliper of 0.2 standard deviations of the logit of the propensity score, based on simulations showing that it removes most of the bias that matching can remove while keeping most participants.

Dropping unmatched participants changes the estimand. The matched analysis estimates the average effect among the 410 matched participants, who exclude the participants with the highest propensity to enrol. Here the change is small, because 97 percent of participants were matched, but the report should state it and describe the people who were dropped. When many participants fall outside the caliper, the evaluator faces a choice between a less biased estimate for a subgroup and a more biased estimate for everyone, and the choice should be made explicitly.

3.5 Balance Diagnostics

Because the purpose of a propensity score is to balance covariates, the main diagnostic is the balance achieved. The standard measure is the standardized mean difference (SMD), which expresses the difference in a covariate's mean between groups in standard deviation units and so does not depend on sample size (Austin, 2009).

Worked example: Balance before and after matching

The table gives each covariate's SMD before and after matching, using the pooled standard deviation of the unmatched sample in both cases so that the change reflects only the change in means, and the variance ratio after matching. The figure shows the same SMDs as a balance plot.

Covariate	SMD before matching	SMD after matching	Variance ratio after matching
Age	−0.082	0.032	1.077
Female	0.023	−0.010	1.004
Lives alone	0.341	0.015	1.000
Baseline UCLA score	0.632	0.049	1.094
Chronic conditions	0.103	0.002	0.988
Emergency department visits in the prior year	0.065	−0.050	0.836

Before matching, the baseline UCLA score (0.632) and living alone (0.341) were badly imbalanced and the number of chronic conditions (0.103) was just above the threshold. After matching, every absolute SMD is 0.050 or smaller, and the variance ratios range from 0.836 to 1.094. On the measured covariates, the matched groups resemble each other as closely as two randomized groups of this size typically would.

Two practices make balance checking honest. Balance should be assessed with SMDs and plots, without significance tests, because a p-value for a difference in means falls or rises with the sample size, and matching changes the sample size (Imai, King and Stuart, 2008). The design should also be finalized before the outcome is examined. Rubin (2001) argued that the matching stage of an observational study corresponds to the design stage of a trial: the analyst may revise the propensity model, add interactions or change the caliper until balance is acceptable, but should do so without seeing outcomes, so that the choice of specification cannot be steered by the result.

3.6 Estimating the Effect

The effect is the difference in mean six-month scores in the matched sample. Standard errors are clustered on the matched pair, because members of a pair are similar. A second model adds the covariates to remove residual imbalance.

3.7 Inverse Probability Weighting

Inverse probability weighting (IPW) uses the propensity score to reweight the sample instead of discarding unmatched people. Each person is weighted by the inverse of the probability of the group they are actually in, which creates a pseudo-population in which the covariates are unrelated to program receipt. The weights depend on the estimand.

Weights become large when a propensity score approaches 0 or 1, and a few large weights make the estimate unstable. Under ATE weighting, the Cedar Valley participant with the lowest score (0.083) would receive a weight of 1 / 0.083 = 12.05, standing in for twelve people. Stabilized weights, which multiply each weight by the marginal probability of the group, and truncation of extreme weights at chosen percentiles are common responses (Cole and Hernán, 2008; Austin and Stuart, 2015). Weighted balance should be checked with SMDs, exactly as for matching.

Matching, weighting and regression adjustment can be combined. Augmented inverse probability weighting fits both a propensity model and an outcome model and gives a consistent estimate if either model is correctly specified, which offers some protection against misspecification of one of them (Hernán and Robins, 2020). Regression adjustment within a matched sample, as in the second model of the worked example in Section 3.6, follows the same logic.

The same weights extend to programs whose receipt changes over time. When participation can start, stop or intensify across several periods, and a covariate that affects later participation and the outcome is itself affected by earlier participation, ordinary regression adjustment fails: adjusting for that covariate blocks part of the program's effect and can open other non-causal paths, while leaving it out leaves confounding. Weights computed at each time point from the covariate and program history up to that point, and multiplied across time points, create a pseudo-population in which program receipt at each time is unrelated to past covariates, and a model fitted in that pseudo-population is a marginal structural model. HSCI 230 Lesson 11, Section 1 (Confounding and Statistical Inference), introduces time-varying confounding at undergraduate level, and Hernán and Robins (2020) give a full treatment of marginal structural models.

3.8 The Limits of Matching

All the adjusted estimates in the worked examples lie between −0.6756 and −0.7239, they agree closely, and the matched groups are balanced on every measured covariate. They are nevertheless biased. The simulation that generated the data included an unrecorded trait, readiness to engage, that made people more likely to enrol and also lowered their later loneliness. The true average effect among participants in the simulated data is about −0.42, so the matched estimate of −0.6756 overstates the benefit by about 0.26 points, and its 95 percent confidence interval excludes the true value. No balance diagnostic could have revealed this, because balance can be checked only on covariates that were measured.

Several strategies reduce the risk or quantify it. The first is design: measure the likely reasons for selection, such as motivation, social support and clinician judgement, at baseline in both groups. Within-study comparisons, in which a randomized experiment and a nonrandomized study of the same intervention are compared, suggest that adjusted estimates come closest to the experimental benchmark when the covariates are rich, well measured and related to the selection process (Shadish, Clark and Steiner, 2008), and a reanalysis of the same study found that pretest measures and measures of why people chose a condition removed most of the bias (Steiner et al., 2010). The second is to combine matching with difference-in-differences when pre-program outcomes are available, so that stable unmeasured differences are also removed. The third is a negative control outcome, an outcome the program cannot plausibly affect but that shares the suspected confounding (Lipsitch, Tchetgen Tchetgen and Cohen, 2010); an apparent "effect" on it signals residual confounding. The fourth is sensitivity analysis. Rosenbaum (2002) developed bounds that show how strongly an unmeasured confounder would need to affect assignment to change a conclusion, and VanderWeele and Ding (2017) proposed the E-value, the minimum strength of association, on the risk ratio scale, that an unmeasured confounder would need with both the program and the outcome to explain away an estimate.

4.1 Evaluating a Single Treated Unit

Many health policies and programs apply to a whole jurisdiction at once. A province changes a drug coverage rule, a city introduces free transit for older residents, or a health authority adopts a program in every clinic. The evaluation then has one treated unit, and the comparison must come from other jurisdictions. Difference-in-differences with a single treated unit is possible, but it depends heavily on which comparison units the evaluator chooses, and with one treated unit the usual standard errors are unreliable.

The synthetic control method was developed by Abadie and Gardeazabal (2003), who studied the economic costs of conflict in the Basque Country, and by Abadie, Diamond and Hainmueller (2010), who estimated the effect of California's 1988 tobacco control program (Proposition 99) on cigarette sales by comparing California with a weighted combination of other states. Instead of choosing one comparison unit or averaging all of them equally, the method builds a synthetic control: a weighted average of untreated units from a donor pool, with weights chosen so that the weighted average reproduces the treated unit's outcomes and characteristics before the intervention. After the intervention, the synthetic control's trajectory estimates what would have happened to the treated unit without it.

The restriction to non-negative weights that sum to one has two consequences. The synthetic control is an interpolation among real units, so it cannot extrapolate beyond the range of the donors, and the weights are transparent: the evaluator can report exactly which units contribute and how much. Abadie (2021) sets out the conditions under which the method is credible. The treated unit should be closely matched over a long pre-intervention period, the donor pool should exclude units that adopted similar interventions or suffered large idiosyncratic shocks, the intervention should not have been anticipated, and the intervention should not spill over to donor units.

4.2 A Worked Example: Cedar Valley as a Whole

Suppose that, once both waves are running, the fictional Cedar Valley Health Authority wants to know whether emergency department use among all residents aged 65 and older in the region has changed. The whole region is now the treated unit. The evaluator assembles annual emergency department visit rates per 1,000 residents aged 65 and older for Cedar Valley and for 14 other health service areas that have no comparable program and that use the same visit definitions. The year in which the first wave launched is the start of the intervention, so year 1 includes only first-wave clinics and year 2 includes both waves. The optimization assigns weights of 0.50 to Area A, 0.30 to Area B, 0.20 to Area C, and zero to the other 11 areas. All values are illustrative.

Each synthetic value is the weighted sum of the donors. For year −6 it is 0.50 × 380 + 0.30 × 432 + 0.20 × 401 = 190.0 + 129.6 + 80.2 = 399.8. No single donor tracks Cedar Valley: Area A is consistently lower and Area B consistently higher, but their weighted combination with Area C stays within 1.8 visits of Cedar Valley in every pre-intervention year. The pre-intervention RMSPE is √[(1.2² + 0.8² + 0.8² + 1.8² + 0.8² + 0.2²) / 6] = √(6.64 / 6) = 1.05 visits. After the program starts, Cedar Valley falls below its synthetic control by 6.0 visits in year 1 and 8.7 visits in year 2, and the post-intervention RMSPE is √[(6.0² + 8.7²) / 2] = 7.47, about 7.1 times the pre-intervention value.

Inference with placebo tests

With one treated unit, conventional standard errors do not apply. Abadie, Diamond and Hainmueller (2010) proposed placebo tests instead. In an in-space placebo test, the method is applied to each donor area in turn as if it had been treated, using the remaining areas as its donor pool, and the ratio of post-intervention to pre-intervention RMSPE is computed for every area. If Cedar Valley's ratio of 7.1 were the largest among the 15 areas, the probability of observing a ratio that large by chance assignment of the program to an area would be 1/15 = 0.067, which is the smallest p-value this donor pool can produce. If Cedar Valley ranked second, the p-value would be 2/15 = 0.13. An in-time placebo moves the start date back, for example to year −3, and checks that no gap opens at the false date. A leave-one-out analysis drops each donor with a positive weight in turn and checks whether the estimate depends on a single area.

The example also shows a substantive limit. Only a small fraction of the region's older residents take part in the program in any year, so a region-wide gap of 8.7 visits per 1,000 implies a large effect per participant. Before attributing the gap to the program, the evaluator should check whether the program's reach could plausibly produce it and look for other region-wide changes in the post-intervention years, such as a change in emergency department triage or a new urgent care service. A synthetic control controls for influences that the donors share; it cannot rule out an event that affected Cedar Valley alone.

Extensions

Several extensions address the method's limits. The augmented synthetic control method (Ben-Michael, Feller and Rothstein, 2021) corrects the estimate with an outcome model when the pre-intervention fit is imperfect. Synthetic difference-in-differences (Arkhangelsky et al., 2021) combines unit weights of the synthetic control kind with time weights and fixed effects, which makes the estimator less sensitive to level differences. The generalized synthetic control method (Xu, 2017) handles several treated units with different start dates. Bouttell and colleagues (2018) describe the use of synthetic control methods for population-level health interventions. When a single treated unit has a long series but no credible donors, the interrupted time series designs of Lesson 8 compare the unit with its own pre-intervention trend.

4.3 Comparing the Designs

The designs in this lesson differ in the comparison they construct, the assumption that makes the comparison valid, and the data they need. The table sets them side by side. No design removes every threat, and the evaluator's task is to choose the design whose assumption is most defensible for the program and then to collect the evidence that tests that assumption.

The choice usually follows from the data that exist or can be collected. The tabs set out the common situations.

Designs can also be combined. In Cedar Valley, a clinic-level difference-in-differences for emergency department visits, an individual-level matched comparison for loneliness, and a region-wide synthetic control make different assumptions and are vulnerable to different biases. Lawlor, Tilling and Davey Smith (2016) describe this strategy as triangulation: when approaches with different sources of bias point in the same direction, confidence in the conclusion increases, and when they disagree, the disagreement shows where further work is needed.

4.4 Choosing a Comparison Group

Whichever design is chosen, its credibility depends on the comparison group. A good comparison group meets the same eligibility rules as the program group, is measured with the same instrument, mode, timing and data system, is observed over the same calendar period, does not receive the program or a similar one, has similar outcome levels and trends before the program, contains enough units for inference, and has data that the evaluation can legally and practically obtain. Lesson 5 covered data sharing agreements, which often determine whether a comparison group that looks ideal on paper is available in practice.

Comparison groups usually come from one of five sources. Later-wave or waitlisted sites, like the Cedar Valley second wave, share the program's selection process to some degree and are often the best available option, although their window as a comparison closes when they start. Similar sites in neighbouring regions avoid contamination but may differ in health system context. Eligible non-participants at the same sites share the setting but differ in motivation, which is strong self-selection. Historical cohorts at the same sites remove selection between places but are exposed to every history and instrumentation threat between the two periods. Synthetic comparisons built from many units are suited to aggregate outcomes. For each candidate, the evaluator should ask which of the Section 1 threats it leaves open and what evidence could show whether those threats operated.

4.5 Worked Example: The Cedar Valley Comparison Group Memo

The memo closes by stating which threats remain after these steps. The comparison window is only four quarters, so long-term effects cannot be estimated against an untreated group, and an unmeasured difference in motivation between participants and comparisons can be bounded but not eliminated. These limits carry forward into the Cedar Valley design justification in Lesson 8.

Bringing It All Together

This lesson began with the counterfactual problem. Every causal claim about a program compares what happened to participants with what would have happened to them without the program, and the second quantity is never observed. A comparison group stands in for it, and the selection bias decomposition shows that the stand-in works only when the groups would have had the same untreated outcomes. Shadish, Cook and Campbell's threats to internal validity, written down with design notation, give the evaluator a systematic way to ask what else could explain an observed change.

The designs in the lesson each replace exchangeability with a different assumption. Difference-in-differences assumes parallel trends and allows stable differences in level, and the Cedar Valley clinic panel showed how it differs from posttest-only and pre-post comparisons, how clustering changes its standard error, and how an event study displays the evidence before and after launch. Propensity score matching and weighting assume no unmeasured confounding, and the Cedar Valley participant data showed that excellent balance on measured covariates can coexist with substantial bias from a trait that was never recorded. The synthetic control method assumes that a weighted combination of donors that tracks a single treated unit before an intervention would have continued to track it afterwards.

Choosing among these designs depends on the data available and on which assumption is most defensible for the program, and combining designs with different weaknesses strengthens the conclusions that can be drawn. The Cedar Valley memo in Section 4.5 shows this choice made for one program, naming a comparison group and the threats it leaves open.

The final reflection asks you to design a comparison-group evaluation for a staggered rollout that combines the methods of this lesson.

Final Knowledge Assessment

Terms, methods and people introduced in this lesson on comparison groups, difference-in-differences, propensity scores and synthetic control.