# Lesson 3: Program Theory: Logic Models and Theories of Change

*Companion-podcast transcript, Sarah and Kiffer*

---

**Sarah:** Welcome back to Office Hours. I'm Sarah.

**Kiffer:** And I'm Kiffer. This week we are on Lesson three of Program Planning and Evaluation, which is about program theory, logic models and theories of change.

**Sarah:** I'll admit that "logic model" is a phrase that makes some of our listeners groan. They have filled in the boxes for a funding application and moved on.

**Kiffer:** That reaction usually comes from seeing logic models used as paperwork. Every program carries an argument about how change happens, and the logic model and the theory of change are two ways of writing that argument down so that someone can check it.

**Sarah:** Let's start with that argument, then. What is program theory?

**Kiffer:** Leonard Bickman gave a definition in nineteen eighty-seven that still works well. He called program theory a plausible and sensible model of how a program is supposed to work. Rossi, Lipsey and Henry, whose textbook underpins this course, add that it covers both the assumptions linking a program's activities to its benefits and the strategy the program uses to deliver those activities.

**Sarah:** So there are two parts.

**Kiffer:** Yes. One part is causal. It explains why the activities should produce change. The other part is operational. It explains what has to be organized, by whom and for whom, so that the activities actually happen.

**Sarah:** Give me the causal part for our running case.

**Kiffer:** The Cedar Valley Connector program is fictional, and we have used it since Lesson one. Clinicians refer adults aged sixty-five and older who screen as lonely to a community connector. The connector meets them up to six times over twelve weeks, works out a plan with them, and links them to groups, volunteer roles, transportation help and services. The causal argument is that loneliness partly reflects a lack of opportunity, that a trusted person can help someone take up opportunities they would not take up alone, that taking part produces relationships people value, and that less loneliness eventually means better health and fewer emergency visits.

**Sarah:** Each of those could be wrong.

**Kiffer:** Each of them could be wrong, and that is the point of writing them down. Most programs never state these beliefs in one place, and different people in the same program often hold different versions of them.

**Sarah:** The reading separates program theory from two other kinds of theory. Why does that matter?

**Kiffer:** Because students mix them up. Program theory belongs to one program. Social science theory explains something across many settings, like the work of Louise Hawkley and John Cacioppo on how loneliness changes the way people read social situations. Evaluation theory is about how to evaluate, like the approaches on the tree we saw in Lesson one.

**Sarah:** There is also this idea of espoused theory and theory-in-use.

**Kiffer:** That comes from Chris Argyris and Donald Schön. Your espoused theory is the explanation you give for what you do, and your theory-in-use is what an observer would infer from what you actually do. Cedar Valley describes itself as person-directed, but an evaluator watching meetings might find connectors steering people toward the two or three groups they know well. That might work perfectly well, but it is a different theory, and an evaluation that measures only the espoused one will misread what is happening.

**Sarah:** Let's get to Chen. The reading leans heavily on Huey-Tsyh Chen's framework.

**Kiffer:** Chen divides program theory into a change model and an action model. The change model has three elements: the intervention, the determinants, and the goals and outcomes. Determinants are the factors the program tries to change because it believes they lead to the outcome. For Cedar Valley, the determinants are things like opportunities for contact, confidence about joining a group, access to transport, and the quality of the new relationships.

**Sarah:** And the action model?

**Kiffer:** The action model has six elements: the implementing organization, which is the health authority; the program implementers, meaning the connectors and coordinator; associate organizations and community partners, such as the clinics and the First Nations health centre that hosts one connector; the ecological context, including rural distances and limited transit; the delivery protocols; and the target population, adults aged sixty-five and older who score six or more on the three-item loneliness scale.

**Sarah:** Why separate them at all?

**Kiffer:** Because a program can be strong on one and weak on the other. A sound change model with a weak action model gives you a good idea poorly delivered. A strong action model with a weak change model gives you a well-run program that does not change the outcome. You also evaluate them with different evidence. The action model is mostly tested with process data, like referral counts and meeting logs. The change model is tested with data on the determinants and the outcomes.

**Sarah:** Let's talk about Carol Weiss. What was her argument?

**Kiffer:** Weiss argued that an evaluation which only compares outcomes cannot explain its own findings. She built on a distinction from Edward Suchman between two kinds of failure. Implementation failure means the program was never delivered as planned, so its theory was never really tested. Theory failure means the program was delivered as planned, but the causal process it relies on did not happen, or did not produce the outcome.

**Sarah:** And in the outcome data, both look the same.

**Kiffer:** Exactly the same. Loneliness did not fall. The responses could not be more different, though. Implementation failure calls for better delivery: filling vacancies, training, stronger partnerships. Theory failure calls for a different program, because doing the same thing more faithfully will not help.

**Sarah:** The reading has a scenario with two clinics.

**Kiffer:** Suppose two clinics both show almost no change in loneliness at twelve weeks. In the first, a connector position was vacant for months and few people were linked to anything, which is implementation failure. In the second, most people were accompanied to a first activity and many kept attending, but they described the groups as pleasant without producing close relationships. That points to a weakness in the link between taking part and feeling connected.

**Sarah:** So the health authority would fill the vacancy in one clinic and rethink the activities in the other.

**Kiffer:** Yes. A black-box evaluation would have treated those two clinics as the same result.

**Sarah:** What about positive findings? Does theory matter when a program works?

**Kiffer:** Weiss said it matters just as much. Suppose Cedar Valley shows that loneliness fell more among participants than among comparable people in the second-wave clinics. The health authority now has to decide what to keep as the program expands. Is the transport fund worth forty thousand dollars? Could the partner grants be cut? Only data on the intermediate links can show which components carried the effect.

**Sarah:** Let me push back a little. Isn't there a risk that the evaluator who writes down the theory then goes looking for evidence that it is right?

**Kiffer:** That risk is real, and the reading names it. The response is to say in advance what evidence would count against each link, and to look seriously at rival explanations. We come back to that in contribution analysis. The other limits are that programs often hold several theories at once, that measuring every link is expensive, and that simple linear theories can misrepresent complex programs. Patricia Rogers has written well on that last point.

**Sarah:** Where does the evaluator find the theory in the first place?

**Kiffer:** From several places. Program documents come first, so for Cedar Valley that means the business case, the report from the eighteen-month pilot, and the connector training manual. Then interviews and workshops with designers, managers, frontline staff and participants. Then observation, which reveals the theory-in-use. Then research and social science theory. And there is a question of whose theory counts. The First Nations partners are co-designing a land-based pathway, and their understanding of connection, which may include family, community, land and culture, belongs in the theory of that pathway. Lesson four goes into how partners govern that work.

**Sarah:** Section two is the logic model itself. Walk me through the components.

**Kiffer:** A logic model runs from inputs to outcomes. Inputs are the resources: money, staff, partners, data systems. For Cedar Valley that includes the eight hundred and forty thousand dollar first-wave budget and the seven connector positions. Activities are what the program does with those resources, like meeting participants and co-developing a plan. Outputs are the countable products of the activities. Then come outcomes in three time frames, short-term, intermediate and long-term. Alongside the sequence sit assumptions, which are the conditions the model depends on, and external factors, which are influences the program does not control.

**Sarah:** You said there is one distinction that matters more than the rest.

**Kiffer:** The line between outputs and outcomes. An output describes what the program delivered. An outcome describes a change in someone or something. The test I use is simple. Could the program produce this result through its own effort alone? If yes, it is an output.

**Sarah:** Try it on a Cedar Valley example.

**Kiffer:** In the first six months, two hundred and forty-one older adults attended a first meeting with a connector. That is an output, because it counts how many people the program reached. Now take a participant who attends a walking group a month later. The connector can encourage that, but the person has to decide to go, the group has to be there, and the bus or the ride has to work. That is an outcome.

**Sarah:** Why does it matter so much where these go?

**Kiffer:** Because programs that put outputs in the outcome column can report success on things they control. Two hundred and forty-one people attending a meeting sounds like a result, yet nothing has changed for those people at that point. It is one of the most common ways performance reports overstate what a program has done.

**Sarah:** Tell me about the Cedar Valley logic model in the reading.

**Kiffer:** It is drawn from top to bottom so that each row has room for specific statements. The outputs include the real counts from the first six months, three hundred and twelve referrals and two hundred and forty-one first meetings. You can already compute something useful from those: about seventy-seven percent of referred people attended a first meeting. The short-term outcomes include attending a first activity, feeling more confident about joining in, and partner groups adopting welcoming practices. Lower loneliness is an intermediate outcome. Emergency department visits appear only at the very end.

**Sarah:** Why so far down?

**Kiffer:** Because in Lesson one we were careful to say that the planners expect fewer emergency visits but have not tested it. Putting that outcome at the end, with several steps before it, keeps the model honest about how far it is from what connectors actually do.

**Sarah:** Does the model have weaknesses?

**Kiffer:** It does. The arrows run between whole rows, so you cannot see that the transport fund mainly affects attendance while the partner grants affect how groups treat newcomers. And the assumptions sit in a box at the bottom without being tied to particular links. Those are exactly the gaps a theory of change fills.

**Sarah:** The section also covers three formats.

**Kiffer:** Linear, nested and outcome chain. A linear model is the familiar one, with components in sequence. It is compact and good for communication, but it hides specific pathways. A nested model puts sub-models for components or sites inside a program-level model. Cedar Valley could have sub-models for the referral and connector component, the partner grants, and eventually the land-based pathway. An outcome chain links specific outcomes to the outcomes they lead to, so every arrow is a link you can test.

**Sarah:** When would you choose which?

**Kiffer:** For most reports I would put a linear model in the main text, because funders and managers can read it quickly, and add nested sub-models for program managers. For the evaluation team, the outcome chain is the most useful, because it tells you which links to measure.

**Sarah:** Let's do common errors. What do you see most?

**Kiffer:** Activities in the outcome column, like "connectors meet participants up to six times." Outputs in the outcome column, which we just discussed. Missing assumptions. Long leaps, where an arrow runs straight from meetings to fewer emergency visits. Vague outcomes, like improved wellbeing, which nobody can measure as written. And outcomes pitched at the whole region when the program serves a few hundred people.

**Sarah:** And there is a builder in the reading.

**Kiffer:** There is an interactive builder with fifteen Cedar Valley statements. Students sort each one into a column, then check their model and read feedback on anything misplaced. The two statements people most often get wrong are the count of first meetings, which is an output, and attending a first walking-group session, which is a short-term outcome.

**Sarah:** On to Section three. How is a theory of change different from a logic model?

**Kiffer:** A logic model summarizes the sequence. A theory of change explains it. The logic model is usually built forward, from inputs toward outcomes. A theory of change is built backward, from the long-term outcome toward the conditions that must come first. And in a theory of change, assumptions are attached to specific links with the evidence for each one, instead of sitting in a box at the bottom.

**Sarah:** Where did the idea come from?

**Kiffer:** From the Aspen Institute's work on comprehensive community initiatives in the nineteen nineties, large neighbourhood efforts that could not be evaluated with trial designs. Carol Weiss argued that they should spell out how and why their activities were expected to reach their goals, and her nineteen ninety-five paper is widely credited with bringing the phrase into evaluation. James Connell and Anne Kubisch then proposed that a theory of change should be plausible, doable and testable, and Andrea Anderson wrote the practical guide that set out backward mapping.

**Sarah:** Walk me through backward mapping.

**Kiffer:** You start by agreeing on the long-term outcome. Then you ask what must be in place immediately before that outcome can occur. Each answer is a precondition. Then you ask the same question about each precondition, and you keep going until you reach conditions your program's activities can produce directly.

**Sarah:** How do you know a precondition belongs there?

**Kiffer:** You ask whether the outcome above it could happen without it. If it could, the precondition is not necessary and probably does not belong on the map. Once the map is drawn, you attach the interventions, state the assumptions, specify indicators and write a narrative. Then you read the whole thing forward, link by link, as a chain of statements joined by "so that." If any sentence sounds implausible, or you can think of an obvious missing step, you revise.

**Sarah:** Then there are assumptions and rationales. I find those two words slippery.

**Kiffer:** Many people do. In the lesson, a rationale is the reason a link should hold, such as research evidence or pilot data. An assumption is a condition that must be true for the link to work, which the program does not control or has not tested.

**Sarah:** Which assumptions matter most for Cedar Valley?

**Kiffer:** We list five in the reading. Two stand out as both important and weakly supported. The first is that participation continues after the connector's support ends at twelve weeks. The program is designed to make that likely, with accompaniment and welcoming practices, but the evidence on how long participation lasts after social prescribing is thin. The second is that new contacts address the kind of loneliness people actually feel.

**Sarah:** Why might they not?

**Kiffer:** A meta-analysis by Masi and colleagues in two thousand eleven found that loneliness interventions that addressed negative patterns in how people think about social situations reduced loneliness more than interventions that simply increased opportunities for contact. So for some participants, opportunity alone may be insufficient, and the evaluation should test this assumption directly.

**Sarah:** And the accountability ceiling?

**Kiffer:** The accountability ceiling is a line across the theory of change. Below it are outcomes the program agrees to be judged on. Above it are outcomes the program expects to contribute to but does not control. For Cedar Valley, I put the ceiling between lower loneliness and better health. Health and emergency visits depend on illness, access to care and much else, and the evidence that reducing loneliness improves health is weaker than the evidence for the earlier links.

**Sarah:** So if emergency visits do not fall in the first year, that is not a failure.

**Kiffer:** On its own it should not be read as one. The outcome stays in the theory and can be monitored, but the program is judged on what sits below the ceiling. The important thing is to agree where the ceiling goes before the data arrive.

**Sarah:** Describe the Cedar Valley theory of change.

**Kiffer:** Reading from the bottom, four interventions each produce an early precondition: people are referred, they engage with a connector and agree a plan, transport and company make attendance feasible, and groups have welcoming practices. Those converge on one middle precondition, attending a first activity the person chose. That leads to continuing to take part and forming contacts the person finds meaningful, which lead to lower loneliness and more participation, and then, above the ceiling, to better health and fewer visits.

**Sarah:** What does that drawing show that the logic model did not?

**Kiffer:** Two things. First, every pathway passes through first attendance, which makes it a natural early indicator. If people are not reaching a first activity, nothing further up can happen. Second, the weakest assumptions sit just below the ceiling, on the links from attendance to continued participation and from contact to lower loneliness. That tells the evaluation team where to spend its interview time.

**Sarah:** Now contribution analysis. Start with the problem it solves.

**Kiffer:** Lessons six to eight teach designs that estimate how much of a change a program caused, by comparing what happened with an estimate of what would have happened without it. That is attribution. Sometimes you cannot do that. Everyone eligible gets the program, or it is too early, or the outcome depends on so many other actors that a single effect estimate means little. John Mayne, who was then at the Office of the Auditor General of Canada, proposed contribution analysis for those situations in two thousand one, in the Canadian Journal of Program Evaluation.

**Sarah:** What question does it ask?

**Kiffer:** It asks whether it is reasonable to conclude that the program made an important contribution, given the evidence on its theory of change and on other factors. The product is a contribution story.

**Sarah:** What are the steps?

**Kiffer:** Mayne later described six. You set out the cause and effect question. You develop the theory of change and the risks to it, including rival explanations. You gather the existing evidence. You assemble the contribution story and assess where it is weak. You seek additional evidence on the weak points. And you revise the story.

**Sarah:** Apply it to Cedar Valley's first six months.

**Kiffer:** Of the two hundred and forty-one people who attended a first meeting, one hundred and eighty-eight had a loneliness score at both the start and twelve weeks, and their mean score fell from seven point one to six point three on a scale from three to nine.

**Sarah:** That sounds like the program worked.

**Kiffer:** It sounds that way, and this is where students need to slow down. People were referred only if they scored six or higher. When you select people because their scores are high, their scores tend to drift down on the next measurement even if nothing happens. That is regression to the mean, and Lesson seven covers it in detail. On top of that, fifty-three participants have no follow-up score, and they may differ from those who do. There could also be seasonal effects, recovery after bereavement, or other programs starting at the same time.

**Sarah:** So what does the contribution story say at this point?

**Kiffer:** It says the early links are well supported, because referrals arrived and delivery broadly followed the plan. It says the fall in loneliness is consistent with the theory but cannot yet be credited to the program. The honest judgement is plausible but unconfirmed.

**Sarah:** And the next step is to look for more evidence.

**Kiffer:** Yes, aimed at the weak points. Ask about participation at six months, interview people who kept attending and people who stopped, and check whether loneliness fell more among people who kept attending, because the theory predicts exactly that pattern. Then find a way to estimate how much decline happens without the program, for example among screened older adults in second-wave clinics, which becomes a comparison-group design in Lesson seven.

**Sarah:** When is a contribution claim actually credible?

**Kiffer:** Mayne set out four conditions: a reasoned theory of change, activities implemented as set out, evidence that the expected results occurred and the key assumptions held, and an assessment of other influencing factors. Contribution analysis gives no effect size and depends on careful judgement about rivals, so I present it as a complement to the designs later in the course.

**Sarah:** Section four is realist program theory. This felt like a shift in the reading.

**Kiffer:** It is a shift. Everything so far assumes one pathway that applies to everyone. Ray Pawson and Nick Tilley, in their nineteen ninety-seven book Realistic Evaluation, argued that the same program produces different results for different people in different settings. Their question is what works, for whom, in what circumstances, and why.

**Sarah:** How do they explain that variation?

**Kiffer:** In their account, programs do not produce outcomes directly. Programs offer resources, and outcomes depend on how people respond. The response is the mechanism, and mechanisms operate only in certain contexts. So the unit of analysis becomes the context-mechanism-outcome configuration.

**Sarah:** Give me one.

**Kiffer:** The reading uses a recently widowed woman, aged seventy-eight, with no existing link to community groups. That is the context. The connector offers to go with her to the first session of a walking group. Dalkin and colleagues, writing in two thousand fifteen, would call that the resource part of the mechanism. Her worry about walking into a room of strangers eases, and she feels expected. That is the reasoning part. She keeps attending and reports feeling less lonely at twelve weeks. That is the outcome.

**Sarah:** Why split the mechanism into resource and reasoning?

**Kiffer:** Because the most common mistake in realist work is to write the program activity as the mechanism. Accompaniment is something the program offers. The mechanism is what accompaniment changes in the person's thinking or feeling. Splitting them keeps you honest.

**Sarah:** What are the other mistakes?

**Kiffer:** Writing a context that explains nothing, such as a demographic label with no reason attached. And writing an outcome nobody could observe. The drafting device I recommend is the if-then-because statement. If the program offers this resource to people in this context, then this outcome follows, because people respond in this way. The because clause is the mechanism, and it is the part the evaluation has to test.

**Sarah:** The reading includes a configuration where the program fails.

**Kiffer:** Yes, and I think it is the most useful one. Consider older adults whose loneliness comes with low mood or long-standing social anxiety. A connector encourages them to join a group early on. They experience it as pressure, an awkward first visit confirms their fear of rejection, and they withdraw. Loneliness does not improve and might get worse.

**Sarah:** That changes what the program should do.

**Kiffer:** It would. The protocol for those participants might move more slowly, or start with mental health support. It also shows why an average can mislead. If bereaved participants improve a lot and people with low mood withdraw, a small average change might lead the health authority to abandon a program that works well for many people.

**Sarah:** How does a realist evaluation get started?

**Kiffer:** With an initial program theory, which is a set of tentative configurations. You build it from program documents, from interviews, from the research literature, including realist reviews of social prescribing by Husk and colleagues and by Tierney and colleagues, from what realists call middle-range theories, and from people with lived experience, like the four older adults on the Cedar Valley steering committee.

**Sarah:** The reading mentions realist interviews.

**Kiffer:** In a realist interview, the evaluator presents a candidate theory and asks the interviewee to confirm it, refute it or refine it. Ray Pawson described this as a teacher-learner relationship, and Ana Manzano has written practical guidance on it. Then you prioritize which configurations to test, collect mixed data, and refine the theory, reporting it against the RAMESES two standards.

**Sarah:** Does realist evaluation have weaknesses?

**Kiffer:** Several. Context and mechanism are hard to separate in practice, the approach takes time and skill, and configurations developed after you see the data can become stories that fit too easily. Stating the initial theory in advance helps with that last problem.

**Sarah:** So how should students think about logic models, theories of change and realist theory together?

**Kiffer:** As three answers to three questions. The logic model answers what the program does and what it expects. The theory of change answers how and why change should happen. Realist theory answers for whom and in what circumstances. Most good evaluation plans have the first two and add realist configurations where they expect people to respond differently.

**Sarah:** Which brings us to the worked example. What does a complete version look like?

**Kiffer:** Two pieces. The first is a one-page logic model with inputs, activities, outputs, and short-term, intermediate and long-term outcomes, along with assumptions and external factors. The second is a one-page theory of change narrative with a pathway diagram. It maps backward from the long-term outcome, places the accountability ceiling, and states its assumptions, each attached to a specific link, with the evidence for each.

**Sarah:** And the realist configurations?

**Kiffer:** You may add one or two context-mechanism-outcome configurations for a group of participants you expect to respond differently. Section four has a full worked example for Cedar Valley, and the logic model in Section two is the other half of it.

**Sarah:** Any advice for someone drafting one?

**Kiffer:** Check every outcome by asking whether your program could produce it by its own effort alone, and read your theory of change forward with "so that" between each step to listen for leaps. And be candid about your weakest assumptions, because in Lesson four those are the links your evaluation questions will be built around.

**Sarah:** That's Lesson three. Thanks, Kiffer.

**Kiffer:** Thanks, Sarah, and thanks to everyone for listening.