# Lesson 4: Interest Holder Engagement and Evaluation Questions

*Companion-podcast transcript, Sarah and Kiffer*

---

**Sarah:** Welcome back to Office Hours. I'm Sarah.

**Kiffer:** And I'm Kiffer. This is the fourth episode for Health Sciences eight twenty-six, Program Planning and Evaluation, and this week we're talking about interest holder engagement and evaluation questions.

**Sarah:** So far the course has been about describing a program: the description, the needs statement, the logic model. What changes this week?

**Kiffer:** The evaluation starts to take shape. A logic model can generate dozens of possible questions, and no evaluation can answer all of them well. So the evaluator has to decide whose questions the evaluation will answer, how those people will take part, and which questions come first.

**Sarah:** And we're still using the Cedar Valley program.

**Kiffer:** We are. The Cedar Valley Connector program is fictional. It's a social prescribing program run by a fictional health authority in British Columbia. Clinicians refer lonely older adults to a community connector, who meets them up to six times over twelve weeks and links them to groups, volunteer roles and transportation help. It started in twelve of the region's twenty-four primary care clinics, and the other twelve start a year later. The health authority's executive has to decide whether to extend the program unchanged, modify it, or stop it. That decision is the main reason the evaluation exists, and it shapes almost everything we talk about today.

**Sarah:** Let's start with vocabulary. Why does the course say interest holder?

**Kiffer:** In Lesson one we saw that the twenty twenty-four evaluation framework from the United States Centers for Disease Control and Prevention adopted the term, and this course follows it. Many of the classic frameworks we discuss this week were published under the older term, so I describe them in the current vocabulary.

**Sarah:** So who counts as an interest holder?

**Kiffer:** Anyone with an interest in the program or its evaluation: people who make decisions about it, deliver it, take part in it or are affected by it, and people who will use the findings. The definition is broad on purpose. If you build your list from the organizational chart, you get the funders and managers and miss almost everyone else.

**Sarah:** Who in particular gets missed?

**Kiffer:** Guba and Lincoln, writing in nineteen eighty-nine, sorted interest holders into three classes. Agents develop, fund, deliver and use the program. Beneficiaries gain from it. Their third class, which they called victims, are people disadvantaged by the program because they're excluded, bear its costs, or lose an opportunity.

**Sarah:** Victims is a strong word for a loneliness program.

**Kiffer:** It is, and it's their term, so I use it carefully. At Cedar Valley it could include lonely older adults with no regular family doctor or nurse practitioner, who are never screened or referred. It could include people in the second-wave clinics who wait a year, and small volunteer groups that receive more referrals than they can handle.

**Sarah:** The lesson also talks about rights holders.

**Kiffer:** In Canada, First Nations, Inuit and Métis governments and organizations are often described as rights holders. The term recognizes the inherent right of self-determination and the rights affirmed in section thirty-five of the Constitution Act. When a First Nation partners with a health authority, it takes part as a government, and its role in the evaluation is set by agreement. Treating it as one more interest group, consulted like a seniors' club, misreads the relationship.

**Sarah:** Once you have a broad list, you can't give everyone an equal say in every decision. How do you choose?

**Kiffer:** That's where Michael Quinn Patton's idea of primary intended users comes in. They're specific, named people who have a stake in the findings and the capacity to act on them. Patton studied how federal health evaluations in the United States were used, and he found that use depended most on what he called the personal factor, an identifiable person or small group who really cares about the evaluation and its findings. Everyone else is an audience, receiving the findings in a form suited to them. At Cedar Valley the team named four primary intended users: the director of primary care, who advises the executive; the program coordinator; an older adult with lived experience of loneliness; and a First Nations representative appointed by the First Nations partners.

**Sarah:** I want to push on that. Isn't it a bit exclusive to choose four people and call everyone else an audience?

**Kiffer:** It's a fair concern. Being a primary intended user is only one way of taking part, and the connectors, clinicians and community groups all have roles, which we'll get to. Notice too that two of the four represent groups with less formal power. Patton also advises having several users, so the evaluation survives when someone changes jobs.

**Sarah:** Let's talk about mapping. The first tool is the power-interest grid.

**Kiffer:** Eden and Ackermann described it, and Bryson adapted it for public and nonprofit settings. You place each group by its power over the program and its interest in it. Players have both. Context setters have power but little current interest. Subjects have high interest and little power. And the crowd has little of either.

**Sarah:** Where do the older adults end up?

**Kiffer:** They're subjects, with the highest stake of anyone and almost no formal power. Here's the risk with the grid. Read as a management tool, it tells you to engage the powerful closely and keep the subjects informed. An evaluator can decide instead to strengthen the voice of the subjects, and the ethics of evaluation often call for that. At Cedar Valley two older adults sit on the evaluation working group, and they're paid for their time.

**Sarah:** And the second tool is the salience model.

**Kiffer:** From Mitchell, Agle and Wood in nineteen ninety-seven. It explains how much attention managers give a group using three attributes. Power is the ability to impose your will. Legitimacy is a socially accepted claim to have a say. Urgency is how time-sensitive or important the claim is. Combining them gives seven types.

**Sarah:** Give me some Cedar Valley examples.

**Kiffer:** The executive is dominant, with power and legitimacy. Referred older adults are dependent, with a legitimate and urgent claim but no power to act on it. For decisions about their members' data and the land-based pathway, the First Nations partners are definitive, because they hold all three. The model describes how attention tends to be distributed. The evaluator still has to decide how it should be distributed.

**Sarah:** The section ends by looking at how roles differ.

**Kiffer:** Each group wants different things and faces different risks. Funders want timely evidence about effectiveness and cost, and the risk is pressure for favourable findings, so the Cedar Valley plan records an agreement that the evaluators will report what they find. Staff may feel what Donaldson, Gooler and Scriven called evaluation anxiety. Participants risk burden and tokenism. And community groups can find that an evaluation adds to their work while giving nothing back.

**Sarah:** Let's move to the second section, on engagement. Why engage interest holders at all? It's slower and more expensive.

**Kiffer:** Both true. Evaluators give four reasons. The first is use, which follows from the personal factor. The second is quality, because people who deliver or receive a program know which survey items will confuse people and which explanations fit local conditions. The third is ethics and rights. And the fourth is capacity, the skills people keep afterward.

**Sarah:** And the costs?

**Kiffer:** Time, money, delay, raised expectations, and a risk to the perceived independence of the findings. A review by Brandon and Fukunaga also found that most studies of involvement in evaluation were descriptive, with limited evidence on its effects. So state why you're engaging each group and check whether it's working.

**Sarah:** The lesson brings in Sherry Arnstein's ladder from nineteen sixty-nine. Why go back that far?

**Kiffer:** Because her central point still holds. Her ladder runs from nonparticipation, through degrees of tokenism, up to degrees of citizen power. She argued that participation without a redistribution of power is an empty ritual. That's a good test for any evaluation plan. If people's views can't change any decision, say so, and don't call the arrangement a partnership.

**Sarah:** And the modern scale is the spectrum from the International Association for Public Participation.

**Kiffer:** It has five levels, each with a promise. Inform promises to keep people informed. Consult promises to listen and report back on how input was used. Involve promises to work with people so their concerns are reflected. Collaborate promises to seek their advice and include it to the greatest extent possible. Empower promises to implement what they decide.

**Sarah:** How does Cedar Valley use it?

**Kiffer:** Separately for each group and each decision. Clinicians in all twenty-four clinics are consulted through a short survey on referral. Connectors are involved in reviewing measures and in sense-making sessions on the findings. Older adult partners collaborate with the director and coordinator on the evaluation questions. The board and the public are informed.

**Sarah:** And empower?

**Kiffer:** The First Nation decides how the land-based pathway is evaluated and how data about its members are used. But the spectrum was written for public participation in decisions an organization owns. The Nation's authority comes from self-determination, so I'd describe this as a decision that belongs to the Nation, which the health authority supports. And whatever level you choose, keep the promise. Offering more influence than decision-makers will honour damages trust far more than an honest offer at a lower level.

**Sarah:** The section then compares collaborative, participatory and empowerment evaluation. I always find these blur together.

**Kiffer:** Fetterman, Rodríguez-Campos and Zukoski separate them by control. In collaborative evaluation, the evaluator stays in charge and works in partnership with interest holders. In participatory evaluation, control is shared. In empowerment evaluation, program staff and community members run the evaluation themselves, with the evaluator as a coach or critical friend. Within participatory evaluation, Cousins and Whitmore distinguished two streams in nineteen ninety-eight. Practical participatory evaluation supports program decisions and use. Transformative participatory evaluation aims at empowerment and social change for people who have been marginalized.

**Sarah:** Does Cedar Valley pick one?

**Kiffer:** It uses each in a different strand. The outcome and impact strand is collaborative, because the second-wave decision needs findings the executive will regard as independent. The process strand is practical participatory, designed with older adults, connectors and the coordinator. The connectors run an empowerment-style self-assessment each quarter. And the land-based pathway is evaluated developmentally under the Nation's leadership.

**Sarah:** Let's talk about patient partners.

**Kiffer:** Canada's Strategy for Patient-Oriented Research, known as SPOR, was launched by the Canadian Institutes of Health Research in twenty eleven. Its Patient Engagement Framework from twenty fourteen has four guiding principles: inclusiveness, support, mutual respect and co-build. It was written for research, and evaluation teams in Canadian health authorities often adopt it voluntarily, as the Cedar Valley working group has done.

**Sarah:** Which principle matters most in practice?

**Kiffer:** I'd stress co-build, which means partners join from the beginning. The older adults were on the working group before any questions were drafted. I'd also stress having at least two partners from any constituency, so nobody has to speak alone, and scheduling meetings around people's health, caregiving and transportation.

**Sarah:** The lesson works through a compensation budget.

**Kiffer:** Using an illustrative rate of twenty-five dollars an hour. Two older adult partners attend ten working group meetings, each counted as three hours, two in the meeting and one preparing. That's one thousand five hundred dollars. All four older adults on the steering committee attend four sense-making sessions of three hours, which is one thousand two hundred dollars. Expenses at fifteen dollars per person per meeting add five hundred and forty dollars, for a total of three thousand two hundred and forty dollars.

**Sarah:** Why pay for preparation?

**Kiffer:** Because reading papers and thinking about them is work. The other principles are to reimburse expenses promptly, offer a choice of payment method and the option to decline, and tell partners that payments can affect income-tested benefits. Honoraria for Elders and knowledge keepers follow the protocols of the Nation involved.

**Sarah:** The last part of the section is about power and conflict.

**Kiffer:** When you bring people together, their differences in power come with them. Power comes from authority, control of money and data, professional vocabulary, fluency in English, health, and history, including the colonial history between health systems and Indigenous Peoples. If you ignore that, you get the tokenism Arnstein described. Everyone attends, the powerful decide, and the report says all voices were heard.

**Sarah:** So what do you do about it?

**Kiffer:** House and Howe proposed deliberative democratic evaluation, built on inclusion, dialogue and deliberation. In practice, you state decision rules before any decision comes up, let partners with less power prepare beforehand, use round-robin turns and anonymous ranking, and record any disagreement that remains in the report.

**Sarah:** The lesson has a scene at the steering committee.

**Kiffer:** The director says the executive will only be persuaded by fewer emergency department visits. The older adults say what matters is whether people form relationships that last. The connectors ask whether caseload data will be used to rank them. And the First Nations representatives say outcomes for their members should include connection to family, community, culture and land, defined by the Nation.

**Sarah:** Four different conflicts.

**Kiffer:** And the evaluator treats them differently. The first two are differences in values about what counts as success, so the evaluator proposes prioritizing questions against agreed criteria. The connectors' concern is a conflict of interests, settled by a written statement that process data will be reported by clinic group and used for improvement. The First Nations request is about authority, so those outcomes will be defined through the partnership's own process.

**Sarah:** On to the third section, equity and Indigenous evaluation. What does equity-focused evaluation add?

**Kiffer:** Whitehead defined health inequities as differences that are unnecessary, avoidable, unfair and unjust. Equity-focused evaluation, as Bamberger and Segone describe it, asks who a program reaches and who it misses, whether benefits are distributed fairly, whether it narrows or widens gaps, and whether the evaluation's own methods exclude anyone.

**Sarah:** Can a program that helps people on average widen gaps?

**Kiffer:** It can. Lorenc and colleagues found that interventions relying on people choosing to take part, and having the resources to do so, can widen inequalities even while improving averages. Cedar Valley depends on a clinician referral and on travel, so older adults without a regular clinician or without transportation may benefit less.

**Sarah:** The lesson has numbers on transportation.

**Kiffer:** Illustrative ones. Of the three hundred and twelve older adults referred in the first six months, one hundred and ninety-eight had their own transportation or a regular ride, and one hundred and sixty-eight of them attended a first meeting, about eighty-five percent. Of the one hundred and fourteen without regular transportation, seventy-three attended, about sixty-four percent. The gap is about twenty-one percentage points.

**Sarah:** Does that mean transportation caused the gap?

**Kiffer:** It doesn't prove it. The comparison is descriptive, and people without transportation may also be frailer or more isolated. But it raises a clear question about whether the forty thousand dollar transport fund reaches the people who need it, and it shows why transportation status should be recorded for every referral.

**Sarah:** How do you decide which groups to compare?

**Kiffer:** PROGRESS-Plus is a checklist of social factors that stratify health, such as place of residence, language, gender, education and socioeconomic status, with a Plus that adds age, disability and sexual orientation. Gender-based Analysis Plus is the Government of Canada's process for asking how diverse groups of women, men and gender diverse people experience a program, and its Plus extends the analysis beyond sex and gender.

**Sarah:** And intersectionality sits underneath both.

**Kiffer:** Kimberlé Crenshaw introduced the term in nineteen eighty-nine to describe how race and gender together shaped discrimination that neither category alone could explain. Olena Hankivsky and colleagues at Simon Fraser University developed an intersectionality-based policy analysis framework that adapts well to evaluation. At Cedar Valley, the practical result is that gender, language and transportation status are added to the referral form.

**Sarah:** Let's turn to Indigenous evaluation. How does the lesson frame it?

**Kiffer:** It starts with context: the Calls to Action of the Truth and Reconciliation Commission, and in British Columbia the In Plain Sight report of twenty twenty, which documented widespread Indigenous-specific racism in health care. Then it stresses that Canadian practice is distinctions-based. First Nations, Inuit and Métis Peoples are distinct, and there are more than two hundred First Nations in British Columbia, each with its own governance and protocols. The principles in the lesson come from published Indigenous scholarship and national First Nations organizations, and they're starting points. The protocols for any particular evaluation are learned from, and set by, the Nation involved, and one Nation's practices are never general rules.

**Sarah:** The first principle is relational accountability.

**Kiffer:** That comes from Shawn Wilson, an Opaskwayak Cree scholar, in his book Research Is Ceremony. Wilson describes knowledge as relational, existing in relationships with people, with the land, with ideas and with the cosmos. So the evaluator is accountable to all of those relationships, through respect, reciprocity and responsibility.

**Sarah:** What does that change for an evaluator?

**Kiffer:** The order and pace of the work. Relationships come before data, and that takes time the plan has to allow for. Obligations continue after the report, including sharing findings in usable forms. At Cedar Valley the team meets the First Nations partners several times before drafting any questions, and the budget includes return visits to share results.

**Sarah:** The second principle is Two-Eyed Seeing.

**Kiffer:** Two-Eyed Seeing, or Etuaptmumk in Mi'kmaw, is a guiding principle brought forward by Mi'kmaw Elder Albert Marshall, who developed it with Murdena Marshall and Cheryl Bartlett at Cape Breton University. As Elder Marshall describes it, it means learning to see from one eye with the strengths of Indigenous knowledges and ways of knowing, and from the other eye with the strengths of Western knowledges and ways of knowing, and learning to use both eyes together for the benefit of all.

**Sarah:** How would that show up at Cedar Valley?

**Kiffer:** In the land-based pathway, Western methods might contribute loneliness scores and participation records, while the Nation's own understanding of wellness and connection, carried in stories, teachings and the guidance of Elders, defines what success looks like. Writers on Two-Eyed Seeing caution against using the phrase as a label for adding some Indigenous content to a design that stays entirely Western. Practising it means co-learning, with Indigenous knowledge holders leading the Indigenous side.

**Sarah:** And the third principle is OCAP.

**Kiffer:** OCAP stands for ownership, control, access and possession, and it's a registered trademark of the First Nations Information Governance Centre, which stewards the principles and offers training on them. Ownership means a community owns information about it collectively. Control means First Nations have the right to seek control over research and information management that affect them. Access means they can access their data wherever it's held. Possession means physical control of the data, which protects ownership.

**Sarah:** Does OCAP apply to all Indigenous data?

**Kiffer:** It applies to First Nations. Inuit and Métis organizations have their own approaches, such as the National Inuit Strategy on Research from Inuit Tapiriit Kanatami. At Cedar Valley, the partners and the health authority agree in writing, before data collection, where data about the Nation's members will be stored, who may analyze them, and how findings are reviewed before release. In Lesson one we saw that much program evaluation falls outside research ethics board review under the Tri-Council Policy Statement, and these obligations apply all the same. Many Nations also have their own review processes. Health authority data, like emergency department visits, also become data about the Nation's members once they're linked to identify First Nations clients.

**Sarah:** The section ends with culturally responsive evaluation.

**Kiffer:** Hood, Hopson and Kirkhart describe it as placing the culture and context of the community at the centre of every stage. Karen Kirkhart argued that culture is a matter of validity: if participants don't recognize your measures, your conclusions may be wrong. At Cedar Valley, some older adults speak Punjabi, Cantonese or Tagalog at home, so the team tests translated survey items in each language, offers interviews in people's preferred languages, and asks community members to help interpret the findings.

**Sarah:** That brings us to the fourth section. What is an evaluation question, as distinct from a research question?

**Kiffer:** Evaluation questions are the small number of high-level questions the evaluation as a whole will answer. A research question seeks knowledge that generalizes beyond the setting. Jane Davidson recommends a few big-picture questions that are explicitly evaluative, asking how good or how well, so that the evaluation ends in a judgement. Morra Imas and Rist describe three types. Descriptive questions ask what is happening. Normative questions compare it with a target or standard. Cause-and-effect questions ask whether the program caused a change, which needs the kind of comparison we develop in Lessons six to eight.

**Sarah:** Give me a weak question and a better one.

**Kiffer:** Does the Connector program work? That's weak. A better version asks to what extent the program reduces loneliness at twelve weeks among referred older adults, compared with similar older adults in second-wave clinics over the same period. It names the outcome, the population, the time point and the comparison. Another weak one is, prove that the program saves money, which presumes its answer.

**Sarah:** How do you actually draft questions with users?

**Kiffer:** Lee Cronbach proposed two phases. In the divergent phase, you gather candidate questions widely, from interest holders, the logic model, the literature and your own judgement. In the convergent phase, you and the users select the few the evaluation will answer. The Cedar Valley working group generated thirty-one candidates and clustered them into seven.

**Sarah:** And then there's a test with invented results.

**Kiffer:** That's Patton's simulation of findings. You show the users a favourable and an unfavourable invented result and ask what they'd do in each case. If both lead to the same action, or to no action, the question is a candidate for removal or rewording.

**Sarah:** Did any Cedar Valley question fail?

**Kiffer:** The emergency department question. Whatever result we showed, the director's answer was the same, because no answer could arrive before the second-wave decision. It needs a year of follow-up and a data linkage agreement. The question still matters, and its timing was the problem, so it moved to a second phase.

**Sarah:** Then the group scored the seven candidates.

**Kiffer:** Against four criteria agreed before anyone scored. Decision relevance counts twice, because informing the second-wave decision is the main purpose. Then importance to interest holders, answerability, and equity relevance, each scored from one to three. The reach question scored the maximum of fifteen, and three questions tied at twelve.

**Sarah:** How do you break a tie?

**Kiffer:** With judgement. The scores structure the conversation, and the users decide. The group merged the change among participants into the comparison question, because it's one piece of estimating the effect. Fidelity and community capacity became sub-questions of the reach question.

**Sarah:** Any cautions about scoring?

**Kiffer:** Weights are value judgements, so agree them in advance and report them. And a group dominated by one interest can give its own priorities an appearance of objectivity. The First Nations partners' question was handled differently. They proposed and worded it themselves, and it was included by agreement without being scored against the health authority's criteria.

**Sarah:** Next comes evaluability assessment.

**Kiffer:** Joseph Wholey developed it at the Urban Institute in the nineteen seventies, after seeing many federal evaluations produce findings nobody used. A program is ready for a useful evaluation when its goals and information needs are agreed, its objectives are plausible, relevant data can be obtained at reasonable cost, and the users agree on how they'll use the findings.

**Sarah:** What did it find at Cedar Valley?

**Kiffer:** The process questions were ready, and so was change in loneliness among participants, with attention to missing follow-ups. The big finding concerned impact. No comparison data had been collected, so that question could be answered only if the second-wave clinics started screening and measuring loneliness before they began the program. The team arranged that immediately. Without the assessment, they'd have discovered a year later that they had no baseline.

**Sarah:** And the last piece is the evaluation matrix.

**Kiffer:** It's the main planning table. Each row is an evaluation question or sub-question, and the columns give the indicators, the data sources, the methods, the timing and who is responsible. A completed matrix shows whether every question has evidence and whether you're collecting data no question needs. Lesson five fills it in.

**Sarah:** So where did Cedar Valley end up?

**Kiffer:** With four questions. First, how well is the program reaching and serving the older adults it was designed for, and which groups is it missing? Second, to what extent does it reduce loneliness compared with similar older adults in second-wave clinics, and do effects differ by transportation, language and gender? Third, what does it cost per older adult served? And fourth, the partners' question, about how the program supports connection to family, community, culture and land for First Nations participants, as the Nation defines these.

**Sarah:** Let's finish with how all of that gets written down. What does a complete version look like?

**Kiffer:** Two things. An interest holder map, as a table listing each group, its interest, its influence or status, the proposed engagement level, and its role, with the primary intended users identified. And a short set of prioritized evaluation questions, each labelled by type and linked to a user and a decision, followed by a justification.

**Sarah:** What should the justification cover?

**Kiffer:** Your prioritization criteria, any questions you deferred and why, how equity enters your questions, and any evaluability concerns. If your program involves First Nations, Inuit or Métis communities, describe the partnership in terms of rights and governance.

**Sarah:** Any advice for an evaluator who is stuck?

**Kiffer:** Start with the decision. Ask who has to decide something about your program in the next year or two, and what they need to know to decide well. Then ask who is affected by that decision and has the least say in it. If your map and your questions answer both, you're in good shape.

**Sarah:** Thanks, Kiffer. Next time, indicators, process evaluation and mixed methods.

**Kiffer:** Thanks, Sarah. See you then.