# Lesson 7: Sampling, Recruitment and Measurement

*Companion-podcast transcript, Sarah and Kiffer*

---

**Sarah:** Welcome back to Office Hours. I'm Sarah.

**Kiffer:** And I'm Kiffer. This week we are on Lesson seven of Health Sciences two oh seven, which covers sampling, recruitment and measurement.

**Sarah:** That sounds like three topics squeezed into one lesson. How do they fit together?

**Kiffer:** They are two decisions that every quantitative study makes before it collects any data. The first is who will supply the data, which is sampling and recruitment. The second is what exactly will be recorded about those people, which is measurement. If either decision is made carelessly, the results describe the wrong people or the wrong thing, and clever analysis later cannot repair that.

**Sarah:** And we are still following the Cedar Valley study?

**Kiffer:** We are. The Cedar Valley Social Connection Study is our fictional running case. A team led by Dr. Maya Hart is studying loneliness and social isolation among adults aged sixty-five and older, working with the fictional Cedar Valley Health Authority. The region has about forty-six thousand older adults.

**Sarah:** Section one is about populations. Why do we need more than one?

**Kiffer:** Because three groups of people are involved in any study, and they are rarely the same. The target population is the group you want your findings to apply to, and it comes from the research question. For Cedar Valley, that is all adults aged sixty-five and older in the region, roughly forty-six thousand people.

**Sarah:** And the second?

**Kiffer:** The source population is the part of the target population the study can actually reach and select from, which in practice means the people on the list the team uses. Cedar Valley used the patient lists of the region's twenty-four primary care clinics and the client list of the Cedar Valley First Nations Health Centre. After duplicates were removed, those lists held forty thousand older adults.

**Sarah:** So about six thousand older adults had no chance of being selected.

**Kiffer:** Exactly. Most of them have no regular clinic in the region. Then the study sample, the participants, is the people who were selected, were eligible, agreed to take part and provided data. In Cedar Valley, that is the one thousand six hundred people who completed the survey.

**Sarah:** I have seen other textbooks use different words for these.

**Kiffer:** Some say accessible population or sampled population where we say source population, and some later courses call it the study population. My advice is to define your terms in a sentence each whenever you write a protocol, and then use them consistently.

**Sarah:** The lesson describes two steps from the sample back to the target population. What are they?

**Kiffer:** The first step goes from the sample to the source population. If the sample was drawn at random from the list, statistical methods can describe how closely it is likely to reflect everyone on the list, and you will learn those in Health Sciences three forty-one. The second step goes from the people on the list to everyone the study is about. No statistical method can take that step. It is a judgement about whether the people missing from the list differ from the people on it.

**Sarah:** And in a loneliness study, I would guess they do.

**Kiffer:** That is the worry. Older adults without a regular clinic may well be more isolated, so the team states the gap plainly as a limitation and compares its respondents with census figures for the region.

**Sarah:** Let's talk about eligibility criteria. What makes a good one?

**Kiffer:** A good criterion follows from the research question, is precise enough that two research assistants would make the same decision about the same person, can be checked, and comes with a reason. Cedar Valley included people who were sixty-five or older on the day the invitation was mailed, lived in the region, lived in a private home, and could give consent. Long-term care residents were left out because the team plans to study that setting separately.

**Sarah:** Are there exclusions the team deliberately avoided?

**Kiffer:** Yes. The Tri-Council Policy Statement, which you studied in Lesson five, asks researchers not to exclude people because of age, language or disability without a valid reason. The advisory group of six older adults pointed out that many lonely older adults have low vision or limited English. So the team added a paper version and a telephone help line, with interpreters in the two other languages most often spoken by older adults in the region.

**Sarah:** Now the sampling frame. That term comes up constantly in this lesson.

**Kiffer:** Because the frame is where the source population comes from. A sampling frame is the list, map or procedure that identifies the people you could select. Everyone on a list has some chance of selection, and nobody missing from it has any chance at all.

**Sarah:** And frames have problems.

**Kiffer:** Five common ones. Undercoverage is when people in the target population are missing. Overcoverage is when the list includes people who have died, moved or entered long-term care. Duplicates double some people's chance of selection. Clustering is when one entry stands for several people, like a household address. And lists go out of date. Undercoverage is the most serious, because you can screen out, de-duplicate and choose one person per household, but you cannot select people who are not on the list.

**Sarah:** How did the Cedar Valley team choose its frame?

**Kiffer:** It compared four candidates. The home care client list covered only a much frailer group. A residential address list had no ages on it. Seniors' centre membership lists missed exactly the isolated people the study cares about most. The clinic and Health Centre lists covered about eighty-seven percent of older adults and included age and postal code, so the team chose those.

**Sarah:** Can clinics just hand patient names over to researchers?

**Kiffer:** They cannot, and that shaped the recruitment plan. Each clinic mailed the invitations itself, using study identification numbers, so the researchers learned a name only when a person chose to respond. The Cedar Valley First Nations Health Centre decided how its own clients would be approached. Under the engagement agreement from Lesson four, it mailed the invitations with a letter from its health director and kept a say in how results about its community would be reported.

**Sarah:** Before we leave Section one, you said loneliness studies have a particular problem.

**Kiffer:** The most isolated people are the least likely to be on lists, to have a regular clinic, or to answer a letter from a stranger. A study that loses them will make a region look more connected than it is. The team wrote that gap into its protocol and planned to reach some isolated residents for interviews through community connectors and outreach workers.

**Sarah:** Section two is about how people are chosen from the frame. What is the main distinction?

**Kiffer:** Probability versus non-probability samples. In a probability sample, every person on the frame has a known chance of selection greater than zero, and a chance mechanism, such as a random number generator, decides who is chosen. In a non-probability sample, people get in through the researcher's judgement, by volunteering, or by being available, so nobody can say what chance any person had.

**Sarah:** And there is a test for telling them apart.

**Kiffer:** Ask whether, before selection began, you could have written down the probability that a named person on the frame would be chosen. In the Cedar Valley survey you could. Each person on the Cedar City part of the frame had a one in ten chance, and each person elsewhere had a three in twenty chance.

**Sarah:** Why different chances?

**Kiffer:** That is stratified sampling. The team split the frame into Cedar City, with fifteen thousand older adults, and the other communities, with twenty-five thousand, and sampled the smaller communities at a higher rate so it could describe them separately. That gave one thousand five hundred people from Cedar City and three thousand seven hundred and fifty from elsewhere, five thousand two hundred and fifty in all. The analysis uses survey weights to restore the proportions, which you will meet in Health Sciences three forty-one.

**Sarah:** What are the other probability methods?

**Kiffer:** Simple random sampling gives everyone on the frame the same chance. Systematic sampling takes every so many entries after a random start. At one partner clinic the chart review needed fifty charts from one thousand four hundred and fifty eligible patients, so it took every twenty-ninth chart, starting from a random number. Cluster sampling selects groups such as clinics or schools at random and then takes people within them.

**Sarah:** So is the chart review a cluster sample?

**Kiffer:** Only partly. The six partner clinics were chosen because they agreed to take part, which is a non-probability first stage. Only the selection of charts within each clinic is random, and the protocol should say exactly that.

**Sarah:** Are non-probability samples always a weakness?

**Kiffer:** They are often the right choice. A convenience sample takes whoever is easy to reach. Purposive sampling picks people deliberately for their experiences. Quota sampling fills set numbers per group with whoever is available, and snowball sampling asks participants to refer others. The twenty-four Cedar Valley interviews use purposive, maximum variation sampling, choosing older adults who live alone across different communities, genders and experiences of moving.

**Sarah:** So where is the limitation?

**Kiffer:** It is specific. A non-probability sample cannot, by itself, support an estimate of how common something is in a population. To say what percentage of older adults are lonely, you need a probability sample.

**Sarah:** Which brings us to the social media link. I can see why people get this one wrong.

**Kiffer:** So can I. In a survey of students entering Health Sciences three forty-one, about one in four classed a survey link posted on social media as simple random sampling. In our story, the graduate research assistant makes the same argument: post the link on the health authority's pages, collect thousands of responses, and since anyone could see it, the respondents are a random sample.

**Sarah:** Let me push on that. Anyone really could see it, and nobody is being picked by the researcher. Isn't that random?

**Kiffer:** It is random in the everyday sense of haphazard. In research, random means that chance, applied to a defined frame, does the choosing with known probabilities. To reach the survey, an older adult has to use that platform, follow the pages, be shown the post by the algorithm, decide to click and decide to finish. Nobody knows any of those chances, and they depend on age, health, digital skills and social networks.

**Sarah:** Which are closely tied to loneliness.

**Kiffer:** That is the heart of it. A loneliness survey that spreads by friends sharing it reaches people who have friends to share it with. The result is a convenience sample of volunteers.

**Sarah:** What if it gets fifty thousand responses?

**Kiffer:** Then it is a very large convenience sample, because size and selection method are separate properties. In nineteen thirty-six the Literary Digest mailed about ten million ballots to people drawn from telephone directories, car registrations and its subscriber lists, got more than two million back, and predicted that Alf Landon would beat Franklin Roosevelt. Roosevelt won by a wide margin. A larger self-selected sample gives a more precise estimate of the wrong quantity.

**Sarah:** Should students avoid social media recruitment altogether, then?

**Kiffer:** It has legitimate uses, such as recruiting for a pilot study, reaching a group with no list, or finding interviewees for a purposive sample. The requirement is honest labelling. Name the platforms, call it a convenience sample, and present its percentages as describing the respondents.

**Sarah:** The second confusion is random sampling versus random allocation.

**Kiffer:** In the same student survey, thirty-five percent said the purpose of randomization in a trial is to make the sample represent the population. That describes random sampling. Random sampling decides who enters the study, and it supports generalizing to the source population. Random allocation takes people already in the study and uses chance to decide which group each one joins. It makes the groups comparable at the start, so that a difference in outcomes can be credited to the intervention.

**Sarah:** So a study can have one without the other.

**Kiffer:** Either, both or neither. Most randomized controlled trials enrol volunteers and then allocate them by chance, which is random allocation within a convenience sample. The Cedar Valley survey is the reverse: random sampling, with nobody allocated to anything.

**Sarah:** Give me a Cedar Valley trial.

**Kiffer:** Suppose the team later pilots a telephone befriending service. Eighty older adults who scored high on loneliness agree to take part, and a computer allocates forty to weekly calls and forty to usual care. The groups should be similar at the start, so a difference after six months can be credited to the calls. Because the eighty were volunteers, the team should be careful about how far the result applies to lonely older adults in general.

**Sarah:** Let's get practical. What does recruitment involve?

**Kiffer:** Recruitment is everything you do to contact the selected people, explain the study and invite them in. Three principles matter. First, in a probability sample you recruit the people who were selected. Replacing a non-responder with a neighbour who is easier to reach turns your sample back into a convenience sample.

**Sarah:** That seems an easy mistake to make under deadline pressure.

**Kiffer:** Which is why it belongs in the protocol. Second, several contacts by more than one mode raise response. Following Don Dillman and his colleagues, the Cedar Valley team sent a pre-notice letter, then an invitation with a web link, then a reminder postcard, then a paper questionnaire to non-responders, and finally a call or letter. Third, every letter and script is part of the ethics application, and it should avoid undue influence. The Cedar Valley letters said that a decision not to take part would not affect anyone's care.

**Sarah:** What happens when someone opens the survey?

**Kiffer:** They answer screening questions first: age in years, whether they live in the region, and whether they live in a private home or in long-term care. Anyone ineligible sees a thank-you message, and the reason is recorded. Screening collects only what is needed, because the person has not yet consented to the main study.

**Sarah:** And the recruitment log?

**Kiffer:** The log records every contact with every selected person and its outcome, by study number only. It sounds dull, and it supplies the most important figure in your methods section, the participant flow diagram.

**Sarah:** Walk me through the Cedar Valley diagram.

**Kiffer:** Five thousand two hundred and fifty were selected. Two hundred and ten invitations were not delivered, leaving five thousand and forty. Three thousand two hundred and forty did not respond, so one thousand eight hundred were screened. Ninety-five were ineligible, forty-five declined consent and sixty stopped before finishing, which leaves one thousand six hundred completed surveys. Each number should equal the one above it minus the loss beside it.

**Sarah:** And the response rate comes from that.

**Kiffer:** The simple version divides completed surveys by invitations delivered. One thousand six hundred divided by five thousand and forty is about thirty-one point seven percent. It tells you how many took part. It cannot tell you whether responders differ from non-responders, which is the question of non-response bias in Health Sciences three forty-one.

**Sarah:** Let's turn to measurement, which is Section three. Where does it start?

**Kiffer:** With a ladder. Measurement means assigning numbers or categories to people by explicit rules, and every column in a dataset sits at the bottom of five rungs. At the top is the concept, such as loneliness. Next is the conceptual definition, which says in words what it means. Then the operational definition, the exact procedure for recording it. That produces a variable, and each person gets a value.

**Sarah:** What is the conceptual definition of loneliness?

**Kiffer:** Many researchers follow Daniel Perlman and Letitia Anne Peplau, who described loneliness as the unpleasant experience that arises when a person's relationships fall short, in quantity or quality, of what the person wants. It is subjective, it is unpleasant, and it rests on a comparison between the relationships you have and the ones you want.

**Sarah:** And that is different from social isolation.

**Kiffer:** Very different. Social isolation is an objective lack of relationships, roles or contact. Mr. Okafor lives with his daughter's family and sees his grandchildren every day, yet since his wife died he feels nobody understands him. He is not isolated, and he may well be lonely. Mrs. Lindqvist lives alone on a farm outside Kestrel Lake and says she enjoys her solitude. She is isolated by most measures, and she may not be lonely.

**Sarah:** So each concept needs its own measure.

**Kiffer:** Yes. Cedar Valley's question names loneliness as the exposure, so it needs a loneliness measure, and it uses the six-item Lubben Social Network Scale to describe isolation separately.

**Sarah:** What goes into an operational definition?

**Kiffer:** Five elements: the source and method, the instrument or exact wording, the time frame, the scoring with its range and any cut-off, and the rule for missing answers. Cedar Valley uses the three-item University of California, Los Angeles Loneliness Scale, usually called the UCLA scale. It asks how often you lack companionship, feel left out and feel isolated. Each item scores one to three, the total runs from three to nine, a score of six or more counts as lonely, and the score is calculated only when all three items are answered.

**Sarah:** The lesson says operational definitions always narrow the concept.

**Kiffer:** Three questions capture part of what Perlman and Peplau described. That is acceptable if you say what you measured. It also means that different definitions of the same concept can disagree. Emergency department use could be self-reported, which depends on memory, or counted from linked administrative records, which only capture facilities whose data reach the data holder. Cedar Valley counts records in the twelve months after the survey, and the protocol explains why.

**Sarah:** Now levels of measurement. I remember the names, but not much else.

**Kiffer:** They come from the psychologist S. S. Stevens in the nineteen forties. Nominal variables are unordered categories, like community. Ordinal variables have an order with unknown distances, like self-rated health from poor to excellent. Interval variables have equal distances and an arbitrary zero, like year of birth. Ratio variables also have a true zero, like age in years or the number of emergency department visits.

**Sarah:** Why does a second-year student need to care?

**Kiffer:** Because the level decides which summaries make sense. The line that matters most in this course is between categorical variables, which get frequency tables, and numeric variables, which get means and standard deviations. That is the choice you will make when you build a Table One in Lesson eleven. And watch for digits that are really names. A clinic code of seven is not more than a clinic code of three.

**Sarah:** The advice is to collect at the highest level you can. Why?

**Kiffer:** You can always move down a level later, and you can never move back up. Age in years can become age groups, and age groups can never give you a mean age. In Cedar Valley, three hundred and ninety-two of the one thousand six hundred respondents, twenty-four point five percent, scored six or higher. The yes or no version is handy, but it treats a five the same as a three. So the team stores the items and the total and creates the yes or no version from them.

**Sarah:** Is collecting less detail ever the right call?

**Kiffer:** Sometimes. A full birth date and postal code could identify someone in a small community like Kestrel Lake, so the survey asks for age in years and not for a birth date. And people often answer income questions in ranges when they would skip a question asking for the exact amount.

**Sarah:** Section three ends with reliability and validity. How far does this lesson go?

**Kiffer:** Only as far as the words. Reliability refers to whether a measure gives consistent results, and validity to whether it measures the concept it is meant to measure. How each is assessed belongs to Health Sciences four ten. Here we use the words to choose instruments, which is Section four.

**Sarah:** So what is a validated instrument?

**Kiffer:** An instrument is a fixed set of items with instructions, response options and scoring rules. A validated one has published evidence about its reliability and validity. The catch is that the evidence comes from particular people, languages and settings. A scale tested with university students in the United States may or may not work the same way for an eighty-year-old in rural British Columbia answering by telephone. So the useful question is whether there is evidence for a population and setting like yours.

**Sarah:** Why not write your own questions?

**Kiffer:** Existing instruments let you compare with other studies and national surveys, come with evidence and scoring rules, and save months of testing. Writing new items is taught in Health Sciences three forty-one. Simple facts like age can be asked with one good question, and multi-item scales are for concepts you cannot observe directly.

**Sarah:** Where should students look for candidates?

**Kiffer:** Start with the methods sections of your key papers from Lesson two. Then look for reviews that compare measures for your concept, which the initiative known as COSMIN, short for Consensus-based Standards for the selection of health Measurement Instruments, supports. Repositories such as the Patient-Reported Outcomes Measurement Information System collect instruments in one place. And Statistics Canada publishes many of its survey questionnaires, so you can use the same wording and compare your respondents with national figures.

**Sarah:** And how do you choose among candidates?

**Kiffer:** The lesson gives nine criteria. Concept fit, population fit, published evidence, burden and readability, mode, language and culture, comparability, scoring, and permission and cost. Each is written as a question you can ask of a candidate.

**Sarah:** How did Cedar Valley use them?

**Kiffer:** The team compared four loneliness measures: the three-item UCLA scale developed by Mary Elizabeth Hughes and colleagues for large surveys of older adults, the twenty-item Version three of the UCLA scale from Daniel Russell, the six-item De Jong Gierveld scale, and a single direct question asking how often you feel lonely. It chose the three-item scale because it is short, suits telephone and written completion, and is widely used.

**Sarah:** What about the direct question?

**Kiffer:** Some people are reluctant to call themselves lonely, so a direct question may yield lower reports. The team added it as a secondary item, following a recommendation from the Office for National Statistics in the United Kingdom to use the two together. The advisory group then read the items aloud and confirmed they were clear by phone, and the First Nations Health Centre asked that the invitation explain why the questions were being asked.

**Sarah:** Once you have chosen an instrument, can you tweak it? Say, shorten the response options?

**Kiffer:** That is the temptation the lesson warns about. Evidence applies to the instrument as it was tested. Change the wording, options, order or recall period, and the evidence may no longer apply. If a change is unavoidable, record it. Translation needs a proper process, which Dorcas Beaton and colleagues describe as forward translation, back translation, expert review and pretesting. And check the terms of use, since some instruments are free, some need registration and some need a licence.

**Sarah:** How do you describe an instrument in a methods section?

**Kiffer:** Name it and its version, cite the development paper, say what it measures, give the number of items with an example, the response options and scoring, the range and direction, and any cut-off with its source. Then cite it in American Psychological Association, seventh edition, style. Lesson twelve covers the formatting.

**Sarah:** And the measurement plan pulls everything together.

**Kiffer:** It is a table with a row for each main concept, giving the conceptual definition, the operational definition, the level of measurement, the source and the permission status. A row with an empty cell shows where work remains, and the plan feeds the survey build in Lesson eight and the data dictionary in Lesson eleven.

**Sarah:** Which brings us to the whole plan. What does a complete sampling and measurement plan contain?

**Kiffer:** For a survey, the plan starts with a short sampling statement naming the target population, source population and frame, with the method labelled correctly as probability or non-probability. It lists eligibility criteria with a screening question for each, describes the recruitment channels and contacts, and includes a blank recruitment log and flow diagram template. Then a measurement plan table covers three to five key variables, with at least one existing instrument cited properly.

**Sarah:** And for an interview study?

**Kiffer:** The plan describes the purposive sampling strategy, recruitment channels and eligibility criteria, and gives conceptual definitions for the two or three concepts the interview guide will explore.

**Sarah:** If you had to leave students with one thing, what would it be?

**Kiffer:** Write your decisions down in words someone else could check. Say who your populations are, how people were chosen and what that allows you to claim, and exactly how each number in your dataset was produced. If you recruited through a link on social media, call it a convenience sample, and if you randomized, say whether you randomized who got in or who got the intervention.

**Sarah:** That's a good place to stop. Thanks, Kiffer.

**Kiffer:** Thanks, Sarah. Next week, in Lesson eight, we build the Cedar Valley measurement plan into a working survey.