Introduction

Lesson 7 dealt with deciding whom to study and what to measure. This lesson deals with collecting the numbers. Quantitative data in health research come from three broad places: researchers can ask people directly through a survey, reuse records that the health system keeps for its own purposes, or read clinical charts and copy specific details into a research form. Sections 1 and 2 cover surveys, Section 3 covers administrative health data, and Section 4 covers chart review.

A survey is a structured set of questions asked in the same way of every respondent, so that answers can be counted and compared. A questionnaire is the instrument itself. Writing good questionnaire items is a skill of its own, taught in HSCI 341 Lesson 3. This section concerns the operational work around the questionnaire: putting it on a platform, making it behave correctly, testing it, sending it out, following up, and keeping track of who has responded.

1.1 The Stages of Running a Survey

Running a survey involves six stages, shown in the figure. Each stage produces something that can be checked before the next begins, which matters because a single error in survey logic can send many respondents past a question they should have answered.

1.2 Choosing a Survey Platform

A survey platform is web software that displays a questionnaire, stores the answers in a database and lets the team manage invitations and export data. REDCap (Research Electronic Data Capture) was developed at Vanderbilt University for research data collection and is licensed at no cost to non-profit institutions that join the REDCap consortium; each institution runs it on its own servers (Harris et al., 2009; Harris et al., 2019). Qualtrics is a commercial survey platform that many universities license. Both can do what most projects need.

Before choosing, a team should establish where the data will be stored and whether the research ethics board and partners accept that location, since a First Nations health partner's data governance agreement may set conditions. The team should also decide who will have accounts and whether the study needs staff data entry or several sites. The Cedar Valley team chose REDCap because the university hosts it, because one project could hold the survey and the chart abstraction form used in Section 4, and because its partners accepted the storage arrangements.

1.3 Building the Survey: Fields, Codes and Logic

On a platform, each question becomes a field, and each field becomes a column in the exported dataset. Building a survey is therefore also building a dataset, and choices made now determine how easy the data will be to analyze in Lesson 11.

A data dictionary is a table that describes every field: its variable name, label, type, codes, validation and the logic that controls when it appears. REDCap stores the whole survey design as a data dictionary that can be downloaded as a spreadsheet, edited and uploaded again. Five rows of the Cedar Valley dictionary are shown below, using REDCap's column headings. Every survey should have a data dictionary of this kind.

Branching logic

Branching logic is a rule that shows a question only to respondents for whom it applies. In the dictionary above, the question about who the respondent lives with appears only if lives_alone equals 0. REDCap writes the rule on the field that should appear, with variable names in square brackets, while Qualtrics uses display logic, skip logic or a branch in the survey flow. Branching shortens the survey and prevents contradictory answers.

Branching also creates two kinds of blank. A blank in live_with for someone who lives alone means the question did not apply, while the same blank for someone who does not live alone means the person skipped it. Recording the rule in the data dictionary lets an analyst tell the two apart.

Field validation and required fields

Field validation is a check that the platform applies to an answer as it is entered. Limiting age to whole numbers from 65 to 110 catches typing errors such as 7 or 770, and date and email fields can be held to a set format. Validation prevents errors that would otherwise surface weeks later during cleaning.

A field can also be marked as required, so that the respondent cannot continue without answering. Under TCPS 2, participants may decline to answer any question, as Lesson 5 explained, so a research ethics board will usually ask why a field must be answered. A common approach is to require answers only to eligibility and consent fields and to offer a "prefer not to answer" option on sensitive items. Qualtrics distinguishes force response, which blocks progress, from request response, which reminds the respondent and then lets them continue.

1.4 Testing and Piloting

Testing means that the research team works through the survey to check that the build behaves as designed. Piloting means that a few people who resemble the intended participants complete the survey under realistic conditions. Testing finds errors in the build, and piloting finds problems with the experience. Piloting the wording of items through cognitive interviews belongs to questionnaire design in HSCI 341 Lesson 3.

1.5 Distribution, Reminders and Incentives

Distribution is the way invitations reach the sample. A platform can send each person an individual link, unique to that person, or provide one public survey link that anyone can open. Individual links let the team know who has responded, remind only non-responders and prevent duplicate answers. A public link is easy to share, and it gives no control over who answers, a weakness that Section 2 examines. Surveys can also use more than one mode, the way questions reach the respondent: web, paper, telephone or in person. A survey that offers several modes is mixed-mode.

Don Dillman and colleagues developed the Tailored Design Method, which uses several planned contacts, each with a distinct purpose, written in a respectful and personal tone (Dillman et al., 2014). The partner clinics and the First Nations Health Centre mailed invitations on the team's behalf to the stratified random sample of 5,250 older adults drawn in Lesson 7, and the team adapted the Tailored Design sequence as shown below. Each invitation carried a web address, a QR code and a unique access code, and paper questionnaires carried the same code, so that every response could be matched to the person invited.

Reminders are among the most dependable ways to raise response. A Cochrane review found that follow-up contacts, shorter questionnaires and monetary incentives were among the strategies associated with higher response to postal and electronic questionnaires (Edwards et al., 2009). When individual links are used, REDCap and Qualtrics can schedule reminders that go only to people who have not yet completed the survey. Each reminder should be shorter than the invitation and should state the closing date and how to opt out.

An incentive is a payment or gift offered to respondents. A prepaid incentive is sent to everyone with the invitation, while a promised incentive is given only to those who complete the survey. Reviews of the research have found that prepaid incentives generally raise response more than promised incentives of similar value (Singer & Ye, 2013). Under TCPS 2, an incentive should acknowledge participants' time without pressuring people to take part, and the research ethics board reviews the amount and how it is delivered.

1.6 Tracking Responses and Calculating a Response Rate

From the first mailing, the team keeps a tracking log that records what happened to each invitation. The log separates people who were never eligible or could not be reached from eligible people who did not respond. The American Association for Public Opinion Research publishes Standard Definitions with detailed rules for these categories; a simple version is enough for many small studies.

If responses from a smaller community such as Kestrel Lake lag behind, the team can ask a local organization to encourage participation or add telephone reminders there. Whether respondents differ from non-responders is the question of nonresponse bias. HSCI 230 Lesson 8 Section 2 teaches how nonresponse produces bias, HSCI 341 Lesson 3 Section 4 applies response rates to questionnaire protocols, and HSCI 341 Lesson 7 Section 1 shows when nonresponse biases an association.

Introduction

A survey can be well built and well distributed and still produce poor data. Data quality refers to how closely the recorded data reflect what real, eligible respondents meant to report. This section examines four threats that arise during and after collection: the effect of survey mode, errors in entering paper forms, responses from automated programs and impostors, and the loss or exposure of data. It ends with the export of a clean, documented file for analysis.

2.1 Mode Effects

Section 1 defined a mode as the way questions reach a respondent: on the web, on paper, by telephone or in person. A mode effect is a difference in survey results that arises because data were collected in different modes. Mode effects have two sources, and they need to be kept apart.

A selection effect occurs when different kinds of people choose, or are reached by, different modes. Older respondents, and respondents with limited internet access, are more likely to return a paper questionnaire than to answer online. If those people also differ in loneliness, the paper and web results will differ even if each person would have given the same answers in either mode. A measurement effect occurs when the same person would answer differently depending on the mode. The best-known example concerns interviewers. When an interviewer reads questions aloud, some respondents give answers that they expect the interviewer to view favourably, a tendency called social desirability bias. People may therefore report less loneliness or fewer sensitive behaviours to an interviewer than on a self-completed form. Presentation also matters: respondents who read a list of options tend to favour options near the top, while respondents who hear a list read aloud tend to favour the last options they heard (Dillman et al., 2014).

In the Cedar Valley survey, 31.1 percent of paper respondents scored 6 or higher on the loneliness scale, compared with 21.0 percent of web respondents. Paper respondents were, on average, older and more likely to live alone in the smaller communities, so much of this gap is probably a selection effect. Both modes were self-completed, without an interviewer, which limits the measurement effect, although it cannot rule one out. The team took three practical steps. It kept the wording, order and response options the same on paper and on the web, which Dillman and colleagues recommend for mixed-mode surveys. It stored the mode as a variable named mode, so that later analyses can compare or adjust for it. It also reported the mode split in the methods section, so that readers can judge the issue for themselves.

Entering and verifying paper questionnaires

Paper questionnaires must be typed into the database, a step called data entry, and every keystroke is a chance for error. The Cedar Valley research assistant entered the 560 paper forms into the same REDCap instrument used for the web survey, so that validation rules applied to both. Two approaches are common for checking entry. In double data entry, two people enter every form independently and a program lists every field where the two entries differ; REDCap offers an optional double data entry module for this purpose. In verification of a sample, a second person re-enters a random subset of forms and the team calculates an error rate.

2.2 Bots and Fraudulent Responses

A bot is a computer program that completes online surveys automatically. A fraudulent respondent is a person who completes a survey for which they are not eligible, or completes it several times, usually to collect an incentive. Both problems are concentrated in surveys that use a public link, offer a payment, and are advertised online. Researchers who have recruited through social media have reported receiving large numbers of suspicious responses within hours of posting a link (Pozzar et al., 2020), and Teitcher and colleagues (2015) described the methods that fraudulent respondents use and the ethical trade-offs in screening them out. Fraudulent responses distort estimates, use up incentive budgets and, if they go undetected, can lead to published findings that describe people who do not exist.

Prevention is easier than detection, so the first defences are built into the survey before launch.

Detecting suspicious responses

Even with prevention, a team should decide in advance which features of a response will count as warning signs, or fraud flags, and what it will do with flagged responses. Writing the rules before seeing the data prevents the team from excluding responses simply because it dislikes their answers. The rules belong in the protocol and the ethics application, and the consent form should tell participants that responses are screened for quality. The table below lists common flags.

Straight-lining, giving the same answer to every item in a grid, is sometimes used as a flag, although a genuinely lonely person might answer "Often" to all three loneliness items honestly, so it is a weak flag on short scales. Internet addresses can count as personal information, so a team that plans to collect them should say so in its ethics application; some platforms record them by default.

A worked example: screening an open link

In this lesson's illustration, the Cedar Valley team also tested an open web link, offering a $10 e-gift card, which community organizations shared to reach older adults who had recently moved into the region and were missing from the mailing list. Over one weekend the link received 212 responses, far more than expected. The team applied its pre-specified decision rule: any response with an answer in the honeypot field is excluded; any other response with two or more flags is excluded; a response with exactly one flag is reviewed by two team members; and a response with no flags is retained. Six responses are shown below.

Across all 212 responses, the rule excluded 131, sent 23 to review and retained 58. After review, the team retained 9 of the 23 and excluded 14, leaving 67 retained responses and 145 exclusions. Response R-035 shows why review matters: a person who had just moved to Riverside might still give a postal code from the previous address, and the reviewers retained that response after checking that the rest of it was consistent. The team then closed the open link and replaced it with access codes handed out by the community organizations. It reported the number of responses received, flagged, reviewed and excluded in its methods section, and kept the open-link responses separate from the 1,600 responses to the mailed survey.

2.3 Data Security

Survey data about health and social life are sensitive, and Lesson 5 set out the obligations that TCPS 2 places on researchers to protect participants' privacy and confidentiality. In practice, protection depends on a set of ordinary safeguards applied consistently. A data management plan, introduced in Lessons 5 and 6, records these safeguards in one place.

2.4 Exporting the Data

An export is a copy of the data taken out of the platform for analysis. Each export is a snapshot, so the team records when it was taken, by whom and with which settings, and gives the file a dated name, following the file naming conventions in Lesson 6 (for example 2026-03-31_CVSCS_survey-deid_v01.csv).

Exports, data dictionaries and codebooks connect this lesson to Lesson 11, which covers data entry, the documentation of variables and a first descriptive table. A clean export with a matching data dictionary is the end point of running a survey.

Introduction

Every time a person in Canada sees a physician, stays in hospital, fills a prescription or dies, a record is created. These records exist so that the health system can pay providers, manage services and register life events, and they are called administrative health data. When researchers use them to answer a research question, the use is called secondary use, because the data were collected for another purpose. The organization legally responsible for a dataset, such as a ministry of health or a provincial vital statistics agency, is its data custodian.

Administrative data are especially valuable in Canada because provincial health insurance covers medically necessary physician and hospital services for nearly all residents, so records exist for almost everyone who uses those services. In British Columbia, Population Data BC is the organization that helps approved researchers obtain and link administrative data from several custodians. Lesson 9 explains how researchers apply for access, how records from different files are linked, and how privacy is protected. This section concerns an earlier question: what each type of record contains, and what it can and cannot tell a researcher.

3.1 How an Administrative Record Is Made

An administrative record is a by-product of a transaction. A physician bills the provincial plan for a visit, a hospital reports a completed stay, a pharmacist dispenses a drug, or a family registers a death. The record contains what the transaction needed: who received the service, who provided it, when, and, depending on the file, a code describing the diagnosis, the procedure or the drug. Diagnoses are recorded as codes from the International Classification of Diseases (ICD), a coding system maintained by the World Health Organization in which each code stands for a disease or condition. Hospitals in Canada use ICD-10-CA, a Canadian version of the tenth revision, while physician billing in British Columbia uses diagnostic codes based on the older ninth revision.

Because records follow transactions, they capture contact with the health system and miss everything that happens outside it. The figure below follows one fictional Cedar Valley participant through a year.

3.2 Four Main Types of Administrative Record

The outline of each record type below uses British Columbia as the example. Other provinces keep similar files under different names, and the details of any file should be checked against the documentation its custodian publishes.

Two further files matter for many studies. Emergency department visits are recorded in hospital information systems, and many hospitals report them to CIHI's National Ambulatory Care Reporting System (NACRS). Coverage of emergency department records can differ between facilities and over time, so a team checks the custodian's documentation for the years it needs. A registry file of people enrolled in the provincial insurance plan records age, sex, area of residence and the dates each person was covered. The registry tells the team who was living in the province and could have generated records, which matters because a person with no hospital records might have been healthy or might have moved away.

3.3 What Administrative Data Can and Cannot Measure

A useful habit is to list every variable the study needs and decide, one by one, which source measures it best. The Cedar Valley team's list is shown below.

3.4 Case Definitions and the Meaning of a Code

A diagnosis in administrative data is a code that someone entered for payment or record-keeping, and a single code can be wrong. A physician who orders a test to rule out diabetes might enter a diabetes code on the claim even though the test is normal. Researchers therefore use a case definition, a rule that combines records to decide who has a condition. Case definitions are tested against a reference standard, such as clinical charts, to see how often they classify people correctly. Validation studies report the results as sensitivity, specificity and positive predictive value, the statistics taught in HSCI 341 Lesson 5 Sections 2 and 3. This is one reason chart review, covered in Section 4, remains important even in a province with rich administrative data.

Codes also change over time. Coding systems are revised, as when hospitals moved from the ninth to the tenth revision of the ICD. Fee items are added and removed, so the apparent rise of a service can reflect a new billing code. Payment models change, and when a clinic moves its physicians from fee-for-service to salary, its billing claims may fall even though patients are seen as often as before. A team studying several years should read the custodian's data documentation for such changes before interpreting a trend.

3.5 Strengths and Limitations

The Cedar Valley design combines the two kinds of data so that each covers the other's gaps. The survey supplies the exposure, loneliness, which no health record contains, together with living arrangements and other social variables. The administrative files supply the outcome, emergency department visits, without relying on memory, and they add physician visits, hospital stays and deaths over the following year. The combined design depends on linkage, which in turn depends on consent and approvals, and Lesson 9 takes up those steps. Section 4 adds a third source, the clinical chart, which holds details that neither the survey nor the administrative files contain.

Introduction

An electronic medical record (EMR) is the digital chart that a clinic or practice keeps for each patient. In Canada the term usually refers to the record held by a single practice, while electronic health record refers to records shared across organizations. A chart review, also called a medical record review, is a method in which researchers read existing charts and copy specific, predefined information into a structured research form. The copying step is called abstraction, the person who does it is an abstractor, and the research form is an abstraction form.

Gilbert and colleagues (1996) examined published chart review studies in emergency medicine and found that most did not report basic features of their methods, such as how abstractors were trained, how variables were defined and whether a second abstractor checked the work. Later guidance repeats the same recommendations: explicit variable definitions, a standard form, trained and monitored abstractors who are unaware of the hypothesis where possible, and a check of agreement between abstractors (Vassar & Holzmann, 2013; Kaji et al., 2014). This section applies them to the Cedar Valley chart review.

4.1 What Charts Contain and When to Review Them

An EMR holds two kinds of information. Structured data are stored in fields with set formats, such as the problem list, the medication list, laboratory results, measurements such as blood pressure, and billing codes. Unstructured data are free text, such as progress notes written after each visit, and scanned documents, such as letters from specialists. Structured fields can sometimes be extracted automatically; the Canadian Primary Care Sentinel Surveillance Network, for example, extracts structured data from primary care EMRs in several provinces for research and surveillance. Much of what matters for social questions, however, sits in free-text notes, and reading those notes requires a person.

Chart review is worth its cost when the needed information is recorded in clinical notes and nowhere else, such as symptoms, social history and referrals. It is also used to validate case definitions in administrative data, as Section 3 explained. Its limitations follow from the fact that charts are written for care: documentation varies between clinicians, clinics and EMR software, what was never written down cannot be abstracted, and reading hundreds of charts takes many hours.

4.2 Planning a Chart Review

The figure below shows the planning sequence. The first four steps take place before any study chart is abstracted, and the agreement check returns the team to the form and manual when a problem appears.

Each variable should answer part of the research question, because every extra field adds abstraction time and privacy risk. The review window is the period of the chart that abstractors read, defined relative to a fixed date, here the index date set in the protocol. A window stated in advance prevents one abstractor from reading five years of notes while another reads two.

4.3 The Abstraction Form and Manual

The abstraction form lists each variable with its allowed values, and the abstraction manual explains, for each variable, exactly what counts, where in the chart to look and how to handle unclear cases. The form below is an excerpt from the Cedar Valley REDCap form, with the manual's definitions summarized in the middle column.

The most important idea in chart abstraction is the difference between no and not documented. A chart that says "lives with husband" supports the value lives with others. A chart that says nothing about living situation shows only that no one wrote it down. If abstractors record silence as "no", the study mistakes gaps in documentation for facts about patients, so each field needs a rule for what silence means. For living_doc, silence is recorded as not documented. For lonely_note, the variable is defined as whether a note mentions loneliness, so a chart without such a note is correctly recorded as no, and the results describe documentation, which is a different thing from the true prevalence of loneliness.

4.4 Training and Monitoring Abstractors

Abstraction is a skill, and two careful people given the same chart can record different values when a definition leaves room for interpretation. Training and monitoring reduce that variation.

4.5 Checking Agreement Between Abstractors

Inter-rater agreement describes how often two abstractors who independently abstract the same chart record the same value. Before main abstraction began, both Cedar Valley abstractors independently abstracted the same 30 charts, which is 10 percent of the 300. For each variable, the team calculated percent agreement, the share of charts on which the two abstractors recorded the same value. The protocol set a target in advance of 90 percent agreement for each key variable.

The table shows the results for lonely_note. The two abstractors agreed on 6 charts where both recorded Yes and on 18 where both recorded No, and disagreed on 6 charts.

Percent agreement is (6 + 18) ÷ 30 × 100 = 80.0 percent, below the target. At the next meeting, the abstractors reviewed the 6 disagreements together and found that 5 involved notes in which a family member, rather than the patient, described the patient's isolation. Abstractor A had counted these notes and Abstractor B had not. The team added the rule on reports from family members to the manual, and both abstractors independently abstracted 20 further charts, agreeing on 19 of them, which is 19 ÷ 20 × 100 = 95.0 percent. During main abstraction, a random one chart in ten was abstracted by both to check for drift, and Dr. Hart settled any disagreement the abstractors could not resolve by discussion.

What percent agreement does not show

Percent agreement is easy to calculate and explain, and it has two limits. The first is that agreement measures consistency between abstractors, which is a different property from accuracy: if both abstractors overlooked referrals in scanned letters, they would agree perfectly and both be wrong. Comparing a sample of abstractions with the lead investigator's careful reading gives some check on accuracy.

The second limit is that two abstractors can agree often by chance, especially when one value is much more common than the other. Suppose referrals appear in about 5 percent of charts, and suppose each abstractor, without reading carefully, simply recorded Yes on a random 5 percent of charts and No on the rest. They would both record No on about 0.95 × 0.95 = 90.25 percent of charts and both record Yes on about 0.05 × 0.05 = 0.25 percent, so they would agree on about 90.5 percent of charts without having read them at all. A high percent agreement for a rare item can therefore hide poor abstraction. Cohen's kappa (Cohen, 1960) is a statistic that adjusts agreement for the amount expected by chance. Kappa is taught in HSCI 241 Lesson 7 and HSCI 341 Lesson 5, and HSCI 410 Lesson 7 covers other reliability statistics. For now it is enough to know that these statistics exist and why they are needed.

4.6 Privacy and Practical Arrangements

Clinic charts are among the most sensitive records in health care, and the clinic, as custodian, decides whether and how researchers may see them. The Cedar Valley team signed an agreement with each partner clinic setting out who may access charts, where, which variables will be recorded and how long data will be kept, and the research ethics board approved the procedures (Lesson 5). Abstractors work on site and enter only coded values into REDCap, never copying notes word for word, because notes contain the names of patients, family members and providers. REDCap data access groups, which restrict each user to the records of a particular site, keep each clinic's records separate when clinic staff help with abstraction. The link file that connects study numbers to clinic chart numbers is held by the data manager and stored apart from the abstraction data.

4.7 Three Sources, Three Kinds of Evidence

The Cedar Valley study uses three quantitative sources, and each answers a different part of the research question.

Bringing It All Together

This lesson followed quantitative data from three sources. Running a survey is an operational sequence of building, testing, piloting, distributing, monitoring and exporting, and each stage can be checked before the next one begins. On a platform such as REDCap or Qualtrics, every question becomes a field with a variable name, coded values, branching logic and validation, and the data dictionary records those decisions. Distribution with individual access codes, a planned contact schedule, reminders and a carefully chosen incentive supports response, and a tracking log allows the team to calculate and report a response rate.

Survey data quality depends on more than the questionnaire. Mode effects combine differences in who chooses each mode with differences in how the mode shapes answers. Paper forms must be entered and verified. Public links with incentives attract bots and fraudulent respondents, which teams prevent with access codes and CAPTCHAs and detect with flags and a decision rule written in advance. Security safeguards and a checked, documented export complete the process.

Administrative health data and chart reviews draw on records created during care. Billing, discharge, prescription and vital statistics records measure contact with services and life events across whole populations without relying on memory, while saying nothing about loneliness and depending on coding and payment practices. Chart review captures what clinicians wrote, and its quality rests on an explicit abstraction form and manual, trained and blinded abstractors, and agreement checks. The Cedar Valley study combines the three so that each source covers gaps in the others.

The final reflection asks you to combine the three data sources from this lesson into a short data collection plan for a new question.

Final Knowledge Assessment

These terms, tools and people appear in this lesson on collecting quantitative data through surveys, administrative records and chart reviews.