This glossary collects the key concepts, people, and ideas you will meet in this lesson. Use it as a reference while you work through the material, or as a review before assessments. Type in the search box to filter entries.

Introduction and Overview

An earlier lesson covered measures of disease frequency in populations. This lesson takes the same probabilistic vocabulary and applies it at the level of a single test administered to a single person. Whether you're evaluating a new screening assay, interpreting a clinical result, or designing a surveillance algorithm, the same four-cell 2×2 logic appears: a test result that's either positive or negative, against a true disease state that's either present or absent (Sackett & Haynes, 2002). The four content sections build up from the basic attributes of a test (this section), through sensitivity and specificity (a later section), to the predictive values that depend on disease prevalence (a later section), and finally to ROC curves and likelihood ratios for tests with continuous output (a later section).

The section begins by defining a test and distinguishing screening tests from diagnostic tests. It then turns to the attributes of a test itself: its analytic sensitivity and specificity, its accuracy and precision, and the agreement between two tests or raters.

What Is a Test?

A test is any device or procedure designed to detect or quantify a sign, substance, tissue change, or body response in an individual. Tests can also be applied at the household or other levels of aggregation. In epidemiology, the term “test” extends broadly to include clinical signs, history-taking questions, survey items, and post-mortem findings.

Box 5.1 sets out why the characteristics of a test matter, both for decisions about individual patients and for the quality of research data.

Screening vs. Diagnostic Tests

Tests are applied for two broad purposes. The two cards below describe screening tests, which are applied to apparently healthy people, and diagnostic tests, which are applied to people already considered abnormal, with examples of each.

Click each card to learn more:

Despite their different uses, the principles of evaluation and interpretation are the same for both screening and diagnostic tests.

The rest of the lesson therefore treats the two kinds of test together, beginning with the attributes of the test itself.

Attributes of the Test Per Se

Some properties of a test describe it as a measurement and can be assessed without knowing the true disease status of the people tested. This part covers analytic sensitivity and specificity, accuracy and precision, and the measures used to quantify precision and agreement for quantitative and for categorical results.

Analytic Sensitivity and Specificity

The analytic sensitivity of an assay refers to the lowest concentration of a chemical compound the test can detect. The analytic specificity refers to the capacity of a test to react to only one chemical compound. These are distinctly different from diagnostic (epidemiologic) sensitivity and specificity, which are discussed in a later section.

Accuracy and Precision

The laboratory accuracy of a test relates to its ability to give a true measure of the substance of interest. To be accurate, a test need not always be close to the true value, but if repeat tests are run, the resulting average should be close to the true value.

The precision of a test relates to how consistent the results are. If a test always gives the same value for a sample (regardless of whether it is the correct value), it is said to be precise.

Figure 5.1 illustrates the two properties with four targets, in which the bullseye represents the true value and the dots represent repeated test results.

Precision and Agreement

Repeatability refers to variability obtained from repeated testing of the same sample within the same laboratory. Reproducibility refers to variability from testing the same sample in different laboratories. Agreement refers to how well two different tests (or raters) agree when applied to the same sample.

Measuring Precision: Quantitative Tests

When a test reports a number, such as a glucose concentration or a systolic blood pressure, its precision is judged by how closely repeated or paired results agree. Three measures are in common use, and the accordion below describes each of them.

Common measures for quantifying variability between pairs of test results include:

Two of these measures have simple formulae. The tabs below give each formula with a worked example and a calculator.

The first tab gives the coefficient of variation (Equation 5.1). Worked Example 5.1 applies it to five repeated measurements of one serum sample, and Calculator 5.1 reproduces that example so that the effect of a larger spread on the coefficient can be seen. The second tab gives the limits of agreement (Equation 5.2), which Worked Example 5.2 applies to two blood pressure monitors and Calculator 5.2 reproduces.

The coefficient of variation and the limits of agreement both describe tests that report a number. Many tests instead place a result in a category, such as positive or negative, and agreement between categorical results needs a measure of its own, which the next subsection introduces.

Measuring Agreement for Categorical Tests: Kappa (κ)

Cohen’s kappa is the usual measure of agreement between two categorical classifications of the same items. Box 5.2 gives the background: why some agreement is expected by chance alone, the formula for kappa (Equation 5.3), and the conventional labels for its values in Table 5.1. Worked Example 5.3 then applies Equation 5.3 to two radiologists who read the same 100 chest X-rays, and Calculator 5.3 reproduces the example so that the effect of changing the cell counts on kappa can be seen.

How Prevalence and Bias Affect Kappa

Kappa depends on more than how well two raters agree. Because p_e is computed from how often each rater uses each category, kappa also depends on the prevalence of the condition among the items rated and on any difference between the raters in how often they call a result positive (Feinstein & Cicchetti, 1990; Byrt, Bishop & Carlin, 1993).

Box 5.3 describes each of these two influences, and the example that follows it shows the prevalence effect with numbers.

A small example shows the prevalence effect. Two readers classify 100 films; reader A calls 3 positive and reader B calls 2 positive, and they never call the same film positive. They agree on the 95 films that both call negative, so p_o = 0.95. Chance agreement is p_e = (0.03 × 0.02) + (0.97 × 0.98) = 0.951, so κ = (0.95 − 0.951) ÷ (1 − 0.951) = −0.02. Agreement of 95% is no better than chance here, because almost all of it is agreement on negatives that chance alone would produce. Byrt and colleagues (1993) therefore recommended reporting a prevalence index and a bias index alongside kappa, together with the prevalence-adjusted bias-adjusted kappa, PABAK = 2p_o − 1, which shows what kappa would be if both categories were equally common and the raters used them equally often. For the two readers, PABAK = 2 × 0.95 − 1 = 0.90, and the gap between 0.90 and −0.02 shows how much of the low kappa comes from the rarity of positives.

Weighted Kappa

For tests measured on an ordinal scale, a weighted kappa accounts for partial agreement. Pairs of test results that are close (e.g., scores of 4 and 5) receive more credit than pairs that are far apart (e.g., scores of 1 and 5). Each cell of the k × k agreement table, where k is the number of categories, receives a disagreement weight that grows with the distance between the two categories assigned, i and j: linear weights are |i − j| ÷ (k − 1), and quadratic weights are (i − j)² ÷ (k − 1)². Weighted kappa is 1 minus the ratio of the weighted observed disagreement to the weighted disagreement expected by chance, κ_w = 1 − Σw_ijo_ij ÷ Σw_ije_ij, where o_ij and e_ij are the observed and chance-expected proportions in each cell. With quadratic weights it is closely related to the intraclass correlation coefficient (Fleiss & Cohen, 1973). The choice of weights changes the value, so a report states which weights were used. Weighted kappa gives a better reflection of agreement for ordinal data than the unweighted version, which treats every disagreement as equally serious.

This section has distinguished screening tests from diagnostic tests and described the attributes of a test itself: analytic sensitivity and specificity, accuracy and precision, and the coefficient of variation, the limits of agreement and kappa as measures of precision and agreement. None of these attributes shows how often a test classifies a person’s disease status correctly, which requires comparison with a gold standard and is the subject of the next section. The key takeaways and the knowledge check below review the material of this section.

Introduction and Overview

An earlier section named the attributes a test should have in the abstract. This section turns to the two quantitative properties that capture most of what we care about: sensitivity (the test's ability to find disease that is truly present) and specificity (its ability to correctly say “no” when disease is truly absent). Both are properties of the test itself, not of the population to which it is applied; that distinction becomes essential at the predictive values in a later section. Se and Sp can still differ between settings when the mix of mild and severe disease differs (the spectrum effect), so values estimated in one setting, such as a hospital study, may not hold in a screening population.

Figure 5.2 shows the 2×2 test table on which the section is built and the direction in which each measure is read from it.

The section first defines the gold standard against which a test is judged, then sets out the 2×2 table and the measures computed from it, and finally shows how sensitivity and specificity link the true prevalence of a condition to the prevalence that a test reports.

The Gold Standard

Sensitivity and specificity are estimated by comparing the results of a test with a reference that is taken to give the true disease status. Box 5.4 recalls how this kind of comparison was classified when questionnaires were validated in Lesson 3, and the paragraphs after it define the gold standard and its limits.

A gold standard (GS) is a test or procedure that is absolutely accurate: it diagnoses all cases of a specific disease and misdiagnoses none. In reality, very few true gold standards exist. Much of the error in test evaluation is due to biological variability: people do not immediately become “diseased” upon exposure, and the timescale for crossing a detectable threshold varies from person to person.

Box 5.5 notes the approaches available when no true gold standard exists.

With a gold standard, or the best available reference, in place, each person tested can be classified twice: once by the test and once by the reference. The next part arranges these two classifications in a 2×2 table.

The 2×2 Contingency Table

The concepts of sensitivity and specificity are most easily understood through a 2×2 contingency table comparing disease status to test results:

Table 5.2. The 2×2 table of disease status by test result, with the notation for its cells and totals.

In Table 5.2 the rows give the true disease status and the columns give the test result. Cells a and d hold the correct results and cells b and c the errors; m₁ and m₀ are the numbers with and without the disease, and n₁ and n₀ are the numbers who test positive and negative.

Key Measures from the 2×2 Table

Four measures are computed within the disease-status groups of Table 5.2. The cards below define sensitivity, specificity, the false positive fraction and the false negative fraction, and give the mnemonics SnNOut and SpPIn for the use of sensitive and specific tests.

Click each card to explore:

Equation 5.4 and Equation 5.5 give sensitivity and specificity in terms of the cells of Table 5.2. Worked Example 5.4 applies both to a study of 188 stool samples tested for norovirus with an EIA, and Calculator 5.4 reproduces the example so that the effect of changing the cell counts on both measures can be seen.

Sensitivity and specificity describe how a test performs within the diseased and the non-diseased groups. When a test is applied to a whole population, these two quantities and the prevalence together determine the proportion that tests positive, which is the subject of the next part.

True and Apparent Prevalence

A survey that uses an imperfect test counts the people who test positive, which is generally a different number from the people who have the condition. This part distinguishes true prevalence from apparent prevalence and shows how each can be calculated from the other. Box 5.6 first recalls the definition of prevalence and its relation to incidence and duration from Lesson 4.

The true prevalence (P) is the actual proportion of the population that has the disease. In Worked Example 5.4, P = 82/188 = 43.6%.

The apparent prevalence (AP) is the proportion that tests positive, which includes both true positives and false positives. In Worked Example 5.4, AP = 74/188 = 39.4%.

Equation 5.6 expresses the apparent prevalence in terms of the true prevalence and the sensitivity and specificity of the test. Worked Example 5.5 applies it to the norovirus study and recovers the apparent prevalence of 39.4% counted in Worked Example 5.4, and Calculator 5.5 reproduces the example.

Estimating True Prevalence from Apparent Prevalence

If the Se and Sp of a test are known, the true prevalence can be estimated from the apparent prevalence using the Rogan-Gladen formula (Rogan & Gladen, 1978):

Equation 5.7 solves Equation 5.6 for the true prevalence. Worked Example 5.6 applies it to an apparent prevalence of 0.150 measured with a test whose sensitivity is 0.363 and specificity 0.876, and Calculator 5.6 reproduces the example so that the effect of different values of Se and Sp on the corrected estimate can be seen.

Validating an Administrative Case Definition

The measures of this section apply to any rule that classifies people as cases or non-cases, including a case definition applied to health administrative records. Box 5.7 recalls the Canadian Chronic Disease Surveillance System from Lesson 2, and Worked Example 5.7 validates a diabetes case definition against chart review and then applies the Rogan-Gladen formula (Equation 5.7) to a provincial prevalence estimate.

This section has defined the gold standard, computed sensitivity and specificity from the 2×2 table, and used them to move between true and apparent prevalence. Worked Example 5.7 also computed a predictive value, which depends on the prevalence in the population tested; the next section develops predictive values in full. The reflection, key takeaways and knowledge check below review the material of this section.

Introduction and Overview

An earlier section covered sensitivity and specificity, which are properties of the test itself. This section introduces the predictive values: what an individual person should believe about their disease status given the test result. Importantly, predictive values depend on disease prevalence in the population being tested, which is why the same test can be useful in one setting and useless in another. This is the most clinically important section in the lesson.

The section defines the two predictive values and computes them from the 2×2 table and from Bayesian formulas, shows how strongly they depend on prevalence, and ends with strategies for raising the predictive value of a positive test.

What Are Predictive Values?

While Se and Sp are characteristics of the test, predictive values tell us how useful the test is for individuals of unknown disease status. Once we decide to use a test, we want to know the probability that the individual has or does not have the disease, given the test result.

Figure 5.3 is an interactive story that follows 1,000 people through a test with 95% sensitivity and 95% specificity at a prevalence of 1%, and shows how the positive predictive value emerges from the counts of true and false positives.

The two tabs below define the predictive values. The first tab gives the positive predictive value (Equation 5.8), which Worked Example 5.8 applies to the norovirus study and Calculator 5.7 reproduces. The second tab gives the negative predictive value (Equation 5.9), which Worked Example 5.9 applies to the same study and Calculator 5.8 reproduces.

In the norovirus study both predictive values were high, at 95.9% for a positive result and 90.4% for a negative result. Both were computed at the prevalence in that study, 43.6%, and the next part shows how they change when the same test is used where the disease is less common.

Effect of Prevalence on Predictive Values

Predictive values depend heavily on the prevalence of disease in the population being tested (an application of Bayes's theorem). This is why PV+ and PV− are not good measures of a test’s intrinsic performance; they vary from population to population.

Box 5.8 holds the sensitivity and specificity of the norovirus EIA fixed and recomputes both predictive values at three levels of prevalence, which Table 5.3 sets out.

As you can see, when prevalence drops to 0.1%, the PV+ falls to just 3%, meaning 97% of positive results are false positives. Meanwhile, the PV− approaches 100%. This is a fundamental challenge in screening low-prevalence populations.

Worked Example 5.10 reaches the same conclusion by counting people. It follows 10,000 people through a test with 99% sensitivity and 95% specificity at a prevalence of 1%.

Interactive 5.1 brings the cutoff, the prevalence and the overlap between the test scores of healthy and diseased people together in one simulator. It shows that moving the cutoff trades sensitivity against specificity, while lowering the prevalence leaves both unchanged and lowers the positive predictive value.

Strategies to Increase PV+

Because a low prevalence produces many false positives, the positive predictive value can be raised by changing the population tested or the way tests are used. The three cards below describe targeting high-risk groups, increasing specificity and using more than one test.

Click each card to explore:

Worked Example 5.11 applies Equation 5.8 to a proposal for universal HIV screening and shows why positive results in a low-prevalence population need confirmation.

This section has shown that the predictive values of a test depend on the prevalence of the condition as well as on its sensitivity and specificity, so that in a low-prevalence population even a highly specific test can produce mostly false positive results. The reflection, key takeaways and knowledge check below review this material, and the next section turns to tests whose results lie on a continuous scale.

Introduction and Overview

Earlier sections treated tests as if they were strictly binary, either positive or negative. In practice, most tests produce a continuous result (a blood pressure reading, an antibody titre, a probability score) that gets dichotomized at a chosen cutpoint. This section makes the cutpoint visible and shows how to choose it well: ROC curves trade sensitivity against specificity at every possible cutpoint, and likelihood ratios let a clinician update probability of disease without doing any of that arithmetic by hand.

The section explains the overlap between the test values of healthy and diseased people, uses ROC curves to describe all possible cutpoints at once, introduces likelihood ratios for moving from a pre-test to a post-test probability, and ends by asking when an accurate test should become a screening program.

Interpreting Continuous Test Results

Many tests produce results on a continuous or semi-quantitative scale (e.g., blood urea nitrogen levels, optical density values, enzyme activity). To classify individuals as positive or negative, we select a cutpoint (also called a cut-off or threshold) to determine what level indicates a positive test result.

Box 5.9 explains why every cutpoint produces errors when the test values of the two groups overlap, and Figure 5.4 shows the overlap and a cutpoint that divides it.

Each cutpoint therefore gives its own pair of sensitivity and specificity values. The next part describes a single graph that displays all of these pairs together.

Receiver Operating Characteristic (ROC) Curves

A ROC curve plots the Se (y-axis) against the false positive fraction (1 − Sp) (x-axis) computed at a number of different cutpoints (Hanley & McNeil, 1982; see also Wikipedia: ROC curve). This graphical tool helps select the optimum cutpoint and evaluate overall test performance.

The accordion below explains how to read a ROC curve, how to choose an optimal cutpoint from it, and how parametric and non-parametric ROC curves differ.

Area Under the Curve (AUC)

The AUC summarises the overall discriminatory ability of the test across all cutpoints. It can be interpreted as the probability that a randomly selected D+ individual has a greater test value than a randomly selected D− individual, equivalent to the Mann–Whitney U statistic (Hanley & McNeil, 1982).

Table 5.4 gives conventional labels for ranges of the AUC.

Table 5.4. Conventional interpretation of the area under the ROC curve (AUC).

Interactive 5.2 builds a ROC curve from two overlapping score distributions and reports the sensitivity, the false positive fraction, Youden’s J index and the AUC. Dragging the cutoff moves a point along the curve while the AUC stays the same, and increasing the separation of the distributions or reducing their spread bows the curve towards the upper-left corner and raises the AUC.

The ROC curve and the AUC summarise a test across all of its cutpoints. A clinician who receives a particular result also needs to know how much that result changes the probability of disease, and the next part introduces the measure that answers this question.

Likelihood Ratios

A likelihood ratio (LR) is the ratio of the probability of a given test result among D+ individuals to the probability of that same result among D− individuals (Deeks & Altman, 2004). LRs combine information from both Se and Sp, and allow the determination of post-test odds from pre-test odds via Bayes's theorem.

The three tabs below give the likelihood ratio for a positive result (Equation 5.10), for a negative result (Equation 5.11) and for a particular category of result (Equation 5.12). Worked Example 5.12 applies Equation 5.10 to the norovirus EIA and Calculator 5.9 reproduces it; Worked Example 5.13 and Calculator 5.10 do the same for Equation 5.11; and Worked Example 5.14 applies Equation 5.12 to a serum marker grouped into three categories, which Calculator 5.11 reproduces.

From Pre-Test to Post-Test Probability

Likelihood ratios allow you to update your assessment of disease probability after receiving a test result:

Equation 5.13 writes the three steps of Worked Example 5.15 as formulae, and Calculator 5.12 applies them to any pre-test probability and likelihood ratio.

Likelihood ratios combine the result of a test with what was known before testing to give the probability of disease for an individual. Whether a test should be offered to a whole population without symptoms is a further question, which the next part addresses.

From a Test to a Screening Program

The earlier parts of this lesson judged a test by its accuracy. Offering a test to a whole population as a screening program raises further questions about the condition, its treatment and the program itself. Box 5.10 recalls newborn screening for phenylketonuria (PKU) from HSCI 130, and the text that follows sets out the principles of Wilson and Jungner (1968) and their revision by Andermann and colleagues (2008).

The accuracy of a test is one requirement for screening among several. A screening program offers a test to people without symptoms, so it does harm to some of them (false positives, unnecessary follow-up and the treatment of disease that would never have caused illness) as well as good to others. Wilson and Jungner (1968), writing for the World Health Organization, set out ten principles for deciding whether screening for a condition is worthwhile:

1. The condition sought should be an important health problem.
2. There should be an accepted treatment for people with recognized disease.
3. Facilities for diagnosis and treatment should be available.
4. There should be a recognizable latent or early symptomatic stage.
5. There should be a suitable test or examination.
6. The test should be acceptable to the population.
7. The natural history of the condition, including its development from latent to declared disease, should be adequately understood.
8. There should be an agreed policy on whom to treat as patients.
9. The cost of case-finding, including diagnosis and treatment, should be economically balanced in relation to possible expenditure on medical care as a whole.
10. Case-finding should be a continuing process.

The first eight principles concern the condition, the test and the treatment, and the sensitivity, specificity and predictive values of this lesson are the evidence for the fifth. Andermann and colleagues (2008) reviewed the criteria proposed in the following forty years and synthesized them into a revised set that adds requirements for the program itself: it should respond to a recognized need; define its objectives and its target population at the outset; rest on scientific evidence of effectiveness; integrate education, testing, clinical services and program management; include quality assurance that minimizes the risks of screening; ensure informed choice, confidentiality and respect for autonomy; promote equity and access for the whole target population; plan its evaluation from the outset; and show that its overall benefits outweigh its harms.

Box 5.11 applies both sets of criteria to colorectal cancer screening in Canada, first through the principles that concern the condition, the test and the treatment and then through the program-level criteria.

Screening programs are judged by outcomes that three biases can distort. Lead-time bias makes survival measured from diagnosis look longer in screen-detected cases because the diagnosis is earlier, even when death is not delayed. Length-biased sampling arises because screening at intervals preferentially detects slow-growing disease, which spends longer in the detectable preclinical phase and has a better prognosis. Overdiagnosis is the detection of disease that would never have caused symptoms or death in the person's lifetime. For these reasons a screening program is evaluated by its effect on mortality from the condition in the whole screened population, ideally in a randomized trial. HSCI 230 Lesson 9, Section 2 (Observer and Detection Bias), and HSCI 230 Lesson 10, Section 2 (Immortal Time and Lead-Time Bias), treat these biases in detail.

This section has shown how the choice of cutpoint determines sensitivity and specificity, how the ROC curve and the AUC summarise a test across all cutpoints, how likelihood ratios convert a pre-test probability into a post-test probability, and how the accuracy of a test fits among the wider criteria for a screening program. The reflection, key takeaways and knowledge check below review this material.

Bringing It All Together

This lesson built up the toolkit for evaluating tests, from the basic distinction between screening (in healthy populations) and diagnosis (in suspected cases), through sensitivity and specificity, into predictive values, and finally into the more sophisticated machinery of cutpoints, ROC curves, and likelihood ratios. The arc moves from how does the test perform? to what does this result mean for this patient in this setting?

The deepest idea in the lesson is that test performance is never just a property of the test. The same sensitivity and specificity produce very different predictive values when prevalence changes, which is why a screening protocol that works in a high-prevalence clinic can collapse into mostly false positives when applied to the general population. Published diagnostic-accuracy studies themselves are subject to design-related bias that inflates reported performance (Lijmer et al., 1999), motivating the QUADAS-2 quality-assessment tool (Whiting et al., 2011) and STARD 2015 reporting standard (Bossuyt et al., 2015). As you finish the assessment, the takeaways below are the practical companions: keep them in mind whenever someone tells you a test is “accurate.”

Final Knowledge Assessment

Complete all 15 questions below with 100% accuracy to finish this lesson. You must also complete the reflection above before submitting.