1.1 Why the Same Question Returns Different Results

A student who types the same words into PubMed, Embase, CINAHL, Google and Google Scholar will receive five different sets of results. The counts differ, the first records differ, and some studies appear in one tool and are missing from another. These differences arise from how each tool stores information and how it decides which records match a query. A reviewer who understands those mechanisms can build a search that finds what it should, explain the search to others, and judge which tools can support a systematic search and which cannot.

This section explains indexing and contrasts searching subject terms with searching the words that authors wrote. Section 2 compares Boolean retrieval in databases with relevance ranking in Google and Google Scholar, Section 3 introduces recall and precision, and Section 4 examines the reproducibility of web search. Writing and translating a full search string belongs to Lesson 4, and grey literature and targeted web searching belong to Lesson 5.

1.2 What a Database Record Contains

A bibliographic database is an organized collection of records, each describing one published item such as a journal article. MEDLINE, the database of the United States National Library of Medicine (NLM), is the best-known example in the health sciences, and PubMed is the free interface through which most people search it. Embase, CINAHL, APA PsycInfo and Web of Science are other major databases that the Cedar Valley team will search.

Each record contains two kinds of fields. The first kind comes from the authors and the publisher: the title, the abstract, the author names, the journal, and any keywords the authors chose. The second kind is added by the database: the subject headings that describe what the article is about, the publication type (for example, randomized controlled trial or review), the language, and other descriptive codes. Adding this second kind of information is called indexing. A search can look in either kind of field, and the results differ depending on which kind it uses.

The example shows why indexing matters. The authors described their participants as "older adults" and "seniors" and their outcome as being "lonely". Another team writing about the same topic might say "elderly people" and "social disconnection". Indexing replaces this variety with a single standard heading for each concept, so that every article about loneliness in people aged 65 and older carries the same headings regardless of the words its authors chose.

1.3 How a Controlled Vocabulary Works

A controlled vocabulary is a fixed list of approved terms used to describe the subject of each record. When the list is organized with relationships among the terms, it is called a thesaurus. Health databases use thesauri with five main features, and each one affects how a search behaves.

First, each concept has one preferred term, also called a subject heading or descriptor, such as Loneliness. Second, each preferred term has entry terms, which are synonyms and variant spellings that point to it. In MeSH, the entry terms for Social Isolation include "Social Exclusion", and the entry terms for Aged include "Elderly". Third, the headings are arranged in a hierarchy of broader and narrower terms, which MeSH calls its tree structure. Fourth, each heading has a scope note that defines how indexers use it; the scope note for Aged, for example, defines it as a person 65 through 79 years of age and directs indexers to the narrower heading "Aged, 80 and over" for older people. Fifth, many databases allow subheadings (MeSH calls them qualifiers) that narrow a heading to an aspect such as psychology, therapy or epidemiology.

The hierarchy makes possible an operation called explode. Exploding a heading retrieves records indexed with that heading and with every narrower heading beneath it. Indexers are instructed to assign the most specific heading that fits an article, so an article about loneliness is indexed with Loneliness rather than with the broader Social Isolation. A search for Social Isolation that is not exploded will therefore miss articles indexed only with its narrower headings. A related option, major topic, restricts a search to records in which the heading describes a main point of the article rather than a minor one. Exploding increases the number of records retrieved, and restricting to major topic decreases it; Section 3 explains how these choices trade off.

1.4 Three Thesauri: MeSH, Emtree and CINAHL Subject Headings

Each major health database has its own thesaurus, and the headings, hierarchies and search syntax differ from one database to the next. A search built with MeSH headings therefore has to be translated, heading by heading, before it can run in Embase or CINAHL. Lesson 4 teaches that translation. The cards below summarize the thesauri the Cedar Valley team will meet.

Because each thesaurus is maintained separately, the same concept can carry a different heading, a different position in the hierarchy, or no heading at all, depending on the database. The tabs below show how a heading for the loneliness concept is written in four common interfaces. The aim at this stage is to recognize that each interface writes headings in its own way; Lesson 4 returns to the details.

1.5 Free-Text Searching and Why It Is Still Needed

A free-text search (also called a keyword or text-word search) looks for words in the fields written by the authors, usually the title, the abstract and the author keywords. It finds a record only when the record contains the exact words searched, or their variants when the searcher uses truncation, which Lesson 4 teaches. A free-text search for "loneliness" would miss the illustrative record shown in Section 1.2, whose title says "lonely", unless the searcher also included "lonely" or used truncation.

Free text is still needed because indexing has gaps, and each gap causes a subject-heading search to miss relevant records. The items below describe the main gaps, and Web of Science and web search engines, which have no thesaurus, rely on free text entirely.

PubMed also performs a step called automatic term mapping. When a user types words without field tags, PubMed tries to match them to MeSH headings, journal names and author names, and then combines the matched heading with a search of the words in all fields. Because an untagged search therefore does more than was typed, systematic searchers write each heading and free-text term with explicit field tags, and they check PubMed's "Details" display to see how a query was translated.

1.6 Combining the Two Approaches

Because each approach covers the other's gaps, the standard practice in systematic searching is to search every main concept with both its subject headings and a set of free-text terms, joined with OR, and then to join the concepts with AND. Chapter 4 of the Cochrane Handbook for Systematic Reviews of Interventions (Lefebvre et al., 2023) recommends this approach for searches that aim to be comprehensive. Section 2 explains exactly what OR and AND do to sets of records, and Lesson 4 shows how to assemble the full strategy in a concept table.

The two approaches also help each other. Librarians often read the headings and abstracts of a few known relevant articles to find headings and wording to add, which is one reason the Cedar Valley librarian asks the intern to collect known relevant studies (Section 3).

2.1 The Inverted Index

Section 1 described the fields in a database record. This section explains how a search system uses those fields to answer a query. A database with tens of millions of records cannot read every record each time a user searches. Instead, it prepares in advance a structure called an inverted index. For every word and every subject heading in the collection, the inverted index stores a list of the records that contain it, called a postings list. The idea resembles the index at the back of a textbook, which lists for each term the pages where it appears. Manning, Raghavan and Schütze (2008) give a full account of the inverted index in their textbook Introduction to Information Retrieval, and the description here follows theirs in simplified form.

When a user enters a query, the system looks up each term in the inverted index and works with the postings lists rather than with the records themselves. A search for "loneliness" therefore does not find "lonely", because the two words have separate entries in the index, unless the searcher asks for both or uses truncation. A search on a subject heading consults the entries for headings, which are separate from the entries for words in the title and abstract.

2.2 Boolean Retrieval in Bibliographic Databases

Boolean retrieval treats a query as a logical statement and returns every record for which the statement is true. It takes its name from the English mathematician George Boole, whose 1854 book The Laws of Thought set out an algebra of logical operations. Three operators do almost all the work in health searching. AND returns records that are in both sets (the intersection), so it narrows a search. OR returns records that are in either set or both (the union), so it broadens a search. NOT returns records in the first set that are absent from the second (the difference), so it removes records.

Boolean retrieval in a bibliographic database has three properties that matter for evidence synthesis. It is exhaustive: the system returns every record that satisfies the logic, and the count it reports is the exact size of that set. It is transparent: anyone who reads the search strategy can work out why each record was or was not retrieved. It is repeatable: the same strategy run in the same database on the same date returns the same set, so another team can check the search and a later team can update it. These properties are the reason systematic reviews rely on bibliographic databases for their main searches, and they are the reason PRISMA-S (Rethlefsen et al., 2021) asks reviewers to report every strategy in full.

Boolean logic also has well-known weaknesses. A record either matches or it does not, with no partial credit, so a record that says "lonely" fails a query that asks for "loneliness". The searcher must anticipate every relevant word in advance. NOT is risky, because it removes any record that mentions the excluded term, including relevant ones; a search for older adults NOT dementia would drop a trial of a befriending program that mentions dementia once in passing. Databases also differ in the order in which they process operators when parentheses are missing, so careful searchers always use parentheses to group terms. Lesson 4 teaches the full syntax.

2.3 Relevance Ranking

Relevance ranking (also called ranked retrieval) works differently. Instead of deciding whether each record matches, the system gives each document a score for how well it appears to answer the query and presents the documents in order of that score. Documents that contain only some of the query words can still be returned, lower in the list. Users of a ranked system look at the first page or two and stop, so the ranking, rather than the logic of the query, decides what the user sees.

Ranking methods developed through decades of published research in information retrieval. Gerard Salton and colleagues at Cornell University developed the vector space model, which represents queries and documents as weighted lists of terms and ranks documents by their similarity to the query. Two weighting ideas from that research remain central. Term frequency gives more weight to a document that uses a query word more often. Inverse document frequency, proposed by Karen Spärck Jones (1972), gives more weight to words that are rare in the collection, because a rare word says more about what a document is about than a common one. Probabilistic models developed by Stephen Robertson and colleagues led to the widely used BM25 formula (Robertson and Zaragoza, 2009), which adds limits on the effect of repeated words and adjusts for document length.

The same principle applies to the toy collection. In a ranked system, a query for loneliness older adults would return record 1 first, because it contains both words. Records 4 and 6, which contain only "loneliness", would come next, because "loneliness" appears in three of the eight records and is rarer than "older adults", which appears in five. Records 2, 3, 5 and 8 would follow. Record 7 contains neither word and would not appear. The ranked system returns seven records in order, where the Boolean query in the exercise returned exactly one.

2.4 Comparing the Two Models for Evidence Synthesis

Relevance ranking is well suited to finding a good answer quickly, which is what most web users want. A systematic or scoping review has a different goal: it aims to find all the eligible studies, or as close to all as resources allow, and to show others exactly how they were found. The table compares the three kinds of tool the Cedar Valley team will use against that goal.

Published evaluations support the roles in the last row. Haddaway et al. (2015) found that Google Scholar shows at most the first 1,000 results of any search and recommended using it as a supplementary source, particularly for grey literature, with a defined number of results screened. Gusenbauer and Haddaway (2020) tested 28 academic search systems against criteria for systematic searching, including Boolean functionality, reproducibility and bulk export, and concluded that Google Scholar was unsuitable as a principal search system for systematic reviews, although useful as a supplementary one. Bramer et al. (2017) showed that no single database retrieved all the studies included in a large set of reviews, which is why the Cedar Valley team plans to search five databases rather than one.

3.1 Two Questions About Any Search

Sections 1 and 2 explained how databases find records. This section asks how well a particular search performs. Two questions capture most of what matters. The first is how many of the relevant records the search found. The second is how much of what the search returned is relevant. The measures that answer these questions, recall and precision, became the standard pair through the Cranfield experiments that Cyril Cleverdon led in England in the late 1950s and 1960s, which compared indexing systems by testing them against a set of documents whose relevance had been judged in advance. The same two measures are still used to evaluate search engines, search filters and the AI-assisted screening tools that Lesson 6 describes.

Any record in a database falls into one of four groups, depending on whether it is relevant to the review question and whether the search retrieved it. The figure shows these groups for a search of a single database.

Recall is the same quantity that a diagnostic test calls sensitivity: the proportion of true cases that the test detects. Precision resembles the positive predictive value of a test: the proportion of positive results that are true cases. Bachmann et al. (2002) proposed the number needed to read as a more intuitive form of precision, by analogy with the number needed to treat. It states how many records a reviewer must read, on average, to find one relevant record. The fourth group, d, is very large in any bibliographic database, so a measure built on it, such as specificity, is close to 100 percent for almost every search and says little about performance. For that reason, search evaluation uses recall and precision rather than sensitivity and specificity as a pair.

3.2 A Worked Example from the Cedar Valley Review

The following example is a teaching device. It assumes that the true number of relevant records is known, which never happens in practice; Section 3.4 explains how recall is estimated when it is unknown. Suppose that MEDLINE contains exactly 30 studies that meet the Cedar Valley eligibility criteria. The librarian drafts two searches. Search A is narrow: it looks for the words loneliness, older adults and intervention in titles only. Search B is broad: it combines the subject headings from Section 1 with free-text synonyms in titles and abstracts for each of the three concepts. All numbers are illustrative.

The comparison shows the trade-off in concrete terms. Search B costs five times as many records to screen and about ten more hours of reviewer time with dual screening. In exchange, it finds 9 more eligible studies and misses 3 instead of 12. Each additional study found by Search B costs about 67 extra records to screen (600 extra records divided by 9 extra studies). Search A misses 40 percent of the eligible studies, which would leave the evidence brief for Cedar Valley built on little more than half of the relevant evidence in MEDLINE.

The example also shows why specificity is uninformative. If MEDLINE holds about 30 million records, both searches leave out more than 99.99 percent of the irrelevant ones, so specificity cannot distinguish a good search from a poor one. Recall and precision, by contrast, differ clearly between the two searches.

3.3 Why Recall and Precision Move in Opposite Directions

The curves in the figure have a typical shape. The first terms a librarian adds find many relevant records, so recall climbs quickly. Each further synonym or heading finds fewer new relevant records and many more irrelevant ones, because rarer wordings are also used in unrelated contexts. Recall therefore levels off while precision keeps falling. No search reaches 100 percent recall at a workable precision in most health topics, and the practical question becomes how far along the curve the team can afford to go.

Most decisions in search design move a search along this curve. The table summarizes the usual direction of each effect. These are tendencies, and an individual change can behave differently in a particular database.

Lesson 4 introduces validated search filters, such as the filters for randomized trials in the Cochrane Handbook, which are published in sensitivity-maximizing and precision-maximizing versions. Those two labels refer directly to the recall and precision trade-off described here.

3.4 Estimating Recall When the Total Is Unknown

In a real review, nobody knows how many relevant records a database contains, so recall cannot be calculated directly. Precision is easier: once the team has screened the retrieved records, precision is the number judged eligible divided by the number screened. Recall has to be estimated, and two methods are common.

The first method uses a benchmark set (also called a validation set or test set) of known relevant studies, collected independently of the search being tested. Sources include the included studies of earlier reviews, studies suggested by content experts, and studies the team already knows. The librarian checks that each benchmark study is present in the database and then runs the draft search. The estimated recall is the proportion of benchmark studies that the search retrieves. A missed benchmark study is also informative, because its title, abstract and headings show which terms the search lacks. The second method, relative recall (Sampson et al., 2006), pools all the eligible studies found by every search method in the review, including other databases and citation chasing, and treats that pool as the reference standard for each individual search.

3.5 Why Reviews Favour Recall

Chapter 4 of the Cochrane Handbook (Lefebvre et al., 2023) advises that searches for systematic reviews should aim for high sensitivity, that is, high recall, while accepting that precision will be low. The reasons can be stated in terms of costs and of bias, and the cards below set them out.

Precision still matters, because reviewer time is finite. The Cedar Valley team has twelve weeks, and screening thousands of records with two reviewers competes with charting, appraisal, the environmental scan and writing. Its plan therefore aims for high recall in the main database searches, and it manages the workload in other ways: removing duplicates before screening (Lesson 7), using a screening platform, piloting eligibility criteria so that decisions are quick, and possibly using active-learning screening tools (Lesson 6). The team will consider lowering recall only after these options, and it will report any limit it sets.

4.1 What Reproducibility Means for a Search

A search is reproducible when another person, following the documented method, can run it again and obtain the same results, or can explain any differences. Reproducibility matters in evidence synthesis for three reasons. It allows peer reviewers and readers to check that a review searched as it claims. It allows the same team, or a different one, to update the review later by rerunning the search for newer records. It also allows a reader to judge the risk that relevant studies were missed. PRISMA-S, the reporting guideline for literature searches developed by Rethlefsen et al. (2021), exists largely to make searches reproducible, and Lesson 4 teaches it in detail.

A related idea is transparency: the extent to which a report shows exactly what was done, even when the result could not be obtained again. A search can be fully transparent without being reproducible. This distinction is central to web searching, because, as this section shows, a reviewer can report a Google search completely and still be unable to guarantee that anyone else will see the same results.

Bibliographic database searches come close to reproducibility, although even they fall short of it. Databases add new records every day, sometimes including older articles added later. Thesauri change each year, as the Social Prescribing heading did in 2025. Platforms change their search engines and syntax. A rerun of a MEDLINE search a year later, limited to the original date range, will usually return a very similar set, and the differences can be traced to these known causes. Web search engines behave very differently, for the reasons set out below.

4.2 Personalization and Context

Personalization means that a search engine adjusts results to the person searching. Google's public documentation states that its systems use the user's location, search history and settings, that the language of the query shapes the language of the results, and that pages a user has visited often may be moved higher in that user's results. It also states that users can see whether personal information affected a result and can turn personalization off. Hannak et al. (2013) measured personalization in Google web search by comparing the results that real accounts and controlled test accounts received for the same queries. They found measurable personalization, with being signed in to an account and geographic location as the main sources of difference.

The term "filter bubble", popularized by Eli Pariser (2011), describes the concern that personalization shows people mainly what fits their past behaviour. Google states that its systems are not designed to create filter bubbles. Studies that have measured personalization in web search have generally found it present but modest in average size. For evidence synthesis, the size of the effect matters less than its existence: any personalization means that two reviewers running the same query can see different results, and neither can tell from the results page alone which records were affected.

Some sources of variation remain even when personalization is turned off. Results depend on the approximate location that the search engine infers from the network connection, which a private browsing window does not hide. They depend on the country version of the search engine, on the interface language, and on settings such as filters for explicit content. They can also depend on the device, because page usability is one of the factors Google describes. A reviewer can control some of these factors and record the rest, but cannot remove them.

4.3 Ranking Opacity and Change Over Time

Ranking opacity means that the method used to order results is not disclosed in enough detail for an outsider to predict or check it. Section 2 showed that Google publishes only general categories of ranking factors and that Google Scholar describes its ranking in a single paragraph. Commercial search engines have reasons to keep the details private, including competition and the need to resist manipulation of rankings by website owners. For a reviewer, the consequence is that no one can explain why a given page appeared in position three rather than position thirty, or whether it would appear at all if the search were run again.

Opacity combines with change over time. Search engines revise their ranking systems continually, and the web index is rebuilt as pages change. Google Scholar's documented use of citation counts in ranking has a further consequence: as articles gain citations, their positions change, so the order of results for the same query drifts from month to month even if no new articles are published. Because Google Scholar shows at most about the first 1,000 results, articles near that boundary can move in and out of view.

Other features of the results page make the problem harder. The counts that Google and Google Scholar display ("about" so many results) are estimates that can change between pages of the same search and should not be reported as the size of a result set. Results pages may include advertisements labelled as sponsored, highlighted answer boxes drawn from a single page, and, since 2024, AI-generated summaries above the results for some queries. Each of these elements is produced by its own undisclosed process, and each can differ between users and days.

Gusenbauer and Haddaway (2020) included reproducibility among the criteria they used to judge academic search systems, and they judged that Google Scholar did not meet their requirements for reproducible searching. This finding, together with the cap on viewable results and the lack of bulk export, is why reviews use Google Scholar and Google as supplementary sources. Supplementary searching is still valuable. Much of the information the Cedar Valley environmental scan needs, such as program descriptions, funding announcements and evaluation reports from community organisations, exists only on the open web and will never appear in MEDLINE.

4.4 Making Web Searches Transparent

Because a web search cannot be reproduced exactly, the reviewer's aim shifts to recording enough for a reader to understand what was searched and to judge what might have been missed. PRISMA-S asks reviewers to report web searches and other supplementary methods alongside database searches, and Lesson 5 provides a full documentation template for grey literature and website searching. The items below give the minimum record that this lesson's mechanisms imply.

4.5 From Search Engines to AI Tools

The mechanisms in this lesson carry forward to the artificial intelligence tools that Lesson 6 examines. AI search engines and research assistants usually begin with a retrieval step that resembles relevance ranking, so they inherit the opacity and the dependence on an undisclosed index described here. They then add a step that generates text from what was retrieved, and that step can produce different wording, different sources and, at times, citations to works that do not exist, even when the same question is asked twice. The same distinction between reproducibility and transparency applies: a reviewer who uses these tools must record what was asked, when, with which tool and version, and must verify each source against the original publication.

Bringing It All Together

This lesson has explained why the same question returns different results in different search tools. Bibliographic databases describe each record with subject headings drawn from a controlled vocabulary, such as MeSH for MEDLINE, Emtree for Embase and CINAHL Subject Headings for CINAHL. Headings gather varied wording under one term and allow a search to include narrower concepts through explosion, but they have gaps: new concepts such as social prescribing receive headings late, some records are never indexed, and indexing varies. A sensitive search therefore combines headings with free-text terms for every concept.

Databases retrieve records with Boolean logic applied to an inverted index, which makes their searches exhaustive, transparent and close to reproducible. Google and Google Scholar rank documents with methods that are documented only in general terms, display only part of their results, and change with the searcher, the place and the day. Recall and precision provide a common language for judging any search. Reviews favour recall because an irrelevant record costs seconds to exclude, while a missed study is invisible and may bias the findings.

For the fictional Cedar Valley review, these principles produce a plan with five databases searched with both headings and free text, a benchmark set to estimate recall, and web searching used as a documented supplement. Lesson 4 turns that plan into a full search string.

The final reflection asks you to explain the lesson's main ideas to a decision-maker in the Cedar Valley case.

Final Knowledge Assessment

These terms, tools and people appear in this lesson on how databases and search engines find information.