Does a search engine need to crawl the public web?

No. It can receive a private collection through uploads or connectors.

What does ranking do?

It orders candidate results according to the system's relevance criteria or other stated sorting rules.

Can a search engine return an answer directly?

Some systems do, but producing an answer is an additional capability beyond retrieving and presenting source items.

AI Glossary / Graphs and symbolic AI / Beginner

Search engine

Definition

A search engine is a system that accepts a query and retrieves relevant items from a collection. It typically uses searchable representations, matching rules and ranking methods to help users find documents, records or other content without inspecting the entire collection themselves.

What it means

A search engine can serve the public web, a company knowledge base, an online catalog or a personal document collection. The collection defines what it can potentially retrieve. A page that exists somewhere is not necessarily present in a particular engine's searchable data.

The engine interprets a query and selects candidates that appear useful. Some systems use exact terms, some use semantic representations, and many combine signals. Ranking estimates usefulness for a request; it is not a certificate that the top result is true, complete or authoritative.

A search engine is also different from a browser. A browser displays and interacts with web content, while a search engine supplies retrieval results. An AI assistant may use a search engine, but generating an answer adds another step with its own possible errors.

How it works

Acquire content through crawling, uploads, connectors or another collection-specific process. Parse it into searchable units and retain useful fields such as title, text, identifiers and metadata. Updates and deletions need a path into the searchable representation.

In a conventional text index, an inverted index associates terms with the documents containing them. This allows a query to find candidate documents without reading every document from beginning to end at query time. Other retrieval representations may support additional matching methods.

Process the query using compatible rules, apply relevant filters and score or otherwise select candidates. Ranking can consider term matches, document fields and other signals appropriate to the system. The result interface may show snippets and links to help users judge relevance.

Evaluate whether users can find the intended information, not merely whether results are returned. Missing content, mismatched vocabulary, poor ranking and stale records can each cause failure at a different stage.

Three practical examples

A website search engine retrieves product pages and lets visitors filter results by category or availability.
An internal document engine helps employees locate a procedure using words from its title or body.
A research application retrieves candidate passages for an AI assistant, while keeping the retrieved sources available for review.

Think of it this way

Think of a library catalog that helps you identify books related to a request. Its organization saves you from opening every book, but the catalog must contain the book and describe it in a way your query can reach.

The analogy does not imply that search engines rely only on catalog labels. They may index full text and other representations. Finding an item still does not establish that every statement inside it is correct.

Worked example

Consider three short documents after simple lowercasing and word tokenization:

D1: “solar panel installation”
D2: “panel maintenance guide”
D3: “solar battery guide”

An inverted index records solar in D1 and D3, and panel in D1 and D2. For the Boolean query solar AND panel, intersecting these two document lists returns only D1. For solar OR panel, their union returns D1, D2 and D3.

The first query requires both terms; the second allows either. Neither calculation ranks documents or recognizes synonyms such as photovoltaic. Those behaviors would require additional rules or representations.

If D1 were absent from the index, the AND query would return no result even though the original document might still exist in the source folder. This separates collection coverage from query processing and demonstrates why “no results” does not always mean “no such information exists.”

Illustrative case study

In a fictional project, Maple Library adds a search box to its training portal. Users searching for “leave policy” receive little help because relevant documents use “annual absence” and the initial matching configuration does not connect the expressions.

The team reviews actual user wording, checks the documents' titles and tests a controlled vocabulary adjustment against other queries. It evaluates whether the change helps without making unrelated documents dominate results.

The case illustrates that search quality depends on the relationship between user language and indexed content. It does not claim a measured improvement or recommend one universal ranking method.

Common misconceptions and mistakes

A search engine searches everything that exists. It operates over the content and sources available to it.
The first result must be true. Ranking is not factual verification.
No results proves the information is absent everywhere. Index coverage and query interpretation can explain the failure.
A browser and a search engine are the same component. They serve different functions.
Every search engine uses generative AI. Conventional retrieval can work without answer generation.
Matching all query words guarantees relevance. Terms can appear in the wrong context or in an outdated document.

Business use

For a business website, define representative user questions and identify the items that should satisfy them. Test exact names, product codes, common wording and ambiguous queries, including cases where an empty result is appropriate.

Monitor freshness and successful navigation to source content as well as response time. A useful-looking snippet may disappoint if the destination is missing, outdated or unavailable to the user.

When search feeds an AI assistant, distinguish retrieval quality from answer quality. Relevant documents may be missed, and a generator may misuse documents that were retrieved correctly. These stages need separate evidence before claiming an effective end-to-end experience.

Check your understanding

Using the worked index, which documents match panel AND guide?

Answer: Only D2. Panel occurs in D1 and D2, while guide occurs in D2 and D3; their intersection is D2.

Related terms

Learn next

Build an inverted index for five short documents. Try both AND and OR queries, then create a query using a synonym absent from the texts. Explain which failures require better matching and which require adding missing content.

Inverted index

Sources

Manning, Raghavan and Schütze, building an inverted index: https://nlp.stanford.edu/IR-book/html/htmledition/a-first-take-at-building-an-inverted-index-1.html

Their introduction to scoring and ranking: https://nlp.stanford.edu/IR-book/html/htmledition/scoring-term-weighting-and-the-vector-space-model-1.html

Relevant author-provided textbook sections were reviewed. The three-document index and Maple Library case are original illustrations.

Previous: Search cost Next: Search game