Is shingling the same as MinHash?

No. Shingling defines the underlying features; MinHash can estimate set similarity using compact signatures.

Can repeated words affect the score?

Yes, through the sequences they create, but repeated identical shingles do not add multiplicity in a set-based score.

Does it find paraphrases reliably?

Not generally. A paraphrase can retain meaning while sharing few exact token sequences.

AI Glossary / Retrieval and knowledge systems / Intermediate

Shingling

Definition

Shingling represents a document using overlapping sequences of a fixed number of consecutive tokens. These sequences, called shingles, can be compared across documents to detect shared wording and near-duplicate content. In a set-based representation, repeated identical shingles appear only once.

What it means

Two documents may differ in a heading or a few words while sharing most of their text. An exact file hash treats them as different, while a representation based only on individual words loses much of the local ordering. Shingles offer a middle ground by retaining short sequences.

A k-shingle contains k consecutive tokens. With k = 3, the phrase check the battery voltage produces check the battery and the battery voltage. Character-based variants also exist, but the information-retrieval sense here uses word-like tokens.

Shingling measures overlap in surface form. It does not establish that documents express the same meaning, that one copied the other, or that either version is authoritative.

How it works

First define preprocessing, including tokenization, case handling and treatment of punctuation or boilerplate. Slide a window of length k across the resulting sequence, moving one token at a time. Collect the shingles as a set when set similarity is intended.

For two sets, Jaccard similarity is the number of shared shingles divided by the number of distinct shingles in their union. Larger k preserves more local context but makes overlap more sensitive to edits. Smaller k produces more matches, including potentially uninformative ones.

Comparing every pair of large collections can be expensive. Hashing, MinHash sketches and candidate-generation methods can help, but introduce their own assumptions and approximation issues. A shared hash value is not absolute proof that the underlying shingles are identical. The similarity threshold and the action taken after a match require separate design decisions.

Three practical examples

A search index groups near-duplicate versions of a page so users are not shown many nearly identical results.
A document archive identifies candidates for version comparison after a small amount of text changes.
A corpus preparation workflow detects repeated passages before reviewing whether they should be deduplicated.

Think of it this way

Think of cutting overlapping strips from two printed paragraphs and comparing the wording on the strips. Shared strips reveal local text overlap even when the complete paragraphs are not identical.

The analogy does not resolve authorship, meaning or importance. One changed word can reverse a claim while leaving many strips unchanged, so high overlap does not justify treating every version as interchangeable.

Worked example

One insertion changes the shingle set

Use lowercase whitespace tokens and word shingles of length three. Document A is “teams review new reports every week”. Its four shingles are teams review new; review new reports; new reports every; and reports every week.

Document B is “teams review new reports every working week”. Its five shingles are teams review new; review new reports; new reports every; reports every working; and every working week.

Three shingles are shared. The union contains 4 + 5 - 3 = 6 distinct shingles, so Jaccard similarity is 3/6 = 0.50. If a fictional review rule flags pairs with similarity at least 0.50, this pair is flagged. A stricter threshold would not flag it. This short example illustrates sensitivity to one insertion; it is not evidence that 0.50 is an appropriate threshold for a real archive.

Illustrative case study

The fictional CedarCorpus team finds many policy documents with similar wording. A shingle-based system groups candidate pairs, but one pair differs in a sentence changing an obligation from optional to mandatory.

The reviewers retain both versions and their dates. They use overlap to prioritize comparison, not to erase differences automatically. They also inspect whether shared page headers are inflating similarity between otherwise unrelated documents.

The case illustrates how near-duplicate detection supports a workflow without deciding its final action. It reports no actual corpus reduction or detection accuracy.

Common misconceptions and mistakes

A shingle set is just a bag of words. It preserves the order within each fixed-length sequence.
High shingle overlap proves equal meaning. Small textual changes can have large semantic effects.
The number of shingles is always the number of windows. Repeated windows collapse to one item in a set representation.

Business use

Shingling can reduce redundant search results, help audit collections and identify material worth comparing. It is especially useful when the question concerns repeated wording rather than broad topical similarity.

Document the preprocessing and k value so similarity scores are reproducible. Handle texts shorter than k explicitly; empty shingle sets make naive Jaccard calculations problematic. Evaluate both false matches and missed near-duplicates on representative documents. Decide whether a match should trigger grouping, review or another action, and preserve provenance when versions may carry different dates, permissions or meanings. A similarity score is evidence for comparison, not an instruction to delete.

Check your understanding

A document has six tokens and uses three-token windows. How many windows are generated before removing duplicate shingles?

Answer: Four, because 6 - 3 + 1 = 4. The final set can contain fewer than four items if windows repeat.

Related terms

N-gram
Tokenization
Near-duplicate detection
Jaccard similarity
MinHash
locality-sensitive hashing
document fingerprint
deduplication

Learn next

Read Jaccard similarity and MinHash. Compare two short documents using both two-word and three-word shingles, then inspect which edits change the scores most.

Sources

Christopher Manning, Prabhakar Raghavan and Hinrich Schutze, Introduction to Information Retrieval, Near-duplicates and shingling: https://nlp.stanford.edu/IR-book/html/htmledition/near-duplicates-and-shingling-1.html

The authors' shingle-set definition, Jaccard comparison and sketching discussion were reviewed. The report sentences and their six-item union are original and were checked explicitly. No threshold, duplication prevalence or performance result is presented as universally applicable.

Previous: Shift theorem Next: Short-Time Fourier Transform