AI Glossary / Generative AI and language / Beginner

Sampling with replacement

Definition

Sampling with replacement is a selection procedure in which an item remains eligible to be selected again after it has been drawn. A resulting sample can therefore contain repeated occurrences of the same original item. Under the common version with independent draws and fixed probabilities, each draw uses the same candidate pool and probability distribution. Replacement means putting an item back into eligibility, not substituting a different item or deleting the original record.

What it means

Imagine selecting document identifiers from a list. After choosing identifier A, a with-replacement procedure can choose A again immediately or later. A without-replacement procedure would remove A from eligibility for the remainder of that draw sequence.

This distinction affects how many unique items appear and how probabilities are calculated. A sample containing 100 draws need not contain 100 distinct observations.

The idea is general despite the supplied language-AI category. It appears in bootstrap analysis, training-data selection, and ensemble methods. The sampling unit might be a row, document, customer, or another defined entity, and that choice should be explicit.

How it works

Define the candidate items and assign selection probabilities. For uniform sampling, each of n items has probability 1/n on every independent draw. For weighted sampling, probabilities can differ while still summing to one.

Draw an item, record its identity, and leave the candidate pool available for the next draw. Repeat until the requested number of draws is reached. Software often implements this by sampling indices, so no physical removal or replacement of records is necessary.

Keep repeated selections when the method requires them. Deduplicating afterward changes the sampling result and generally changes the procedure's statistical properties. Also distinguish the number of draws from the number of unique original observations represented in them.

Three practical examples

1. A bootstrap evaluation repeatedly draws document-level results with replacement to study variation in an average score.

2. A bagging procedure trains different constituent models on bootstrap samples, so some original training examples appear multiple times in a given sample and others are omitted.

3. A training sampler gives rare categories a larger chance of being drawn while allowing repeated examples, producing a deliberately weighted training stream.

Think of it this way

Think of drawing a numbered card, recording the number, and returning the card to a thoroughly mixed deck before drawing again. The same card can reappear because it remains in the deck.

The analogy has limits. In real data, repeated draws are references to the same observation, not newly collected independent evidence about the world. The draw mechanism can be independent even though duplicated content comes from one original record.

Worked example

Take four equally likely items A, B, C, and D, and draw four times with replacement. One possible sample is A, C, A, D. It contains four draws but only three distinct items; B is absent.

On each draw, A has probability 1/4. The probability that A is never selected in four independent draws is (3/4)^4 = 81/256, about 31.64%. Therefore, its probability of appearing at least once is 175/256, about 68.36%.

By symmetry, the expected number of distinct items is 4 × 175/256 = 2.734375. An actual sample contains an integer number of unique items, of course. Drawing as many times as there are items does not guarantee that every item appears; without replacement, four draws would include all four.

Illustrative case study

Fictional case: Hazel Metrics bootstraps a small set of reviewed summaries. An analyst notices repeated document identifiers in a resample and removes them as apparent duplicates.

A colleague explains that the repeats are intentional. Removing them changes the size and weighting of the resample, so the team restores the specified procedure. It separately investigates whether the original evaluation dataset itself contains accidental duplicate documents.

The distinction matters: duplicate source records may be a data-quality issue, while repeated selections within a with-replacement resample are part of the method. The team documents both checks rather than treating all repetition as the same problem.

Common misconceptions and mistakes

Sampling with replacement does not imply uniform probabilities. A sampler can use fixed unequal weights and still permit repeated selections.

It also does not guarantee a duplicate in every sample. Repetition is allowed, but a particular set of draws can happen to contain only distinct items.

Another mistake is treating repeated selections as additional independent observations from the original population. They reuse existing evidence. Finally, sampling more times than the candidate count is possible with replacement, whereas a without-replacement sample cannot exceed the available distinct items under the ordinary definition.

Business use

Understanding replacement helps teams interpret training samples and evaluation uncertainty correctly. Reports should distinguish original dataset size, requested draw count, and unique-item count when those differences matter.

For bootstrap-based analyses, choose the sampling unit to match the data structure. If several rows belong to one customer or conversation, independently resampling rows may not reflect the intended uncertainty. For weighted training, document the selection probabilities and evaluate on data appropriate to the real workload rather than assuming the resampled training mix represents deployment traffic.

People also ask

Can the same item be selected twice in a row?

Yes. It remains eligible after the first draw, so immediate repetition is allowed.

Does replacement mean changing the dataset?

Not necessarily. A program can draw indices and reuse records without modifying the original dataset.

Are draws always independent?

Independence with fixed probabilities is the common simple setup, but the broader phrase only specifies renewed eligibility. If probabilities adapt or draws otherwise depend on history, that should be stated separately.

Check your understanding

You draw 20 times with replacement from ten records. Is that impossible, and must the resulting sample contain ten unique records?

Answer: It is possible because records can repeat. It need not contain all ten unique records; some may never be selected during those 20 draws.

Related terms

• Bootstrap
• Bagging
• Random forest
• Oversampling
• Sample size
• Empirical distribution
• Resampling
• Sampling without replacement
• weighted sampling
• independent draws

Learn next

Draw a few four-item samples from four labeled cards, replacing the card each time. Record both draw count and unique count. Repeat without replacement and compare the outcomes. Then inspect one software sampler's replacement and probability settings so its behavior matches the intended analytical method.

More in Generative AI and language

• Response set - A response set is the collection of outputs produced by a language model for a corresponding collection of prompts.
• Sampling - Sampling means drawing an outcome according to a probability distribution.
• Sampling rate - Sampling rate is the number of signal measurements taken or represented per second, usually expressed in hertz.
• Self-attention - Self-attention is a neural-network mechanism that updates representations of positions in a sequence by combining information from positions in that same...
• Sequence-to-sequence (seq2seq) - Sequence-to-sequence, often shortened to seq2seq, describes a model or task that transforms an input sequence into an output sequence.
• Sequential importance sampling - Sequential importance sampling, or SIS, is a probabilistic inference method that builds candidate sequences incrementally and updates a weight for each...

Sources

Sources: Google Machine Learning Glossary, https://developers.google.com/machine-learning/glossary — primary definition and bagging context. NumPy Generator.choice documentation, https://numpy.org/doc/stable/reference/random/generated/numpy.random.Generator.choice.html — primary interface for replacement and weighted selection. SciPy bootstrap documentation, https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html — primary use in bootstrap resampling. The four-item probability calculation and fictional duplicate-review case are original illustrations.

Previous: Sampling risk Next: Sardinas - Patterson algorithm