AI Glossary / Statistics and probability / Intermediate
Data collection
Definition
Data collection is the planned process of obtaining and recording observations or measurements for a defined purpose. It includes deciding what to observe, from whom or what, when, and by which method. In AI projects, collection choices influence what a model can learn, whose experiences are represented, and how confidently its performance can be evaluated.
What it means
Data do not arrive as a neutral picture of the world. A questionnaire captures answers to particular questions; a transaction log records activity within a particular system; a sensor records a physical signal under particular operating conditions. Each source has boundaries and possible errors.
Primary data are gathered specifically for the study or project. Secondary data already exist and are reused for another purpose. Neither category is automatically superior: relevance depends on definitions, coverage, quality, timing, and appropriate access.
A collection plan connects the question to evidence. “Collect everything” is not a substitute for deciding what information is needed and how it will support a decision.
How it works
Start with the intended population, unit of observation, outcome, and period. Translate concepts into measurable variables with clear units and definitions. Choose a census, sample, experiment, observation process, or suitable existing source.
Design and pilot the instruments: questionnaires, logging events, sensor configurations, or annotation procedures. Establish identifiers, timestamps, allowed values, and a way to distinguish missing data from meaningful zero values. Define applicable permissions, access controls, and retention arrangements before collection begins.
During collection, monitor coverage, response, equipment or software failures, and deviations from the protocol. Preserve provenance and documentation so later users can understand how each field was produced. Validate records without silently rewriting unusual but legitimate observations. Collection ends with a documented dataset suitable for its purpose, not simply a large file.
Three practical examples
- An AI support project collects resolved queries and independently reviewed answer-quality labels, recording which channels and languages are included.
- A retailer gathers a probability sample of customer feedback using tested questions and tracks nonresponse instead of treating all invited customers as respondents.
- A maintenance project collects sensor readings with timestamps, measurement units, device calibration information, and confirmed event records for later evaluation.
Think of it this way
Think of setting up a camera before taking a photograph. The angle, field of view, exposure, and timing determine what the image can show. Editing afterward may improve clarity, but it cannot reliably recover people who were never in the frame. Data collection similarly determines much of the evidence available for later analysis.
Worked example
Consider an original collection exercise involving a defined group of 200 employees. All are invited to answer one clearly specified question about using an AI assistant during the previous week. A total of 120 provide valid answers, and 72 of those answer yes.
The response proportion is 120/200 = 60%. The reported usage proportion among respondents is also 72/120 = 60%, but these two percentages describe different quantities.
Without information about the 80 nonrespondents, the usage proportion for all 200 employees is not established as 60%. If none of the nonrespondents used the assistant, the full-group proportion would be 72/200 = 36%. If all did, it would be (72+80)/200 = 76%.
The interval from 36% to 76% is a logical bound based on complete uncertainty about nonrespondents, assuming the recorded answers are accurate. It is not a statistical confidence interval. Follow-up information or defensible assumptions may narrow the uncertainty, but collecting more detail from the same respondents alone does not resolve the missing group.
Illustrative case study
In this fictional case, Northbank Learning collects examples to evaluate an AI writing assistant. The first dataset consists of messages voluntarily submitted by highly engaged users. The team initially plans to call it representative of every employee's writing needs.
Reviewers identify missing job roles, language groups, and occasional users. They revise the collection plan, define a sampling approach, and pilot instructions so participants understand what content is appropriate to submit. They keep a separate evaluation set and document exclusions.
The revised approach improves the relevance of the evidence being gathered, but the case makes no numerical claim about model improvement. It illustrates that dataset quality begins before training or evaluation.
Common misconceptions and mistakes
Large volume does not guarantee representative coverage. Repeated observations from a narrow group may add precision about that group while revealing little about absent groups.
Administrative records and digital logs were often designed for operational purposes. A missing event may mean an action never occurred, a logging failure, or a workflow that bypasses the system. Those interpretations should not be treated as interchangeable.
Do not confuse a missing response with a negative response or a genuine zero. Also, a census attempt can still contain nonresponse, measurement, and processing errors. AI-generated labels are derived judgments, not automatically verified observations, and should be documented and checked accordingly.
Business use
A business collection plan should state the decision it supports, the population covered, the responsible owner, and the criteria for usable evidence. Allocate effort to difficult-to-reach or poorly measured parts of the population rather than only collecting convenient records.
Track the cost of collection alongside completeness, timeliness, consistency, and accuracy. Preserve enough metadata to detect changes in definitions or instrumentation over time. For an AI project, ensure that evaluation data reflect intended use and that information unavailable at prediction time is not accidentally included as an input. These choices often matter more than increasing dataset size indiscriminately.
People also ask
Is collecting existing records still data collection?
Yes, sourcing and assembling secondary data is part of obtaining evidence, although its original collection purpose must be considered.
Does a high response rate prove absence of bias?
No. It is useful operational information, but bias also depends on how respondents differ from nonrespondents and on measurement quality.
Can collection be fully automated?
Parts can be, but automation does not remove the need to define variables, check coverage, and monitor failures or changing conditions.
Check your understanding
Check your understanding: What does the example's 60% usage figure directly describe?
Answer: The 120 respondents.
Why is 36% to 76% not a confidence interval?
Answer: It is a bound over all possible answers from the 80 nonrespondents.
What should a missing sensor reading become in a dataset?
Answer: An explicitly documented missing value or status, not an invented zero measurement.
Related terms
- Sampling
- Census
- Measurement error
- Nonresponse
- Primary data
- secondary data
- data provenance
- annotation
- dataset documentation
- data validation
Learn next
Write a short collection protocol for one business question, including the population, variables, timing, missing-data codes, and quality checks. Pilot it on a small set of observations and revise ambiguous definitions before scaling collection.
More in Statistics and probability
- Cuzick - Edwards test - The Cuzick - Edwards test is a statistical test for spatial clustering of cases relative to a comparison population represented by controls.
- Cyclical effect - A cyclical effect is a recurring rise and fall in a time series.
- Data assimilation - Data assimilation is the process of combining observations with a model-based estimate to infer the state of a changing system.
- Data editing - Data editing is the process of checking data for errors, omissions, and inconsistencies and handling identified problems through documented procedures.
- Data generating process - A data generating process, often abbreviated DGP, is the mechanism or probabilistic specification that produces observations.
- Data normalization - Data normalization is the transformation of numerical data into a scale or form suitable for analysis or machine learning.
Sources
Sources: Australian Bureau of Statistics, Data sources, https://www.abs.gov.au/statistics/understanding-statistics/statistical-terms-and-concepts/data-sources, and Types of error, https://www.abs.gov.au/statistics/understanding-statistics/statistical-terms-and-concepts/types-error. The employee nonresponse calculation and fictional AI evaluation case are original teaching material.
Previous: Data collaboratives Next: Data colonialism