AI Glossary / Generative AI and language / Intermediate

Exploratory data analysis

Definition

Exploratory data analysis, or EDA, is an approach to understanding a dataset through summaries, visualizations, and iterative investigation before committing to a final model or explanation. It looks for structure, unusual observations, missingness, relationships, and possible problems in how data was collected or represented. EDA helps generate questions and assess assumptions. It does not turn every observed pattern into a confirmed finding, and it is a general statistical practice rather than a technique specific to generative AI.

What it means

A dataset's columns and row count reveal only part of its story. Values may use inconsistent units, represent repeated observations, change over time, or omit particular kinds of cases. EDA combines numerical summaries with views of the actual distribution and relevant context. Averages can hide skew, groups can behave differently, and apparent outliers may be errors or important real events. The purpose is to understand what the data can reasonably support before treating it as a dependable basis for modeling.

How it works

Begin with the data's origin, unit of observation, column meanings, and collection process. Check types, ranges, duplicate keys, missing values, and category frequencies. Use plots and grouped summaries to examine distributions, relationships, and changes over time. Investigate surprising results against source records where possible, recording decisions instead of silently removing inconvenient observations. Form hypotheses and identify needed follow-up evidence. In predictive work, protect final test data from exploratory choices that would leak information into model design or evaluation.

Three practical examples

• Prediction preparation: discover that a supposed numerical feature contains mixed units before fitting a model.
• Text-dataset review: examine document lengths, languages, repeated content, and label frequencies before training or evaluating a language system.
• Operational analysis: compare response-time distributions by request type to distinguish routine behavior from a small set of unusually slow cases.

Think of it this way

Think of inspecting ingredients before cooking: check what is present, what is missing, and whether anything looks unusual before choosing the recipe. The analogy captures preparation and questioning. It does not mean every unexpected ingredient should be discarded or that inspection alone proves the final dish will succeed.

Worked example

Suppose a fictional response-time sample in seconds is [1, 2, 2, 3, 42]. The mean is 50/5 = 10 seconds, while the median is 2 seconds. Reporting only the mean could suggest a typical request takes around 10 seconds, although four of the five requests finish within 3 seconds. A simple ordered plot reveals the long tail and prompts investigation of the 42-second case.
The team checks the source record and finds that the slow request involved a much larger document. It does not automatically delete the value as an error. Instead, it records document size and compares response times within size groups. With only five observations, it avoids fitting a strong relationship or claiming a stable percentile. If the value had been stored in milliseconds while the others were seconds, the correct action would be unit correction with an audit trail. EDA identifies the question and possible explanations; additional evidence determines which explanation is supported.

Illustrative case study

A fictional document-classification team finds that one label is strongly associated with a particular file-name pattern. Further inspection shows that files from two collection sources were named differently and labeled under different procedures. The apparent predictive signal may reflect collection artifacts rather than the document content the model is meant to understand. The team reviews sampling and evaluation splits before training. This prevents a visually striking association from becoming an unquestioned feature that would fail when file naming changes.

Common misconceptions and mistakes

EDA is not just producing many charts, and a beautiful chart is not automatically informative. Correlation does not establish causation. Trying many subgroup comparisons can produce interesting-looking patterns by chance, so exploratory findings need appropriate confirmation. Missing values may be informative rather than random. Outliers should not be removed solely because they weaken a preferred story. Inspecting the final test set to choose features or preprocessing can compromise its later role as independent evaluation evidence.

Business use

Use EDA to identify data limitations, guide modeling choices, and focus further collection or review. Keep a record of questions, observations, and unresolved explanations. Separate confirmed data corrections from optional analytical transformations. Involve people who understand how records were generated, especially when fields have operational meanings that a chart cannot reveal. For AI-assisted analysis, verify calculations and code against the data; a fluent explanation of a pattern is not evidence that the underlying computation or interpretation is correct.

People also ask

Does EDA require a hypothesis before starting?

It can begin with broad questions and develop more specific hypotheses, but findings discovered through exploration should be labeled accordingly.

Can EDA use numerical summaries as well as graphics?

Yes. The combination often reveals more than either alone.

Is EDA finished once a model is trained?

No. New data, prediction failures, or changed collection processes can require renewed exploration, while preserving the integrity of evaluation data.

Check your understanding

Why should the 42-second observation not be deleted automatically in the worked example?

Answer: It may represent a real large-document request rather than a recording error. Removing it without investigation would hide part of the workload and could produce a misleading account of performance.

Related terms

• Descriptive (or summary) statistics
• Outlier
• Distribution
• Visualization (graphics)
• Confounding
• Data leakage
• Data Quality
• Descriptive statistics
• data profiling
• missingness
• visualization
• hypothesis generation
• confirmatory analysis

Learn next

Take a small dataset and compare mean, median, range, missing counts, and a simple plot for each important variable. Investigate one surprising observation using its source context. Then distinguish a question generated by the exploration from a claim supported by independent confirmation. Study sampling and validation design before turning exploratory patterns into model features or business conclusions.

More in Generative AI and language

• Exploration bonus - An exploration bonus is an extra incentive used by a learning algorithm to favor actions or states that are uncertain, unfamiliar, or insufficiently tested.
• Exploration function - An exploration function is a rule that combines an agent's current value estimates with information about how well actions or states have been explored.
• Exploration problem - The exploration problem is the challenge of deciding how an AI agent should gather experience when it does not yet know which actions lead to useful outcomes.
• Exploratory forecasting - Exploratory forecasting examines how the future might develop from present conditions, past trends, and alternative assumptions.
• Factored Language Model - A Factored Language Model represents a word or token through several associated factors, such as its surface form, stem, grammatical category, or learned...
• Factuality - Factuality is the extent to which factual claims in an AI response are accurate about the world, within the relevant time, scope, and context.

Sources

NIST/SEMATECH, exploratory data analysis handbook: https://www.itl.nist.gov/div898/handbook/eda/eda_d.htm NIST, What is EDA?: https://www.itl.nist.gov/div898/handbook/eda/section1/eda11.htm Scikit-learn, common pitfalls and data leakage: https://scikit-learn.org/stable/common_pitfalls.html

Previous: Exploration problem Next: Exploratory forecasting