Is ingestion the same as ETL?

No. Extraction and loading overlap with ingestion, while ETL also includes transformation. Teams may combine these stages, but the responsibilities should remain clear.

Must ingestion be real time?

No. A daily or hourly batch may meet the business requirement. Choose freshness deliberately rather than assuming the fastest pattern is always best.

Does automatic schema handling guarantee compatibility?

No. Accepting a new field or preserving unexpected data does not prove that downstream reports and models interpret the changed structure correctly.

AI Glossary / Data engineering and architecture / Beginner

Data ingestion

Definition

Data ingestion is the process of bringing data from its sources into a destination where it can be stored or processed. It covers receiving or collecting the data and loading it through a defined, monitored path.

What it means

Sources can include business databases, uploaded files, application events, sensors and external services. Destinations may be databases, data lakes, queues or analytics platforms. Ingestion connects the producer of information to the environment that will use it.

The process may handle a bounded batch, such as yesterday's sales files, or an ongoing stream of events. The appropriate pattern depends on required freshness, volume, source capabilities and operating cost. Frequent ingestion does not automatically make downstream reports or AI answers equally fresh.

Ingestion is also different from complete preparation. A successfully received file may still contain invalid values, duplicate records or text that cannot be extracted. The system should distinguish arrival, acceptance and readiness for the intended use.

How it works

A typical ingestion flow authenticates to the source, identifies new or changed data, reads it and writes it to a destination. It records enough progress information to resume after interruptions and enough source metadata to explain where the data came from.

The flow needs explicit handling for duplicates, malformed records, changing schemas and late arrivals. Some systems reject unexpected records; others preserve them separately for review. Neither choice makes the values correct automatically.

Retries are especially important. A source may resend data after an acknowledgment is lost. A destination can use stable event identifiers, appropriate upserts or other idempotent handling to avoid unintended duplication. Claims about exactly-once behavior must be scoped to the relevant components and failures; receiving a message once does not by itself guarantee every downstream business effect occurs once.

Three practical examples

1. AI knowledge base: newly approved documents are copied into a landing area with their version, source and access metadata, ready for extraction and indexing.

2. Retail analysis: a daily sales export is loaded into an analytics platform, with file counts and totals reconciled against the source system.

3. Equipment monitoring: sensor readings arrive continuously and are recorded with device identifiers and event times, while delayed or malformed readings are handled explicitly.

Think of it this way

Think of a receiving desk accepting deliveries into a warehouse. Staff record what arrived, where it came from and which packages need inspection. Arrival does not mean every item is ready for sale. Similarly, ingestion establishes a controlled intake, while validation and preparation determine whether the received data is fit for its next use. Digital retries and copies make duplication a more complicated issue than the analogy suggests.

Worked example

A fictional document service expects a daily manifest of 100 files. The ingestion run downloads 98, rejects one because authentication to its source fails, and cannot find one listed file. Its report should show 98 received and two unresolved, rather than simply “run completed.”

Of the 98 received files, extraction later succeeds for 95 and fails for three. Ingestion coverage is therefore 98 out of 100, while extractable coverage is 95 out of 100. Searchable coverage could be lower still if indexing has not finished.

On retry, the service downloads the two missing files. Stable source-version identifiers allow it to recognize the earlier 98 rather than create duplicate document versions. This example shows why separate stage counts, durable progress and a clear identity rule matter. The percentages describe the invented manifest only, not a general performance benchmark.

Illustrative case study

Illustrative fictional case: Shoreline Orders builds a near-real-time analytics feed. After a temporary connection failure, the feed replays several order-change events. A dashboard incorrectly adds the amounts again because the destination treats every arrival as a new sale.

The team clarifies whether each record represents a new business event or an updated state. It introduces appropriate identifiers and replay handling, then tests interruption and recovery with the same source sequence.

It also records event time separately from ingestion time, so a delayed change does not appear to have occurred when it was received. The fictional case illustrates semantic and recovery design, not a claim that one identifier solves every ordering or transactional problem.

Common misconceptions and mistakes

Do not equate a successful connection with complete data arrival. Do not equate arrival with valid, approved or searchable content. Do not silently discard schema changes that may contain important information.

Avoid using a processing timestamp as though it were the time the business event occurred. Do not assume retries are harmless if destination writes are not designed for them. Conversely, do not disable recovery merely to avoid duplicates; define the required delivery and processing behavior and test the relevant failure cases.

Business use

Use ingestion monitoring to answer five practical questions: what was expected, what arrived, how fresh it is, what was rejected and what remains unresolved. Add source and version information so downstream users can trace anomalies.

For an AI project, include document permissions and deletion or replacement handling where relevant. A newly ingested version should not leave an obsolete version silently active in retrieval. Agree ownership for failures and recovery. The useful business measure is timely availability of suitable data, not merely the number of bytes transferred or jobs marked successful.

Check your understanding

A pipeline receives every file, but ten percent cannot be parsed. Is the whole dataset ready for analysis?

Answer: No. Receipt is complete, but preparation is not. Report the parsing failures separately and decide how affected records should be repaired, excluded or otherwise handled.

Related terms

Data pipeline
Batch processing
Stream processing
ETL
ELT
change data capture
schema evolution
idempotency
data freshness

Learn next

Follow one dataset from source to destination and define what each stage considers successful. Then test a missing file, a duplicate event and a changed field. Learn checkpointing and idempotent writes next so recovery behavior is understood before relying on unattended ingestion.

Sources

Databricks, data ingestion overview: https://www.databricks.com/blog/what-is-data-ingestion

Databricks, batch and streaming processing: https://docs.databricks.com/aws/en/data-engineering/batch-vs-streaming

Databricks, loading patterns and unexpected-data handling: https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/patterns

Apache Kafka, delivery and processing semantics: https://kafka.apache.org/40/design/design/

Manifest and replay scenarios are original teaching examples.

Previous: Data independence Next: Data integrity