Is every pipeline an ETL pipeline?

No. Pipelines can follow ETL, ELT, streaming or other patterns and may include validation, indexing or model-related steps.

Does an orchestrator process the data itself?

It may invoke work, but orchestration and data processing are distinct responsibilities. The actual computation can run in other services or engines.

Can a pipeline be partly manual?

Yes, for example when an approval gate is required. The manual step and its effect on dependencies and freshness should be explicit.

AI Glossary / Data engineering and architecture / Beginner

Data pipeline

Definition

A data pipeline is a defined sequence or network of steps that moves data from sources toward a usable destination, often transforming and checking it along the way. It includes the dependencies and operating behavior needed to repeat that work reliably.

What it means

A pipeline might collect sales records, standardize dates, join product information and publish a reporting table. Another might turn approved documents into extracted text and a searchable index. The steps depend on the purpose; there is no mandatory universal number of stages.

The word “pipeline” can suggest a straight line, but real workflows often branch and converge. Two sources can load independently before a later step combines them. A quality failure may route records to review rather than to the main output.

Data ingestion is one possible part of a pipeline. The broader pipeline includes what happens after arrival and how the final output becomes available. A scheduled script can be a simple pipeline, while a larger one may use specialized orchestration and processing services.

How it works

Define the sources, output and transformations, then specify which steps depend on which others. Decide how work starts: a schedule, arriving data or another event. Record enough run and input information to understand what each execution processed.

Orchestration coordinates tasks, their order, retries and timeouts. The processing code or engine performs the actual transformations. Apache Airflow, for example, represents task dependencies through directed acyclic graphs; that orchestration structure does not itself validate every business calculation inside a task.

Reliable operation requires explicit failure and recovery behavior. A retry should not accidentally duplicate final records or publish half an output. Monitoring should cover freshness, completeness and quality as well as whether individual tasks exited successfully. Branches and joins require particular care so dependent work uses compatible inputs.

Three practical examples

1. Business reporting: daily invoices and payments are loaded, checked and reconciled before a cash-collection summary is refreshed.

2. AI search: approved source versions are extracted, divided into passages, enriched with metadata and indexed, followed by checks that expected documents are searchable.

3. Forecasting: sales and promotion data are prepared into versioned training features, with validation before a model-training job is allowed to use the dataset.

Think of it this way

Think of preparing an event: venue confirmation and equipment delivery can proceed separately, but setup needs both completed. Some steps can be retried without consequence; others, such as placing an order, need duplicate protection. A data pipeline has similar dependencies and recovery concerns. Unlike an event checklist, it can process continuous streams and large numbers of records automatically.

Worked example

A fictional daily report needs sales data and a product reference table. Sales ingestion takes 8 minutes, reference ingestion takes 3 minutes, validation and joining take 5 minutes, and publication takes 2 minutes.

If the two independent ingestions run concurrently, the earliest completion under these simplified assumptions is max(8, 3) + 5 + 2 = 15 minutes. Running everything sequentially would take 18 minutes. These invented timings exclude scheduling and resource contention.

Now suppose the reference load fails. The pipeline must not silently use yesterday's reference table unless that fallback is explicitly permitted and reported. A successful sales load does not make the final report ready.

The calculation illustrates dependency-aware coordination. It does not imply that increasing concurrency always improves completion time: shared resources, source limits and downstream bottlenecks can change the result.

Illustrative case study

Illustrative fictional case: Maple Reference rebuilds an AI index every night. Its workflow marks the job successful when text extraction completes, even though indexing sometimes fails afterward. Editors believe newly approved articles are available when they are not.

The team changes the publication criterion to include index completion and a coverage check against the approved source manifest. It keeps the previous valid index available until the replacement meets those checks and records the source version used by each build.

The fictional lesson is to define success at the consumer-facing output, not merely at an intermediate task. The case does not establish a particular vendor's atomic index-switching capability or a measured reliability improvement.

Common misconceptions and mistakes

Do not confuse a scheduled task with a complete operational design. Do not assume “all tasks succeeded” proves the business output is correct; tasks may accept empty or stale inputs unless checked.

Avoid unrestricted retries for non-idempotent writes or external effects. Do not hide rejected records inside a general success status. Do not assume a pipeline must move every intermediate dataset physically if a supported design can reference it safely. Finally, distinguish a reproducible input version from a mutable path named latest.

Business use

Use a pipeline to make repeated data work explicit, observable and recoverable. Assign an owner, define the expected output and freshness, and document what happens when an input is late or a step fails.

For AI projects, capture source and transformation versions, approval state and downstream publication status. Test a partial failure and recovery, not just the ideal path. Useful operating measures include output freshness, expected-versus-processed coverage and unresolved failures. A pipeline is successful when it delivers the intended usable data under its agreed conditions, not simply when an orchestration dashboard looks green.

Check your understanding

A report needs two source loads. One succeeds and one fails, but the final script runs against an older cached input. Is the report necessarily current?

Answer: No. The workflow must define and verify the required input versions. Successful execution with stale data can still violate the report's freshness requirement.

Related terms

Data ingestion
ETL
ELT
Orchestration
Directed acyclic graph
Data Quality
task dependency
idempotency
data lineage

Learn next

Draw one existing data workflow with its inputs, outputs and dependencies. Mark which steps can run independently and which writes need duplicate protection. Then define a concrete publication check and test how the workflow recovers from a failed intermediate step.

Sources

Databricks, data pipelines and their components: https://www.databricks.com/blog/what-are-data-pipelines

Apache Airflow, core concepts: https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/

Apache Airflow, primary documentation source describing DAG responsibilities and dependencies: https://github.com/apache/airflow/blob/main/airflow-core/docs/core-concepts/dags.rst

Databricks, pipeline operation and orchestration context: https://docs.databricks.com/aws/en/ldp/concepts/how-to-use-pipelines

Timing and index-publication scenarios are original fictional examples.

Previous: Data physicalization Next: Data point