Can a pipeline have more than one sink?

Yes, but consistency and completion across them require explicit design.

Is a sink always a database?

No. Files, object storage, messaging systems, and supported service endpoints can serve as destinations.

Does a successful transformation guarantee a successful sink write?

No. Mapping, permissions, destination availability, and commit failures can still prevent or alter delivery.

AI Glossary / Data engineering and architecture / Intermediate

Sink (pipelines)

Definition

A sink is the destination stage or component to which a data pipeline writes its output. Depending on the system, the word may refer to the destination itself, such as a table or file store, or to the connector and operation that perform the write. A sink is paired conceptually with a source, which supplies input. It is a role within a pipeline, not a particular storage technology.

What it means

A pipeline is incomplete until its results reach a usable destination with the intended meaning. Transformations may produce correct records, but the sink determines how those records are stored or delivered. It can append events, replace a dataset, update existing rows, publish messages, or perform another supported write behavior. The same database can be a sink for one pipeline and a source for a later pipeline, so the label depends on the flow being described.

How it works

Configure the destination connection, authentication, mapping, format, and write behavior. The pipeline passes transformed records to the sink, which serializes or maps them and interacts with the destination. Batching, parallelism, transaction support, and commit behavior depend on the connector. Define what happens when a record is rejected, a partial write occurs, or the job retries. Successful delivery semantics require coordination across the pipeline and destination; a sink label alone does not imply exactly-once effects.

Three practical examples

Reporting: a nightly transformation writes cleaned sales records into a warehouse table that reporting tools can query.
AI preparation: an annotation pipeline writes versioned training records into a designated dataset location with a documented schema.
Event processing: a stream processor publishes derived events to a message destination for another application to consume.

Think of it this way

Think of the receiving station at the end of a conveyor. It must place items in the right containers and record what arrived, not merely let them fall off the belt. The analogy is limited because digital writes can be retried, duplicated, rolled back, or committed in separate batches, and those details determine what downstream users actually see.

Worked example

A toy batch contains 100 records with stable event identifiers. The sink successfully appends 60 records before the job fails. If a retry appends all 100 again without duplicate handling, the destination may contain 160 rows, including 60 repeated events. A design using an appropriate unique event key and idempotent write behavior can avoid producing an additional effect for the repeated identifiers, but the exact implementation must support that behavior. If two legitimate events accidentally share an identifier, careless deduplication could instead discard valid data. The worked example therefore connects retry behavior to identity design; it does not claim that enabling a generic “retry” option guarantees safe delivery.

Illustrative case study

Fictional example: Willow Insights validates transformed records in a preview and assumes the final dataset will match. Its production sink writes to a different folder and uses append behavior on reruns. The team documents the target path, batch identifier, publication process, and replay expectations, then tests a controlled interruption. This invented case demonstrates why sink configuration is part of data correctness, without claiming a real incident or a measured reliability improvement.

Common misconceptions and mistakes

Do not confuse a sink with an error bin; rejected records may have a separate destination. Do not assume append, overwrite, and upsert are interchangeable. An upsert needs a meaningful matching key and product-specific semantics. Parallel sinks may complete in different orders, and writing to two destinations does not automatically form one atomic transaction. Also distinguish internal exactly-once state processing from end-to-end output guarantees. Read the connector's supported behavior and test failures rather than relying only on the pipeline engine's headline description.

Business use

Sink design affects freshness, duplicate rates, recovery, cost, and whether business users can safely consume results. For AI projects, it also affects whether a training dataset can be reproduced and whether a failed run exposes an incomplete version. Specify destination ownership, schema expectations, access, retention, and publication criteria. Monitor records attempted, accepted, rejected, and committed using definitions appropriate to the connector. Reconciliation should reflect intended transformations rather than assuming source and sink counts must always be identical.

Check your understanding

A job failed after writing part of a batch. Is rerunning it automatically safe because the input file has not changed?

Answer: No. The sink's retry and write semantics determine whether repeated records create duplicates, replace data, or have idempotent effects. Stable input alone does not settle that question.

Related terms

Data pipeline
ETL
ELT
Checkpoint
source
connector
destination
append
overwrite
upsert
idempotency
exactly-once processing
dead-letter queue

Learn next

draw one pipeline from source to sink and document the destination's write mode, record identity, commit boundary, and failure response. Run a small interrupted-write experiment in a test environment, then reconcile the destination before and after retrying. Include downstream visibility in the success criteria.

Sources

Sources: Microsoft Learn, Sink Transformation in Mapping Data Flow, https://learn.microsoft.com/en-us/azure/data-factory/data-flow-sink ; Microsoft Learn, Copy Activity, https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-overview ; Apache Flink, Fault Tolerance Guarantees, https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/guarantees/ ; Apache Flink, An Overview of End-to-End Exactly-Once Processing, https://flink.apache.org/2018/02/28/an-overview-of-end-to-end-exactly-once-processing-in-apache-flink-with-apache-kafka-too/ . Primary documentation excerpts consulted. Connector versions and configuration affect guarantees; the retry example is original.

Previous: Singularity Next: Situated