Does loading always increase the number of target records?

No. It can update or replace records without increasing the count.

Is it safe to retry every failed load?

Not automatically. The retry behavior must avoid unintended repeated effects.

Does a matching row count prove a correct load?

No. Keys, values, scope and rejected inputs may still need verification.

AI Glossary / Data engineering and architecture / Advanced

Data loading

Definition

Data loading is the process of writing incoming data into a target system so it can be stored and used there. The destination may be a database, warehouse, lake or another repository. Loading includes deciding how incoming records relate to existing target records. It is distinct from extracting data from a source, even when the same pipeline performs extraction, transformation and loading together.

What it means

A file arriving at a destination does not necessarily mean its contents have been incorporated into the target dataset. The load may need to interpret fields, enforce constraints and decide whether each incoming record creates, updates or replaces information. A process can also reject some inputs or fail before committing its changes. For an AI workflow, distinguish data received, data successfully stored and data ready for the next operation. Those stages can have different completion conditions and different evidence.

How it works

The pipeline maps incoming values to the target structure and applies a defined loading strategy. It may append new rows, replace a defined dataset or merge changes using identifiers and update rules. The implementation determines transaction boundaries and failure behavior. PostgreSQL documents bulk loading with COPY and explains how transactions can prevent partially committed groups of inserts. Those are concrete database mechanisms, not universal defaults. A complete loading design also specifies how retries, rejected records and pre-existing data are handled so repeated runs preserve the intended result.

Three practical examples

A warehouse receives a new batch of sales records and adds only the records defined as new by the pipeline's key rules.
A reference-data process replaces a controlled target snapshot after checking the incoming dataset.
An AI document repository records successful imports separately from rejected files so its coverage report reflects the material actually available.

Think of it this way

Imagine a store receiving a delivery. The boxes reaching the loading bay is different from their contents being counted and placed in the correct stock locations. Some items may replace damaged stock, while others increase the inventory. Receiving the same delivery note twice should not automatically double the inventory. Data loading has similar decisions about acceptance, placement and repetition, although the actual rules depend on the system and business process rather than the physical analogy.

Worked example

Consider an original target table with 100 records identified by unique keys. An incoming batch contains ten records with previously unseen keys and five updates to five existing keys. Under a stipulated merge rule, the result contains 110 records: ten are added and five existing records change. It does not contain 115 records merely because fifteen inputs were processed. If the identical batch is safely reapplied under the same key-based replacement rules, the count remains 110. This assumes deterministic updates and no additive side effects. A different operation, such as incrementing a counter each time, would need additional safeguards to avoid applying the effect twice.

Illustrative case study

A fictional reporting service loses its connection after sending a batch and cannot immediately tell whether the target committed it. A blind append retry creates duplicate business records in a test environment. The team defines a batch identity and an appropriate record-matching strategy, then tests the uncertain-completion case again. It also records which inputs were rejected. The scenario illustrates why loading needs more than a successful write attempt. It does not claim that every merge operation is automatically idempotent or that a row count alone proves all intended values were loaded correctly.

Common misconceptions and mistakes

Loading is not necessarily appending. A load may update or replace existing information according to explicit rules. Another mistake is equating processed input rows with newly created target rows. A process may report inputs read even when some are rejected or rolled back. Faster bulk loading also does not remove the need for validation and recovery behavior. Finally, retrying is not inherently safe: whether it repeats an effect depends on the operation, identifiers and transaction design. The target's final state must be checked against the intended result.

Business use

The concept helps teams define when a pipeline stage is complete. Specify what should be added, updated or replaced and how errors affect the overall result. Track accepted, rejected and committed records using terms whose meanings are clear. Test repeated batches and interruptions, not only a successful first run. For AI projects, connect loading evidence to the material that downstream processing can actually access. A useful completion report explains the target state and remaining exceptions rather than merely announcing that a transfer or job ended.

Check your understanding

How many records were in the toy target after the merge?

Answer: One hundred and ten.

Why not one hundred and fifteen?

Answer: Five incoming records updated existing keys.

Why was replay safe only under stated assumptions?

Answer: Replacing values deterministically differs from applying an additive effect each time the batch is processed.

Related terms

ETL
ELT
bulk import
upsert
transaction
idempotency

Learn next

Design a small loading example with new keys, existing keys and one invalid record. State the expected target before running it, then consider a repeated batch and an interruption. Explain what evidence would distinguish a fully committed result from partial progress or an uncertain outcome.

Sources

PostgreSQL Global Development Group, Populating a Database: https://www.postgresql.org/docs/current/populate.html The 100-record merge example, delivery analogy and reporting-service case are original educational illustrations. The reference supplies concrete bulk-loading and transaction context; the example does not claim that COPY itself performs the stipulated merge or deduplication behavior.

Previous: Data lifecycle Next: Data management plan