Must a data stream be infinite?

No. Some streams are bounded; ongoing event sources are often modeled as unbounded because their eventual end is not known.

Can a stream be stored?

Yes. Retained event logs can support replay and historical processing, subject to their configured retention.

Is arrival time always the event time?

No. A record can arrive long after the event occurred, especially after network or source delays.

AI Glossary / Data engineering and architecture / Beginner

Data stream

Definition

A data stream is a sequence of data items made available progressively over time. In data engineering, those items commonly represent events, observations or changes, such as sensor readings, application actions or transaction updates. A stream may continue indefinitely, allowing processing to begin before the complete sequence is available.

What it means

The defining idea is progressive arrival, rather than a particular file format or speed. A stream can carry structured records, messages or other data. It may arrive steadily, in bursts or after delays. A finite file can also be read as a stream, although operational event streams are often treated as unbounded.

Streaming does not mean information must disappear after being read. Systems can retain events for replay or historical analysis. Nor does it guarantee instantaneous results. The time between an event occurring and a useful result becoming available depends on capture, transport, processing and output behavior.

How it works

A producer creates or publishes data items. Consumers read them and perform work such as filtering, transformation or aggregation. An intermediary may buffer and retain items so producers and consumers can operate at different rates.

Records often include an identifier, a timestamp and a payload describing what happened. Event time refers to when the event occurred according to its recorded timestamp; processing time refers to when a computation handles it. Network delays can make these different and cause records to arrive out of event-time order.

Processing an ongoing sequence often requires keeping state, such as a running total or recent observations. Windows define subsets, for example events from a particular minute. The design must explain how it treats duplicates, missing events, late arrivals and ordering. A platform's ordering guarantee may apply within a partition rather than across every event in the entire system.

Three practical examples

1. Equipment monitoring: sensors produce measurements as machinery operates. A processor calculates recent averages and identifies unusual patterns without waiting for a day's data to become a finished file.

2. Website operations: an application publishes interaction events. Separate consumers update operational dashboards and prepare data for later analysis, using documented definitions and access rules.

3. AI features: an online service updates recent activity features from incoming events. It checks that the feature's time window and event interpretation match the assumptions used during model development.

Think of it this way

Think of a conveyor carrying labeled parcels. Workers can inspect parcels as they arrive instead of waiting for the entire shipment. A queue can absorb a short rush, while labels help identify items and their intended destination.

The analogy explains progressive processing and buffering. Digital streams can also be replayed, partitioned or copied to several consumers, and arrival order may differ from the order in which the underlying events occurred.

Worked example

Suppose 100 sensors each publish one 200-byte record every second. The stream contains 100 records per second and 20,000 bytes per second of payload. Over one hour, that is 360,000 records and 72,000,000 bytes, or 72 megabytes using decimal units.

Now suppose a consumer handles only 80 records per second for five minutes while arrivals remain at 100. The backlog grows by 20 × 300 = 6,000 records. If the consumer then handles 120 records per second, it has only 20 records per second of spare capacity while new records continue arriving.

Clearing the backlog therefore takes another 300 seconds under these assumptions. Buffering prevents immediate loss only within its capacity and retention limits; it does not remove a sustained processing shortfall. Headers, replication and other overhead are excluded from the byte calculation.

Illustrative case study

Fictional case: StreamOrchard tracks warehouse scanner events. A dashboard groups events by the time they arrive at the server. After a network interruption, queued scans arrive together and appear as a sudden burst of warehouse activity.

The team distinguishes scanner event timestamps from server processing timestamps. For activity reporting, it groups by the intended event-time definition and establishes how late records revise earlier totals. It keeps arrival timing for diagnosing network behavior, where it answers a different question.

The case illustrates the importance of time semantics. It does not imply that event timestamps are always correct; clock quality and timestamp generation still require attention.

Common misconceptions and mistakes

Do not assume a stream is perfectly ordered, complete or duplicate-free. Those properties depend on the source, transport and processing design.

Do not use “real time” as a substitute for a latency requirement. Specify how quickly a result must arrive and what delay is tolerable.

Do not confuse replay with a new real-world event. A consumer may see an old record again during recovery or reprocessing, so downstream effects need an appropriate handling strategy.

Business use

Streams support decisions that benefit from fresh information, such as operational monitoring and rapidly changing application state. They also introduce responsibilities for continuous operation, backlog monitoring and recovery.

Teams should define event meaning, identifiers, timestamp rules and schema evolution before connecting many consumers. For AI applications, distinguish a stream of inputs used for inference from online learning that changes model parameters. Receiving streaming data does not itself mean the model learns continuously. Measure data freshness and downstream correctness alongside records processed per second.

Check your understanding

Why does a consumer running at 120 records per second take five minutes to clear the example's backlog?

Answer: New records still arrive at 100 per second. Only the remaining capacity of 20 per second reduces the 6,000-record backlog.

Related terms

Event
Stream processing
Data stream management system
producer
consumer
message broker
event time
processing time
windowing
backpressure
replay

Learn next

Learn about event time and windowing next. Take a small invented sequence with timestamps and rearrange its arrival order. Compare totals grouped by occurrence time with totals grouped by processing time, then decide how a late record should affect an already displayed result.

Event

Sources

Apache Kafka, event streams, producers, consumers, retention and partition ordering: https://kafka.apache.org/intro/

Apache Flink, event time, processing time and streaming analytics: https://nightlies.apache.org/flink/flink-docs-stable/docs/learn-flink/streaming_analytics/

Primary documentation consulted. The sensor rates, backlog calculation and fictional scanner scenario are original illustrations; implementation guarantees require system-specific documentation.

Previous: Data store Next: Data stream management system