Must high frequency mean milliseconds?

No; the scale is relative to the application.

Are more records always better?

No. Redundancy, noise, and systematic collection errors can limit their value.

Can high frequency data be irregular?

Yes; event-based records commonly have unequal time gaps.

AI Glossary / Statistics and probability / Intermediate

High frequency data

Definition

High frequency data are observations recorded at short time intervals relative to the process being studied. In financial statistics, the term commonly refers to intraday transactions, quotes, or other market events. More broadly, it can describe rapidly sampled sensors and operational logs. There is no universal time threshold: the relevant frequency depends on the question and application.

What it means

A daily total can hide a short interruption or sudden burst of activity. More frequent observations can reveal when changes occur and how a process evolves within a day, minute, or second.
However, a larger number of closely spaced records does not necessarily provide a proportionally larger amount of independent information. Consecutive readings can be highly related, and finer measurement can reveal noise or recording artifacts as well as useful signal.

How it works

Define the observation unit and clock. Data may arrive on a regular grid, such as ten readings per second, or whenever an event occurs, such as each transaction. Event-based timestamps are often irregular, but irregular spacing is not required for high frequency data.
Separate event time from arrival time. A reading may describe an event that occurred earlier than its arrival in a database. Clock offsets, time zones, duplicate messages, corrections, and out-of-order records can distort apparent sequences if they are ignored.
Specify what each value represents. A point measurement, interval average, cumulative counter, event count, and most recent quote require different aggregation rules. Adding point temperatures, averaging cumulative counters, or interpreting no transaction as a zero price produces misleading summaries.
Choose between analyzing event-time data directly and transforming them to a time grid. Resampling requires explicit rules for intervals with several observations and intervals with none. Carrying forward a value or interpolating between readings introduces an assumption; it does not create new measured information.
Account for temporal dependence and recurring patterns. Intraday seasonality, bursts, and common shocks may invalidate methods that assume independent identical observations. Training and evaluation should respect chronology and prevent future information from entering earlier predictions.
In market data, microstructure features such as bid-ask effects and price discreteness can contaminate some estimates at very fine sampling intervals. Primary research shows that suitable sampling frequency depends on the market and estimator. Faster sampling is therefore not automatically statistically better.
For sensors, measurement bandwidth and the collection process also matter. A device that reports frequently may repeat stale values or lack the precision needed to resolve rapid change. Check how readings are generated before interpreting dense timestamps as high-quality information.
Match storage and processing choices to the decision horizon. Retaining event records can support later audits, while carefully defined aggregates may be sufficient for routine planning. Document transformations for reproducibility.

Three practical examples

A production team uses rapid vibration readings to examine brief equipment disturbances.
An AI operations team analyses request-level timestamps to investigate short latency spikes hidden by hourly averages.
A market researcher studies intraday quotes and transactions while accounting for irregular timing and microstructure effects.

Think of it this way

Think of replacing a daily photograph of a busy station with a video. The video reveals motion and brief crowding, but adjacent frames are not independent scenes. A blurred or mistimed video can still mislead despite having many frames.

Worked example

Suppose an illustrative sensor log records a value of 10 at second 0, a value of 20 at second 1, and a value of 10 at second 9. The monitoring interval ends at second 10.
The ordinary mean of the three recorded values is (10 + 20 + 10)/3 = 13.3333. That gives each recorded event equal weight.
Now assume explicitly that each recorded value remains constant until the next update. The value 10 lasts for one second, 20 lasts for eight seconds, and 10 lasts for the final second. The time-weighted mean is (10 x 1 + 20 x 8 + 10 x 1)/10 = 18.
Neither calculation is an arithmetic error: they summarize different things. The event-weighted mean describes the recorded values; the time-weighted mean describes the assumed stepwise trajectory over the interval. If values changed between updates, the second result would depend on an incorrect assumption.
A 10-hertz sensor produces 600 readings in a 60-second interval when endpoint conventions give ten readings per second. This greater record count does not tell us how many effectively independent observations it contains.

Illustrative case study

Fictional case: Aspen Systems investigates complaints about brief delays in an AI assistant. Its hourly dashboard looks normal, so engineers inspect request-level logs with event and receipt timestamps.
They find that one logging service batches messages, making arrival times appear more concentrated than the underlying requests. After correcting the time interpretation, the team identifies a genuine short congestion episode and computes interval metrics with documented rules for missing and delayed records.
The evaluation separates later days from training periods and compares both typical latency and high-percentile delays. Aspen improves monitoring while retaining enough raw records to audit unusual episodes. It does not assume that a denser dashboard alone improves service quality.

Common misconceptions and mistakes

Common mistakes include equating high frequency with high-frequency trading, assuming observations are evenly spaced, and treating every record as independent. A high sampling rate is also different from low delivery latency: historical data can be recorded frequently but arrive much later.
Interpolation, duplicated values, and timestamp precision should not be mistaken for genuine measurement resolution. Aggregating too early can conceal short events, while keeping every record without checking quality can preserve artifacts.

Business use

High frequency data can improve incident diagnosis, process monitoring, and short-horizon forecasting. Their value depends on reliable timing, meaningful measurement, and analysis suited to dependence and noise. Business teams should justify the chosen resolution by the decisions it supports and measure whether added detail changes those decisions usefully.

Check your understanding

Why is the time-weighted mean 18 in the example?

Answer: The value 20 occupies eight of the ten seconds under the stated assumption.

Does interpolation add observations actually measured by the sensor?

Answer: No.

What must be checked before ordering events?

Answer: Timestamp meaning, clock alignment, and possible delivery delays.

Related terms

Time series
Autocorrelation
sampling frequency
irregular sampling
timestamp
temporal aggregation
market microstructure noise
data latency
event stream

Learn next

Compare event-weighted and time-weighted summaries, then study resampling and time-series validation. Practice documenting what happens to duplicates, missing intervals, and late-arriving records.

Time series

Sources

BIS, minutely-data glossary: https://data.bis.org/help/glossary?item=Minutely Chaboud and colleagues, Frequency of observation and the estimation of integrated volatility in deep and liquid financial markets: https://www.bis.org/publ/work249.htm BIS-hosted applied research on high frequency financial datasets: https://www.bis.org/ifc/events/ifc_8thconf/ifc_8thconf_54pap.pdf

Previous: High-dimensional statistics Next: High-level action