Must a trajectory reach a terminal state?

No. It can be a segment or a finite recording from a continuing task.

Can two trajectories contain the same states but different returns?

Yes. Actions, timing, rewards and their order can differ.

Is a rollout the same thing?

Rollout often refers to generating a trajectory by executing or simulating a policy; usage varies by context.

AI Glossary / Reinforcement learning and robotics / Advanced

Trajectory

Definition

In reinforcement learning, a trajectory is an ordered sequence of interactions between an agent and its environment. It records how states or observations, actions and rewards unfold over time. A trajectory may describe a complete episode or only a segment of experience, depending on the task and recording convention.

What it means

A trajectory describes what happened during one course of interaction. A policy describes how actions are selected. The same policy can generate different trajectories because action selection, initial conditions or environment transitions may be random.
The term also appears in robotics, where it can mean a motion specified over time. In this reinforcement-learning entry, the emphasis is the sequence of experience used for learning and evaluation. It is more informative than a geometric path because actions and rewards can matter even when physical positions do not change.

How it works

A common record stores transitions as (state, action, reward, next state), with termination information. Using one standard timing convention, action A_t is taken in state S_t, then reward R_(t+1) and next state S_(t+1) are observed. Consecutive transitions connect through their shared state.
In a partially observed environment, the available record may contain observations instead of the full underlying state. Episode identifiers, timestamps, policy versions and action probabilities can also matter for interpreting collected experience. A time-limit truncation should be distinguished from a genuine terminal event where the task ends.
Learning algorithms use trajectory information differently. Some use complete returns, some bootstrap from later value estimates, and others sample individual transitions from a replay buffer. Shuffling records does not erase their origin, but it can remove the ordering needed by sequence-dependent calculations unless that structure is retained separately.

Three practical examples

Simulated delivery: an agent's record includes each decision to move or wait, the resulting location and the reward received. Delays may matter even when the spatial route is unchanged.
Policy evaluation: researchers compare returns across many trajectories from the same policy and report variation rather than selecting one favorable run.
Learning from logs: a team uses recorded experience while retaining information about the policy that generated it. Data from an earlier behavior policy is not automatically equivalent to fresh experience from a new one.

Think of it this way

Imagine a travel diary that records where you were, what you decided, what happened next and the outcome of that decision. A map of the route alone omits waiting, detours and costs. The diary is one realized journey; the travel strategy that produced it could lead to a different journey another day.

Worked example

Consider a three-step episode with states: start, corridor, loading point and terminal destination. The agent takes actions move, collect and deliver. The rewards received after those actions are -1, -1 and 10. The transition record is (start, move, -1, corridor), then (corridor, collect, -1, loading point), then (loading point, deliver, 10, terminal destination).
With discount factor gamma = 0.9, the return from the beginning is G_0 = -1 + 0.9 times (-1) + 0.9 squared times 10 = 6.2. From the corridor, G_1 = -1 + 0.9 times 10 = 8. From the loading point, G_2 = 10. The recursion G_0 = R_1 + gamma G_1 gives -1 + 0.9 times 8 = 6.2, confirming the calculation.
The undiscounted reward total is -1 -1 +10 = 8. It differs from the discounted return because the final reward receives less weight when evaluated from the start. Reordering the same three rewards to 10, -1, -1 gives discounted return 10 -0.9 -0.81 = 8.29. The unordered collection of rewards therefore does not determine the discounted return.
Now suppose logging stops after the first two transitions because a recording budget expires, while the environment itself has not terminated. The observed discounted prefix is -1 -0.9 = -1.9. A two-step bootstrapped estimate adds gamma squared times an estimated value of the loading-point state. If that value estimate is 7, the result is -1.9 + 0.81 times 7 = 3.77. That estimate is not the completed episode's realized return of 6.2. This distinguishes truncation, estimated value and observed return; no policy training is claimed.

Illustrative case study

Fictional company Maple Simulation compares two policies for a simplified warehouse environment. Its analysts store complete episode identifiers and distinguish delivery completion from simulator time limits. They evaluate both policies across a range of starting conditions rather than comparing only their best demonstrations.
When a reported return changes after a logging update, the team checks reward timing and truncation handling before concluding that the policy improved. It also keeps the policy version attached to each record. This is a fictional diagnostic workflow and does not report a real warehouse deployment or measured productivity gain.

Common misconceptions and mistakes

A trajectory is not the policy itself, and one high-return trajectory does not establish high expected policy performance. A sequence of observed images is not necessarily a sequence of complete Markov states. Hidden conditions can affect what happens next.
Another mistake is treating every stopped recording as a terminal episode. Depending on the task definition, truncation may leave future value to estimate. Mixing reward-index conventions can also shift discounts by one step. Finally, a trajectory collected under one policy does not directly reveal what would have happened under a different action sequence.

Business use

Trajectory records help teams trace decisions and outcomes in sequential systems. They can support debugging, evaluation and learning, provided the records preserve the information required by the method. Useful logging connects actions to subsequent observations and rewards, identifies episode boundaries and records the policy or controller involved. More logged transitions do not automatically mean broader coverage of relevant situations.

Check your understanding

Why did reordering the example rewards change the return?

Answer: Discounting weights rewards according to when they occur.

Why was 3.77 an estimate rather than the realized complete return?

Answer: It used an estimated continuation value after a truncated prefix.

Why keep the behavior policy version?

Answer: It helps identify how the recorded actions were selected and whether the data suits the intended analysis.

Related terms

Policy
Episode
Reward
Return
Replay buffer
State transition
Rollout
Partial observability

Learn next

Write a short transition sequence, calculate returns backward and distinguish terminal events from recording limits. Then compare algorithms that use full returns with those that bootstrap.

Sources

Google Machine Learning Glossary, trajectory, describes the reinforcement-learning transition-sequence meaning: https://developers.google.com/machine-learning/glossary#trajectory . The episode calculations and fictional case are original teaching illustrations.

Previous: Training set Next: Trajectory optimization