Can order effects occur in surveys?

Yes. Earlier questions can influence later interpretation or recall.

Does a break eliminate them?

It may reduce fatigue or temporary carryover, but it may not remove learning or contextual effects.

Is counterbalancing always enough?

No. Its adequacy depends on the types of period and carryover effects and the planned analysis.

AI Glossary / Statistics and probability / Intermediate

Order effect

Definition

An order effect is a change in responses or performance attributable to the sequence in which conditions, tasks, questions, or treatments are presented. It can arise from practice, fatigue, adaptation, context, or effects carried over from an earlier condition. If order is confounded with the condition being studied, an apparent condition difference may partly or entirely reflect the sequence.

What it means

A person testing two interfaces may perform better on the second simply because the task is now familiar. Alternatively, they may perform worse because they are tired. A survey respondent may interpret a general question differently after answering a specific one.
These are not necessarily failures of the person or the instrument. They are features of the measurement situation that the design must address if the goal is to compare conditions fairly.

How it works

Identify what can change as participation progresses. General practice or fatigue may create a period effect that depends on being first, second, or later. Carryover depends on what happened previously, such as learning a strategy in one condition that remains useful in the next.
Avoid assigning every participant the same order when that would confound condition and position. Randomizing order distributes sequence differences probabilistically. Counterbalancing deliberately assigns different orders so conditions appear in specified positions with planned balance.
For two conditions, a common arrangement assigns some participants to A then B and others to B then A. Under appropriate assumptions, this helps separate a common period effect from the condition contrast. It does not automatically remove differential carryover or interactions between condition and period.
With several conditions, complete counterbalancing uses all possible orders and can become expensive. A Latin square balances positions, while balancing which condition precedes another requires additional design properties; ordinary position balance alone does not guarantee carryover balance.
Consider training, rest intervals, alternative task forms, and between-subject designs when appropriate. A pause can reduce fatigue but may not undo learning. A washout period is useful only when the relevant residual effect can plausibly dissipate.
Analyze the design that was actually implemented, preserving participant, period, and sequence information. Report deviations, missing later sessions, and evidence of interactions. An adjustment model cannot reliably recover effects that the design makes indistinguishable.

Three practical examples

Interface testing: alternate whether users try an AI-assisted editor or the standard editor first.
Survey design: examine whether asking about a recent service failure changes answers to a later overall satisfaction question.
Repeated assessments: use suitable task versions and planned ordering when measuring the same participants across several conditions.

Think of it this way

Think of tasting a delicate drink immediately after a strongly flavored one. The second taste depends partly on what came before. Reversing the order for some tasters can reveal the problem, but it does not guarantee that every lingering flavor cancels out.

Worked example

Suppose an original simplified task model gives a baseline score of 50 and adds six points whenever a participant performs the task second. Initially assume A and B have identical direct effects and no carryover.
In sequence AB, A scores 50 and B scores 56, creating an apparent B-minus-A difference of six. In sequence BA, B scores 50 and A scores 56, creating a B-minus-A difference of negative six.
With equal numbers in the two sequences and comparable participants, averaging the two contrasts gives zero, correctly recovering the stipulated absence of a direct difference.
Now suppose B has a true four-point direct advantage while the common second-period bonus remains six. AB gives A = 50 and B = 60, a contrast of ten. BA gives B = 54 and A = 56, a contrast of negative two. Their average is four.
This cancellation depends on the stated additive period effect, balanced sequences, and absence of differential carryover. It is not a general proof that counterbalancing removes every order-related distortion, and the example does not provide a standard error.

Illustrative case study

Fictional case: Willow Research evaluates an AI writing assistant using two editing tasks. An initial pilot always presents the assistant second, and users become faster across sessions.
The team revises the design to balance interface order, checks task comparability, and records session position. It also considers whether skills learned with the assistant could transfer to the standard editor. The report separates observed differences from assumptions about practice and carryover rather than attributing every second-session improvement to AI.

Common misconceptions and mistakes

Do not treat random order as a guarantee of exact balance in a small sample. Randomization and deliberate counterbalancing are related but different design choices.
A period effect is not identical to carryover: the former concerns position or time, while the latter depends on an earlier condition. Their influence can overlap.
Counterbalancing does not necessarily erase irreversible learning, unequal carryover, or selective dropout. If participants who struggle stop before later conditions, simple sequence averages may be misleading.
Also distinguish order effects from order statistics, which are observations sorted by value. The shared word order refers to different concepts.

Business use

Businesses comparing tools, advertisements, prototypes, or training methods should plan presentation order alongside the content being tested. Record sequence and position in the dataset, and use task versions that avoid making one condition systematically easier. When persistent learning makes repeated exposure unsuitable, a between-person comparison may answer the question more cleanly, provided its own design requirements are met.

Check your understanding

Why did the first AB sequence falsely favor B?

Answer: B always received the stipulated second-period practice benefit.

What would invalidate the simple cancellation argument?

Answer: Differential carryover, unequal sequence representation, or other departures from the additive assumptions could do so.

Related terms

Counterbalancing
Carryover effect
Latin square
period effect
crossover design
practice effect
fatigue effect
question order effect

Learn next

list the sequences in a repeated-measures study and check separately whether each condition is balanced by position and by the condition immediately preceding it.

Sources

Sources: David Howell, University of Vermont, Order effect: https://www.uvm.edu/~statdhtx/methods8/Glossary/Glossary.html . Penn State STAT 509, Crossover Designs: https://online.stat.psu.edu/stat509/Lesson12 . Pew Research Center, question-order discussion: https://www.pewresearch.org/writing-survey-questions/ . UVM and Pew text and Penn State search excerpts were checked; the Penn State full page timed out. Numerical scores and fictional case are original.

Previous: Orchestration Next: Order of accuracy