Does a zero assignment difference prove the tool never helps?

No; the question and uncertainty must be considered.

AI Glossary / Statistics and probability / Advanced

Intention-to-treat

Definition

Intention-to-treat is the principle of analyzing participants according to their original randomized assignment, including those who did not follow the assigned intervention. It preserves the comparison created by randomization. The resulting question concerns assignment to an intervention strategy, rather than automatically identifying the effect of actually receiving it or adhering perfectly.

What it means

Suppose employees are randomly offered an optional AI assistant. Some offered the assistant never use it, while some controls obtain access elsewhere. An intention-to-treat comparison keeps employees in their original groups. Moving people between groups based on subsequent behavior can introduce differences in motivation, workload, or ability.
The principle does not make unavailable outcomes observable. A report must explain missing measurements and the assumptions used to address them, rather than treating the intention-to-treat label as a complete analysis plan.

How it works

Define the assignment, outcome, measurement time, and target effect before examining results. Retain each person's randomized group in the analysis dataset and seek outcome measurements even if the person stops the intervention, where continued follow-up is permitted.
Choose an analysis appropriate to the outcome and randomization scheme. For a simple binary outcome with complete follow-up, compare outcome proportions using all randomized participants in their assigned groups. Other designs may require regression adjustment or methods accounting for group randomization.
Record uptake, switching, withdrawals, and missing outcomes separately. Missing-data methods require stated assumptions and sensitivity analysis. Neither dropping every incomplete case nor replacing every missing value with failure is universally justified. Describe the actual analysis population and denominators explicitly.

Three practical examples

A randomized software rollout compares employees offered a new assistant with employees assigned the existing workflow. The assignment comparison includes employees who decline the offer.
A trial of training invitations measures completion of a later assessment. People who skip training remain in their invitation groups when assessing the invitation strategy.
A randomized reminder experiment compares response rates by original message assignment. Participants who ignore the message are part of the strategy's observed performance, rather than automatic exclusions.

Think of it this way

Keep the original team sheets when comparing two assigned strategies. Rewriting the teams afterward according to who participated enthusiastically changes the comparison and can remove the protection that random assignment provided.

Worked example

Consider an invented experiment assigning one hundred employees to an assistant offer and one hundred to the existing workflow. The outcome is successful completion of a specified task by a fixed deadline. Assume every employee's outcome is observed.
Among the one hundred offered the assistant, sixty use it: forty-eight succeed and twelve do not. Forty decline it: twelve succeed and twenty-eight do not. The offer group's total is therefore sixty successes out of one hundred, or 60 percent.
Among the one hundred controls, twenty obtain the assistant elsewhere: sixteen succeed and four do not. Eighty do not use it: forty-four succeed and thirty-six do not. The control group's total is also sixty successes out of one hundred, or 60 percent.
The intention-to-treat difference is 60 percent minus 60 percent = zero percentage points. This is the observed assignment contrast; its uncertainty still needs assessment. A zero estimate is not proof that the true effect is exactly zero.
Now regroup people by actual use. There are eighty users across the two assignment groups, with forty-eight plus sixteen = sixty-four successes. Their success proportion is 64/80 = 80 percent. The 120 nonusers have twelve plus forty-four = fifty-six successes, giving 56/120 = 46.67 percent. The as-treated difference is about 33.33 percentage points.
That larger contrast does not demonstrate a causal benefit from use. Actual users were not randomized as users. The constructed table could reflect differences affecting both uptake and success; the aggregate counts alone cannot resolve those explanations.
A simple comparison restricted to participants who followed assignment also changes the groups. The sixty offer-group users have 80 percent success, while the eighty controls who did not use the assistant have 55 percent success. Their difference is twenty-five percentage points. Merely selecting these adherent participants does not establish a causal effect of perfect adherence.
Finally, consider a separate missing-outcome version of the experiment. Each assigned group still contains one hundred employees, but ten outcomes per group are unavailable. Suppose fifty-four successes are observed among ninety measured offer-group participants, compared with forty-five among ninety measured controls. The available-case difference is 60 percent minus 50 percent = ten percentage points.
If all ten missing offer-group outcomes are failures and all ten missing control outcomes are successes, the full-group difference would be 54 percent minus 55 percent = minus one percentage point. Reversing those assumptions produces 64 percent minus 45 percent = nineteen percentage points. These extreme bounds illustrate missing-outcome uncertainty; they are not confidence intervals or recommended imputations.

Illustrative case study

Fictional company Birch Assistance evaluates an optional drafting tool through randomized invitations. Its analysts keep invitation assignment separate from actual usage and task completion. They continue collecting permitted outcome data from people who stop using the tool.
The report presents the invitation comparison first, followed by uptake and explicitly qualified supplementary analyses. Missing outcomes receive a separate sensitivity assessment. Managers can then distinguish evidence about offering the tool from unanswered questions about sustained use. This case is fictional.

Common misconceptions and mistakes

Intention-to-treat does not mean everyone received treatment. It describes how randomized groups are retained in analysis.
It does not guarantee baseline balance or eliminate every bias.
An intention-to-treat estimate is not automatically conservative. The direction of differences caused by nonadherence depends on the circumstances.
A modified intention-to-treat label can conceal exclusions. Readers need the actual rules, counts, and reasons.
Simple adherer-only comparisons can be biased; causal per-protocol estimation requires more than deleting nonadherent participants.

Business use

Use assignment-based analysis to evaluate an offer or rollout strategy under the conditions studied. Report uptake alongside the outcome contrast so decision-makers understand implementation. Keep causal claims about actual usage distinct, and avoid generalizing automatically to different workforces or deployment conditions.

Check your understanding

What is the complete-data assignment difference?

Answer: Zero percentage points.

Why is the user comparison different?

Answer: It groups people by subsequent behavior.

Are the missing-data bounds confidence intervals?

Answer: No; they vary assumptions about unobserved binary outcomes.

Related terms

Estimand
Missing data
Sensitivity analysis
Randomization
adherence
as-treated analysis
per-protocol analysis
treatment assignment

Learn next

Study causal effects of assignment versus receipt, methods for missing outcomes, and transparent reporting of participant flow.

Sources

Harrell, Glossary of Statistical Terms: https://hbiostat.org/glossary/ . White and colleagues, Strategy for intention to treat analysis with missing outcome data: https://www.bmj.com/content/342/bmj.d40 . CONSORT 2025 explanation: https://www.bmj.com/content/389/bmj-2024-081124 . Numerical tables and company scenario are original illustrations.

Previous: Intensive sampling Next: Intentional stance