Must every protocol be public?

No; disclosure depends on the context, although accessible plans can support scrutiny and reuse.

Can a protocol be revised?

Yes, with appropriate versioning, explanation and any required review.

Does preregistration forbid exploration?

No; additional analyses should be distinguished from the prespecified work.

AI Glossary / Statistics and probability / Advanced

Scientific protocol

Definition

A scientific protocol is a documented plan describing how a study or experiment will be carried out. It connects the research question with the design, materials, procedures, measurements and analysis, so that people can understand what was intended and assess what actually happened.

What it means

A protocol turns a broad intention such as “compare two AI systems” into decisions that another researcher can inspect. Which systems and versions will be compared? On which tasks? What counts as success? What happens when a system fails to respond?

The document supports consistent execution and transparent interpretation. It is not evidence that the design is valid simply because it is detailed. A carefully recorded experiment can still measure the wrong outcome or use an unsuitable sample.

How it works

Start with the question and the quantity the study aims to estimate. Specify the study units, selection criteria, procedures, measurements and analysis. Include the resources and responsibilities needed to execute the plan. Detail should match the study: a laboratory procedure and a human-participant study have different requirements.

Give the protocol a version and date. Preserve amendments and explain when and why changes occurred. Distinguish the original plan from actual execution and from analyses developed after seeing results.

Preregistration adds a time-stamped registration of the plan before the relevant research or analysis. A protocol can exist without preregistration, and registration alone does not establish methodological quality.

Three practical examples

1. AI benchmark: specify model versions, prompts, task selection, scoring criteria, repeat runs and treatment of failed responses before comparing performance.

2. Sensor experiment: record calibration, measurement timing, environmental conditions and data-processing steps so that another team can examine the procedure.

3. Workplace study: define participant selection, allocation, outcome measurement and analysis, together with the applicable arrangements for consent, privacy and review.

Think of it this way

Think of a protocol as a study's operating plan and decision record. It tells the team what to do and lets a later reader distinguish a planned choice from a response to an unexpected result. It cannot guarantee that the underlying question or design is sound.

Worked example

Consider a fictional, miniature protocol for comparing systems A and B on twenty held-out tasks. The teaching protocol fixes both model versions, one prompt per task, one attempt per system and a thirty-second response limit. Both systems receive the same tasks. A fixed rubric determines whether an answer passes.

Before running the comparison, the protocol defines the primary outcome as the proportion of all twenty assigned tasks passed. A timeout counts as a nonpass. The primary comparison is A's pass proportion minus B's pass proportion. These choices answer a reliability question that includes failure to deliver an answer within the allowed time.

Suppose both systems pass twelve tasks. A alone passes four, B alone passes one, and neither passes three. The four counts sum to twenty. A passes 12 + 4 = 16 tasks, while B passes 12 + 1 = 13.

The planned proportions are therefore 16/20 = 80% and 13/20 = 65%. Their difference is fifteen percentage points. Equivalently, the paired difference is (4 - 1)/20 = 0.15; tasks passed by both systems contribute no difference.

Now suppose two of B's nonpasses were timeouts, both on tasks A passed. An analyst proposes dropping B's timeouts from its denominator. That would produce 13/18, approximately 72.22%, instead of 65%.

Comparing A's original 80% with that figure gives approximately 7.78 percentage points. The arithmetic is valid, but the comparison mixes denominators and changes the question. It no longer applies the prespecified all-assigned-task rule equally.

A secondary analysis of performance conditional on returning an answer may be useful if clearly defined and labeled. It must not silently replace the primary outcome. Nor should the initial fifteen-point difference be described as statistically established superiority: this example has not calculated an uncertainty interval or conducted a hypothesis test.

A complete study protocol would also justify task selection and sample size, specify an analysis that respects paired tasks, address uncertainty and define any multiplicity handling. Twenty tasks were chosen here for transparent teaching arithmetic, not as a general recommendation for adequate evaluation.

Illustrative case study

Fictional Cedar Research notices the denominator discrepancy during review. The team retains the planned all-task result and presents any additional analysis under a separate, accurate description.

For its next study, Cedar adds clearer timeout logging and a worked scoring example to the protocol. It records these as a new version rather than rewriting the original plan as though the extra detail had always existed. The change improves traceability; it does not retrospectively change which choices were specified before the first results. This invented case illustrates documentation, not a real product comparison.

Common misconceptions and mistakes

“A protocol is just a list of steps.” A study plan also connects its procedures with the research question and analysis.

“Writing it down proves the study is unbiased.” Design, measurement and execution still require scrutiny.

“Any change makes the study worthless.” Changes may be necessary; timing, reasons and consequences should be transparent.

“A protocol and preregistration are identical.” One is the plan; the other records a plan through a registration process.

Business use

Use a protocol when results will inform consequential product, process or investment decisions. Ask whether a different analyst could identify the same eligible records, apply the same scoring rules and calculate the same intended comparison.

For AI projects, preserve relevant prompts, system settings, dataset versions and failure rules. Link the protocol to the actual execution record. Separate deviations, corrections and exploratory analyses so readers can judge what the evidence supports. A template helps organization, but copying headings without resolving the underlying decisions provides little protection against ambiguity.

Check your understanding

Why is B's planned denominator twenty?

Answer: The stated outcome includes every assigned task, including timeouts.

Why does 13/18 answer a different question?

Answer: It conditions the denominator on B returning an answer.

What should accompany an amendment?

Answer: Its date, rationale and relationship to the original plan and available results.

Related terms

Experimental design
preregistration
reproducibility
research question
outcome measure
analysis plan
protocol deviation
version control

Learn next

Write a one-page protocol for a small comparison. Ask another person to identify an ambiguous eligibility, scoring or failure rule before any results are collected.

Sources

NIH Office of Human Subjects Research Protections, How to write a protocol 101, study objectives and procedures; clinical context: https://irbo.nih.gov/documents/429/How_to_write_a_protocol_101_1_2.pdf

Center for Open Science, Preregistration, planned analyses and transparent changes: https://www.cos.io/initiatives/prereg

Previous: Schwa Next: Score-based generative model