Is method error always systematic?

No; method-related effects can include systematic and variable components.

Does agreement between two methods prove accuracy?

No; both may share a similar discrepancy.

Can changing a questionnaire create a trend break?

Yes; a changed measurement process can affect comparability even if the underlying phenomenon is unchanged.

AI Glossary / Statistics and probability / Intermediate

Method error

Definition

Method error is the component of measurement error attributable to how a measurement is obtained. It can arise from the procedure, collection mode, preparation, presentation, or other features of the method. In surveys, for example, the same intended question may produce different responses when asked face to face, by telephone, or through a self-completed form. A method difference requires investigation before either method is judged accurate.

What it means

A method is part of the measurement system, not merely a neutral route to an already determined number. How a question is presented can affect interpretation or willingness to disclose. How an object is prepared or positioned can affect an instrument reading.
The term refers to a source of discrepancy, not necessarily to a careless mistake or an incorrect mathematical algorithm. Some effects are systematic and repeatable; others vary with the item, participant, operator, or environment. The important question is whether the method changes the recorded result relative to the intended measurement target.

How it works

Define the target and reference conditions first. Two procedures may appear to disagree because they measure different things, such as immediate experience versus remembered experience. Clarify that distinction before attributing disagreement to error.
Identify procedural features that could influence results. Examples include question wording, visual versus spoken presentation, response options, sample preparation, timing, and order of measurement. Document which features differ between methods.
Use a comparison design that separates method effects from other differences where possible. Measuring stable reference items with both procedures can reveal discrepancies relative to a reference. Random allocation of comparable participants to survey methods can help distinguish method effects from differences in who chooses each mode.
Paired measurements can be informative, but order effects, learning, fatigue, and changes in the measured item may complicate them. Randomized or balanced ordering and an appropriate interval between measurements can help when justified by the task.
Estimate the size and variability of differences, examine whether they depend on measurement level or subgroup, and consider uncertainty. A single overall offset may not adequately describe a method that behaves differently at low and high values. Revise the method and evaluate the revision before assuming that documentation alone has removed the problem.

Three practical examples

Customer research: a company tests whether a satisfaction question produces different responses when an interviewer asks it versus when customers complete it privately.
Automated inspection: engineers compare two preparation procedures for reference samples used to assess an AI-based measurement system.
Personal tracking: someone notices that recalling activity at the end of a week differs from recording it daily and investigates the role of the collection method.

Think of it this way

Think of photographing the same object under two lighting arrangements. A consistent difference in the pictures may come from the lighting rather than the object. To understand the discrepancy, compare the methods under controlled conditions and decide which image properties match the intended purpose.

Worked example

Four stable reference items have assigned values 10, 20, 30, and 40 units. Method A returns 11, 21, 31, and 41. Method B returns 10, 19, 30, and 39. Treat the assigned reference values as the comparison targets for this arithmetic example.
Method A has errors +1, +1, +1, and +1, giving a mean error of +1 unit. Method B has errors 0, -1, 0, and -1, giving a mean error of -0.5 units. The paired differences A minus B are 1, 2, 1, and 2, with mean 1.5 units.
If only the A-minus-B differences were available, the mean difference of 1.5 would show disagreement but would not tell us which method was closer to the reference. Here the reference measurements allow separate comparisons. Method A is consistently high, while Method B is sometimes low.
Four reference items do not establish behavior over an entire operating range. Actual validation would also consider reference uncertainty, repeat measurements, relevant conditions, and whether the methods alter the items. Subtracting the observed mean offset is only justified if an additive correction is appropriate for the intended use.

Illustrative case study

Fictional case: Aspen Feedback changes from telephone interviews to an online questionnaire. The reported satisfaction score changes, but the customer team cannot tell whether customer experience, respondent composition, or the collection method caused the change.
It runs a planned overlap study, checks response patterns and question interpretation, and documents the break in the historical series. It avoids attributing the entire shift to product improvement. This fictional scenario illustrates investigation of method-related differences and makes no claim about the accuracy of either survey mode.

Common misconceptions and mistakes

Do not interpret any difference between methods as proof that one is wrong. A suitable reference or a defensible validation design is needed to assess accuracy. Do not assume that the cheaper, newer, or automated method is intrinsically less biased.
In surveys, mode differences can combine measurement effects with coverage and nonresponse differences. Comparing two self-selected respondent groups does not cleanly isolate how the mode changes an individual's answer. A census can still contain method error even though sampling error is absent.
Do not treat more observations as an automatic cure for a systematic procedural effect. A large study may estimate a biased quantity very precisely. Also avoid applying one correction across contexts without checking whether the method effect is stable.

Business use

Method-error analysis matters when changing data collection, merging historical sources, or creating reference labels for AI evaluation. A model trained on one collection process may learn artifacts that fail when the process changes.
Preserve procedure versions, collection modes, timing, and relevant operator or environment information. Before interpreting a trend, assess whether a method change could explain part of it. Report method-comparison evidence separately from sampling precision, and use bridge studies where continuity of a series is important.

Check your understanding

two methods differ by two units on average. Do we know which is accurate?

Answer: not without additional reference or validation evidence.

will doubling the sample necessarily remove a persistent method offset?

Answer: no.

why record a data-collection mode?

Answer: it helps investigate whether procedural differences affect measured results and their comparability.

Related terms

Measurement error
Calibration
Nonresponse error
Mode effect
Systematic error
Questionnaire design
Measurement invariance
Method comparison

Learn next

Next, distinguish method effects from differences in sample composition. Then study paired method-comparison designs, randomized split-panel surveys, and the evidence needed to bridge a changed measurement process.

Sources

NIST/SEMATECH, glossary, method error: https://www.itl.nist.gov/div898/handbook/glossary.htm U.S. Census Bureau, Questionnaire Testing and Evaluation Methods: https://www.census.gov/about/policies/quality/standards/appendixa2.html

Previous: METEOR Next: Method of moving asymptotes