Is activity bias always caused deliberately?

No. It can emerge naturally from collecting whatever interactions a platform produces.

Does one record per user solve it?

It can reduce unequal contribution, but does not automatically address missing users, selection into the platform or measurement problems.

Are event-level averages wrong?

Not inherently. They can answer event-level questions; the problem is interpreting them as a different quantity without justification.

AI Glossary / Governance, ethics and safety / Advanced

Activity Bias

Definition

Activity bias occurs when data or conclusions disproportionately reflect highly active users while less active or inactive users are poorly represented. In AI, frequent contributors can supply a large share of training examples, feedback or evaluation records. A model or report based on those records may consequently describe active participants better than the wider population it is intended to serve.

What it means

A platform can collect millions of interactions without hearing equally from the people who use it. One person who posts every day contributes far more records than someone who visits occasionally. Someone who stopped using the service may contribute none during the collection period.
This becomes a bias problem when the data are used to answer a question about a different target population or unit. The average interaction and the average user are not automatically the same thing.
The term here concerns unequal representation associated with user activity. It should not be confused with action bias, the tendency to favor doing something over inaction.

How it works

The mechanism begins with how records enter the dataset. More activity creates more opportunities to be observed, rated or selected. If activity relates to preferences, skills or satisfaction, the resulting record distribution can differ from the intended user distribution.
Repeated observations also do not create the same information as observations from new people. Treating many records from one user as independent evidence about the population can give a misleading impression of coverage.
Possible responses include reporting activity groups separately, limiting contributions per person, sampling users rather than events, or using justified weights. These choices change the question being estimated. They require an explicit target and cannot manufacture reliable information about people who supplied no relevant data.

Three practical examples

A recommendation model learns mainly from people who rate hundreds of items. Evaluation separately checks users with only a few interactions, whose preferences may be less well represented.
A product team summarizes comments to identify priorities. It examines whether a handful of frequent posters account for most mentions of a requested feature.
An educational service collects feedback only after repeated use. It also investigates why some learners stop early, since their difficulties may be absent from the active-user feedback stream.

Think of it this way

Imagine a meeting in which a few participants speak repeatedly while others say little. Counting every statement equally tells you which views dominate the conversation, but not necessarily how many people hold each view.
Confusion arises when the volume of speech is presented as a direct measure of the whole group's preferences.

Worked example

Consider an original fictional platform with 100 users. Ten highly active users each submit 20 ratings, giving 200 ratings in total. Ninety occasional users each submit one rating, giving another 90.
Suppose each active user has an average score of five and each occasional user gives a score of three. The rating-weighted average is (200 × 5 + 90 × 3)/290 = 1,270/290, approximately 4.38.
An equally weighted average across these 100 users is (10 × 5 + 90 × 3)/100 = 320/100 = 3.20. The difference is approximately 1.18 rating points.
Although active users are only 10% of the user population, they supply 200/290, approximately 69% of the ratings. Their experience therefore dominates the record-level average.
In this constructed example, averaging each user's score first recovers the stated user-level average because every user is observed. If 30 users provided no ratings, that calculation would describe observed users only. Their missing preferences cannot be recovered merely by changing the weights of existing ratings.

Illustrative case study

Fictional Birch Courses trains a content recommender using lesson completions and ratings. A small group of enthusiastic learners contributes many observations, and the model's overall evaluation appears strong.
The team compares results for frequent and occasional learners. It finds that the latter group receives less useful recommendations and that learners who abandoned the service are absent from the rating-based assessment.
The developers retain an event-based metric for the interaction question it answers, but add user-level and activity-group evaluations. They also gather separate feedback about early abandonment. The resulting picture is less flattering than one headline average, yet more useful for deciding which parts of the learning experience need improvement.

Common misconceptions and mistakes

A large dataset can still be unrepresentative. Repeated records from the same active participants do not guarantee broad coverage.
Activity is not necessarily a protected characteristic, but uneven activity can interact with accessibility, opportunity and other circumstances. Investigate those relationships rather than assuming activity groups are interchangeable.
Discarding all highly active users is not a general solution. Their behavior may be important to the intended task. The issue is how their evidence should contribute to the stated objective.
Low activity does not automatically mean dissatisfaction. People can use a service infrequently for many reasons, so explanations require evidence beyond the activity count itself.

Business use

Before analyzing platform logs, specify whether the decision concerns interactions, active accounts, all customers or another population. This determines what a representative analysis would need to describe.
Compare the share of users with the share of records contributed by each activity group. Examine model performance and feedback patterns across those groups, while accounting for limited evidence in small groups.
Document who is missing. If the project concerns retention, a dataset made solely from continuing users may omit precisely the experiences that matter. Combine behavioral records with appropriately designed information gathering rather than treating silence as satisfaction or a negative rating.

Check your understanding

Why do ten users dominate the rating average?

Answer: They provide 200 of the 290 records.

What is the equally weighted user average in the example?

Answer: 3.20.

Why cannot reweighting reveal inactive users' opinions automatically?

Answer: Their relevant outcomes are unobserved and require additional evidence or explicit assumptions.

Related terms

Learn next

For a small dataset, compare records per user and calculate both an event-weighted and user-weighted summary. State the target of each, then list the people or experiences that neither summary observes.

Sources

Artificial Intelligence Ontology, Activity Bias: source-term definition concerning active versus less active users. https://raw.githubusercontent.com/berkeleybop/artificial-intelligence-ontology/main/aio.owl NIST SP 1270, Towards a Standard for Identifying and Managing Bias in Artificial Intelligence, glossary definition of activity bias, retrieved through indexed publication text. https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=934464 The rating arithmetic, diagnostic suggestions and fictional course platform are original teaching illustrations.

Previous: Active-set method Next: Activity expressions