Does it require text?

No. Any appropriately encoded binary features can be considered.

Can it accept continuous measurements directly?

The Bernoulli model requires binary values; meaningful binarization is a modeling decision, not a harmless formatting change.

Does naive Bayes always beat larger models on small datasets?

No. Compare alternatives under the same honest evaluation conditions.

AI Glossary / Statistics and probability / Intermediate

Bernoulli Naive Bayes

Definition

Bernoulli Naive Bayes is a supervised classification method for binary features, such as whether a word appears in a document. It combines class probabilities with probabilities of feature presence and absence, assuming the features are conditionally independent given the class. Binary features do not restrict the model to two classes.

What it means

Suppose a message is represented by yes-or-no indicators: contains a refund request, mentions a delivery, and includes an invoice number. Bernoulli Naive Bayes learns how frequently each indicator is present within each labeled category.

It then scores a new message using both what appears and what does not. This is different from a text model that uses the number of times each word occurs. Repeating a word ten times does not increase its binary presence indicator beyond one.

The word naive describes a simplifying conditional-independence assumption. It does not mean the method is useless, but it warns against interpreting its calculated probabilities as automatically well calibrated.

How it works

Let x_j be zero or one, and let theta_cj be the probability that feature j is present in class c. The model gives a feature vector likelihood equal to the product of theta_cj raised to x_j and (1 - theta_cj) raised to (1 - x_j).

Multiply that likelihood by the class prior, then normalize the class scores to obtain model probabilities. Implementations usually sum logarithms to avoid numerical underflow from many small factors.

A common smoothed estimate is theta_cj = (n_cj + alpha)/(n_c + 2alpha), where n_cj counts class-c training records containing feature j and n_c counts class-c records. Positive alpha prevents an unseen presence or absence from receiving an estimated probability of zero. Class-prior estimation is a separate choice.

Three practical examples

A service desk classifies short requests using indicators for selected terms, then sends uncertain or consequential cases to staff.
A document organizer distinguishes invoice, contract, and memo categories from binary phrase and layout indicators.
A researcher uses binary sensor-event indicators as a simple classification baseline before comparing models that capture dependencies between events.

Think of it this way

Think of a checklist associated with several folders. Each folder has a different pattern of commonly checked and unchecked boxes. The model compares a new checklist with those patterns. It treats the boxes as separate clues within each folder, even when real-world boxes may be closely related.

Worked example

Consider an original two-feature example with classes urgent and routine. There are four training messages in each class, so use class priors of one-half. Feature A appears in three urgent messages and one routine message. Feature B appears in zero urgent messages and two routine messages.

With alpha = 1, the urgent presence probabilities are 4/6 for A and 1/6 for B. The routine probabilities are 2/6 for A and 3/6 for B.

A new message has A present and B absent. Its urgent score is (1/2) × (4/6) × (5/6) = 5/18. Its routine score is (1/2) × (2/6) × (3/6) = 1/12.

Normalizing gives P(urgent | features) = (5/18)/(5/18 + 1/12) = 10/13, approximately 76.92%. Equal-cost classification chooses urgent.

The absence of B contributes a factor of 5/6 versus 3/6; it is not ignored. Without smoothing, the observed zero count would imply a zero presence probability for B in urgent messages. The calculated 76.92% is a model output from tiny invented data, not an externally validated reliability claim.

Illustrative case study

Fictional case: Cedar Support builds a request-routing prototype. Staff label historical tickets, and the team creates binary indicators using a vocabulary learned from the training portion alone.

The model is fast enough to provide a baseline, but inspection reveals that several indicators are near-duplicate phrases. Multiplying their contributions makes some predictions excessively confident. The team evaluates errors on later tickets and compares calibration with actual category frequencies.

The final pilot assists routing while preserving staff correction and escalation. Its acceptance criterion is fewer costly misroutes under realistic workload, rather than a high training score. The company retains the feature definition and training date so the experiment can be repeated.

Common misconceptions and mistakes

Bernoulli features have two values, but the classifier can distinguish more than two classes.
Conditional independence means independence within a specified class, not necessarily independence across the whole dataset.
Missing information is not automatically the same as an observed zero. Decide how unknown values are represented.
A sparse feature matrix's stored zeros still represent absent features in the likelihood.
More repeated occurrences do not strengthen a binary indicator. If counts matter, compare a suitable count-based model.
A strong class prediction does not establish a causal relationship between a feature and the class.

Business use

Bernoulli Naive Bayes can support inexpensive baselines and interpretable feature-level investigations. Document tokenization, vocabulary, binarization thresholds, smoothing, and class priors. Apply the same transformations at training and prediction time.

Evaluate performance by category and by the costs of errors. An ordinary accuracy figure may hide poor recall for a rare but important class. If model probabilities drive decisions, assess calibration on held-out data and choose decision thresholds accordingly.

Also monitor changing language and labeling policies. A stable implementation can deteriorate when users adopt new phrases or when category definitions change. Retraining should be tied to evidence, with comparisons against the previous model.

Check your understanding

Answer: 1.

What probability is used when a feature is absent?

Answer: One minus its class-conditional presence probability. 2.

Why does the smoothing denominator contain 2alpha?

Answer: There are two possible feature outcomes receiving the symmetric smoothing contribution. 3.

Can correlated indicators affect confidence?

Answer: Yes; the model may count related evidence as though it were conditionally independent.

Related terms

Learn next

Learn Bayes' theorem and binary likelihoods, then compare Bernoulli and multinomial text representations. Examine confusion matrices and calibration before choosing a deployment threshold. Practice tracing one classification through its prior and feature contributions.

Sources

Scikit-learn developers, Naive Bayes user guide, Bernoulli variant and conditional-independence model: https://scikit-learn.org/stable/modules/naive_bayes.html#bernoulli-naive-bayes

Scikit-learn developers, BernoulliNB API documentation: https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.BernoulliNB.html

The official documentation was opened and the core model checked. Training counts, arithmetic, and the fictional case are original teaching material.

Previous: Bernoulli mixture model Next: Bernoulli trial