Nuisance function - A nuisance function is a function needed to estimate or make inferences about a target quantity, even though that function is not the primary object of...
Objective - An objective is a quantity that a learning or optimization process tries to improve, usually by minimizing or maximizing it.
Out-of-bag evaluation - Out-of-bag evaluation estimates a bagged model's predictive performance by using, for each training example, only the component models that were trained...
Pairwise metric - A pairwise metric is a rule used to compare two observations, commonly by calculating their distance or similarity.
Pass at k (pass@k) - Pass at k, written pass@k, measures the chance or observed rate that at least one of k generated attempts solves a task under a specified correctness test.
Permutation variable importances - Permutation variable importances measure how much a fitted model's performance changes when the values of an input variable are shuffled across evaluation...
Perplexity - Perplexity measures how well a probabilistic model predicts an observed sequence, commonly a sequence of words or tokens.
Pinball loss - Pinball loss is an asymmetric error measure used to train and evaluate quantile predictions.
Plackett-Luce model - The Plackett - Luce model is a probability model for rankings that assigns each item a positive worth and represents an ordering as successive choices from...
Policy evaluation - Policy evaluation is the task of estimating how much future reward an agent can expect when it follows a specified policy.
Positive class - The positive class is the category a classification task designates as the condition or event being detected.
Post-sample evaluation - Post-sample evaluation assesses a forecasting model using observations from after the period on which it was estimated.
Precision - Precision is the fraction of a model's positive predictions that are actually positive according to the reference labels.
Precision at k (precision@k) - Precision at k, written precision@k or P@k, is the fraction of the first k items in a ranked result list that are relevant.
Precision-recall curve - A precision-recall curve shows how a classifier's precision and recall change as its decision threshold varies.
Preferential Bayesian optimization - Preferential Bayesian optimization is a sequential search method that uses comparisons between alternatives to seek a preferred solution.
Probabilistic Programming - Probabilistic programming is an approach to expressing statistical models as programs that represent uncertainty through probability distributions.
Propensity scores - Propensity scores are the probabilities of receiving a particular treatment or exposure conditional on observed pretreatment characteristics.
Proper accuracy scoring rule - A proper accuracy scoring rule evaluates probability forecasts so that reporting the true outcome probabilities achieves the best expected score.
Proxy metric - A proxy metric is a measurable quantity used as an indirect indicator of an outcome or quality that is harder to measure directly.
Random experiment - A random experiment is a process with an uncertain outcome drawn from a defined set of possible outcomes, called its sample space.
Randomized weighted majority algorithm - The randomized weighted majority algorithm combines a set of predictors by choosing predictions randomly according to their current weights, then reducing...
Ranking loss - Ranking loss, in the multilabel evaluation sense used here, measures the fraction of relevant-versus-irrelevant label pairs that a model orders incorrectly...
Recall - Recall is the fraction of actual positive cases that a system correctly identifies as positive.
Recall at k (recall@k) - Recall at k, written recall@k or R@k, is the fraction of a defined set of relevant items that appears among the first k ranked results.
Remediation - Remediation is corrective action taken to address an identified weakness, failure or harmful outcome.
ROC (receiver operating characteristic) Curve - A receiver operating characteristic curve, or ROC curve, plots a classifier's true-positive rate against its false-positive rate as the decision threshold...
ROUGE-L - ROUGE-L is a member of the ROUGE text-evaluation family that compares a candidate with a reference using a longest common subsequence, or LCS.
ROUGE-N - ROUGE-N is a reference-overlap measure that compares a candidate text with a reference using contiguous sequences of N tokens, called n-grams.
ROUGE-S - ROUGE-S is a ROUGE evaluation measure based on matching skip-bigrams between a candidate text and a reference.
Sequential exchangeability - Sequential exchangeability is a causal assumption that, at each treatment decision, treatment is independent of the relevant future potential outcomes...
Side-by-side evaluation - Side-by-side evaluation compares two systems by judging their outputs for the same input.
Silhouette Coefficient - The Silhouette Coefficient measures how well an observation fits its assigned cluster compared with the nearest competing cluster.
Similarity measure - A similarity measure is a rule that assigns a numerical score to how alike two objects are in a chosen representation.
Squared hinge loss - Squared hinge loss is a classification loss that squares the positive shortfall from a target margin.
Structural causal model - A structural causal model, or SCM, represents how variables are generated by causal mechanisms.
Super learner - Super learner is an ensemble-learning approach that uses cross-validation to select or combine predictions from a library of candidate algorithms according...
Test and Evaluation, Verification and Validation - Test and Evaluation, Verification and Validation, commonly abbreviated TEVV, is a collection of activities that builds evidence about an AI system's...
Test loss - Test loss is the value of a chosen loss function on a dataset reserved for evaluating a trained and selected model.
Thurstone model - A Thurstone model represents comparative judgments using latent values disturbed by normally distributed variation.
Top-k accuracy - Top-k accuracy is the proportion of evaluated examples for which the target label appears among the k highest-ranked predictions.
Toxicity - In AI evaluation, toxicity refers to abusive, threatening, insulting or otherwise harmful conversational content as defined by a particular assessment scheme.
Training loss - Training loss is the value of the learning objective measured on examples used to fit a model.
Treatment-confounder feedback - Treatment-confounder feedback occurs when earlier treatment changes a later variable that both influences subsequent treatment and predicts the outcome.
Treatment effect - A treatment effect is the causal difference between outcomes under two specified interventions or treatment conditions.
True positive rate - True positive rate is the proportion of actual positive cases that a classifier correctly identifies as positive. It is also called recall or sensitivity.
Validation - Validation is the assessment of a model or system against criteria that help determine whether it is suitable for its purpose.