1.58-bit large language model - A 1.58-bit large language model uses ternary weight values, typically -1, 0 and +1, in its low-precision linear computations.
Adaptive sampling - Adaptive sampling is an approach in which information from earlier samples influences which samples are selected next or how sampling effort is allocated.
Adaptive Synthetic Sampling - Adaptive Synthetic Sampling, usually abbreviated ADASYN, is an oversampling method for imbalanced classification.
AI slop - AI slop is an informal, critical term for low-quality AI-generated content produced with an emphasis on quantity rather than usefulness, accuracy or...
Attention - Attention is a neural-network mechanism that combines information using weights that depend on the current input or processing context.
Attention head - An attention head is one attention computation within a multi-head attention mechanism.
Attention mask - An attention mask specifies which positions may participate in an attention calculation.
Attribute sampling - Attribute sampling is a decision-tree training technique that randomly selects a subset of input features to consider when choosing a split.
Autoencoder - An autoencoder is a neural network trained to reconstruct an input from an internal representation of that input.
Auxiliary Classifier GAN - An Auxiliary Classifier GAN, or AC-GAN, is a class-conditional generative adversarial network whose discriminator predicts both whether an example is real...
Backbone - A backbone is the main feature-producing network within a larger model.
Bellman equation - A Bellman equation expresses the value of a decision as an immediate expected reward plus the discounted value of what happens next.
Bellman operator - A Bellman operator is a mathematical transformation that takes a proposed value function and returns updated values using immediate rewards and...
Bellman update - A Bellman update revises an estimated state value or action value using an immediate reward and the estimated value of what follows.
Bidirectional language model - A bidirectional language model uses context before and after a target position to model language at that position.
Bidirectional Transformer Language Model - A Bidirectional Transformer Language Model uses Transformer layers that allow a token's representation to incorporate surrounding context from both earlier...
Block sampling - Block sampling, in probabilistic inference, updates a group of random variables together rather than changing only one variable at a time.
Candidate sampling - Candidate sampling is a training approach that evaluates a limited selection of output classes instead of all possible classes for every example.
Channel - A channel is a distinct component of a data representation.
Constitutional AI - Constitutional AI is an approach to shaping AI behavior through an explicit set of written principles used in training and feedback.
Convenience sampling - Convenience sampling selects data or participants mainly because they are easy to obtain.
Conversational coding - Conversational coding is an iterative way of creating or changing software through dialogue with an AI system.
Correlated sampling - Correlated sampling, in simulation-based comparison, deliberately links the randomness used to evaluate alternatives.
Cross-attention - Cross-attention is an attention mechanism in which queries come from one set of representations and keys and values come from another.
Diffusion model - A diffusion model is a generative model that learns to produce data through a process related to reversing progressive corruption.
Distillation - Distillation is training a student model using information supplied by a teacher model or ensemble.
Dot-product attention - Dot-product attention is a mechanism that scores the compatibility of a query vector with key vectors using their dot products, converts those scores into...
Exploration - Exploration in reinforcement learning means taking actions to learn more about their outcomes or the environment, rather than always choosing the action...
Exploration bonus - An exploration bonus is an extra incentive used by a learning algorithm to favor actions or states that are uncertain, unfamiliar, or insufficiently tested.
Exploration function - An exploration function is a rule that combines an agent's current value estimates with information about how well actions or states have been explored.
Exploration problem - The exploration problem is the challenge of deciding how an AI agent should gather experience when it does not yet know which actions lead to useful outcomes.
Exploratory data analysis - Exploratory data analysis, or EDA, is an approach to understanding a dataset through summaries, visualizations, and iterative investigation before...
Exploratory forecasting - Exploratory forecasting examines how the future might develop from present conditions, past trends, and alternative assumptions.
Factored Language Model - A Factored Language Model represents a word or token through several associated factors, such as its surface form, stem, grammatical category, or learned...
Factuality - Factuality is the extent to which factual claims in an AI response are accurate about the world, within the relevant time, scope, and context.
Fast decay - Fast decay describes a quantity decreasing quickly over a specified sequence of steps or period of time.
Fine-tuning - Fine-tuning is additional training that adapts a pretrained model to a particular task, domain, or desired behavior.
Flash Attention - FlashAttention is an efficient way to compute attention in neural networks by reducing data movement between different levels of GPU memory.
Fraction of successes - The fraction of successes is the number of trials meeting a defined success condition divided by the total number of eligible trials.
Function Calling with LLMs - Function Calling with LLMs is a way for a language model to request the use of defined software functions through structured outputs.
Generated text - Generated text is written content produced by a computational system, such as a language model, in response to inputs or instructions.
Generative grammar - Generative grammar is an approach to describing language through explicit principles or rules that specify possible linguistic structures.
Generative Language Interface - A Generative Language Interface is a descriptive term for an interface that uses a generative language model to help people interact with information or...
Generative model - A generative model represents a process or distribution that can produce data, such as text, images, sounds, or structured records.
Gibbs sampling - Gibbs sampling is a Markov chain Monte Carlo method for drawing approximate samples from a joint probability distribution by repeatedly sampling variables...
Glitch token - A Glitch token is a token associated with unusually unreliable or anomalous behavior in a language model, such as difficulty reproducing its text or...
Golden response - A golden response is a carefully selected reference answer used to represent an expected or high-quality response to a particular input.
Graph Language Model - A Graph Language Model combines language representations with information about graph structure, such as entities and their relationships.