Sequential importance sampling - Sequential importance sampling, or SIS, is a probabilistic inference method that builds candidate sequences incrementally and updates a weight for each...
Skip-gram negative sampling - Skip-gram negative sampling, commonly abbreviated SGNS, is a method for learning word vectors by distinguishing observed word-context pairs from randomly...
Specificational coding - Specificational coding is an approach to creating software by writing and maintaining a human-language specification file that describes the required...
Speculative decoding - Speculative decoding is an inference technique that accelerates autoregressive generation by proposing several tokens with a cheaper draft process and...
Stratified sampling - Stratified sampling divides a population or dataset into distinct groups, called strata, and selects samples separately within those groups.
Subword token - A subword token is a unit of text used by a tokenizer that can represent part of a word, a whole word, or another learned text fragment.
Supervised fine-tuning - Supervised fine-tuning, or SFT, adapts an already trained model using examples that specify the desired output for an input.
Sycophancy - Sycophancy in an AI assistant is a tendency to agree with, flatter, or accommodate a user's stated views at the expense of an independent, evidence-based...
Temperature - Temperature is a setting that changes how concentrated or spread out a model's next-token probability distribution is before sampling.
Temperature sampling - Temperature sampling is a decoding method that draws a model's next token from a probability distribution adjusted by a temperature value.
Thompson sampling - Thompson sampling is a method for making repeated decisions when the rewards of available actions are uncertain.
Token-mixing - Token-mixing is the exchange or combination of information across token positions inside a neural network.
Tokenization - Tokenization is the process of turning text into smaller units called tokens so a language-processing system can work with it.
Tokenizer - A tokenizer is the component that converts text into the token units and numerical identifiers expected by a language-processing system.
Top-k sampling - Top-k sampling is a text-generation method that limits each next-token draw to the k highest-scoring candidate tokens.
Top-p sampling - Top-p sampling, also called nucleus sampling, generates text by sampling from the smallest leading set of next-token candidates whose cumulative probability...
Transformer - A Transformer is a neural-network architecture built around attention mechanisms that combine information from different positions in an input.
Transformer Language Model - A Transformer Language Model is a language model that uses Transformer layers to build contextual token representations and make language predictions.
Transformer Network - A Transformer Network is a neural network assembled from Transformer-style components, especially attention and feed-forward layers that build contextual...
Undersampling - Undersampling reduces the number of training examples from selected groups, commonly an overrepresented class, to change the balance of a dataset.
Unidirectional language model - A unidirectional language model predicts tokens using context from one direction in a sequence.
Unsupervised Pretrained Network - An unsupervised pretrained network is a neural network whose parameters are first learned using an objective that does not require the eventual task's...
Variational autoencoder - A variational autoencoder, or VAE, is a probabilistic generative model that learns a latent-variable representation of data together with an approximate...
Vector quantization - Vector quantization represents an input vector by one of a finite set of representative vectors called a codebook.
Vibe coding - Vibe coding is an informal term for creating software by describing desired behavior to a generative AI system and iterating on what it produces.
Word tokenization - Word tokenization divides text into word-level units and, depending on the rules, separate punctuation or other tokens.
Zero Redundancy Optimizer - Zero Redundancy Optimizer, usually written ZeRO, is a family of distributed-training memory optimizations that partitions model states across data-parallel...