Does SGNS need manually labeled examples?

The usual training pairs are derived from text and a noise-sampling procedure rather than manually annotated categories.

Does it produce a different vector for every context?

Standard SGNS produces static vocabulary representations; contextual embedding models work differently.

Which vector table is the final embedding?

Implementations and applications can use the input table or another declared combination. State the choice when comparing results.

AI Glossary / Natural language and speech / Advanced

SGNS

Definition

SGNS stands for skip-gram with negative sampling, a method for learning word embeddings by distinguishing observed word - context pairs from sampled noise pairs. Training adjusts word and context vectors through a binary classification objective. The resulting vectors provide compact, learned representations of distributional patterns in the training corpus.

What it means

Words appearing in similar contexts can acquire useful vector relationships. SGNS learns from those contexts without requiring a person to label every pair as a linguistic category. Its positive examples come from observed neighborhoods; its negative examples come from a chosen sampling distribution.
Negative does not mean an antonym or a word with negative sentiment. It means a pair used as noise in the training objective. Such a pair may still be linguistically plausible elsewhere.
Standard SGNS representations are static: a vocabulary item has a learned vector rather than a newly contextualized vector for each occurrence. Different senses of an ambiguous word can therefore be blended.

How it works

Choose a context-window policy and collect center - context pairs from text. Maintain separate input and output vector tables. For a pair, take the dot product between the center's input vector and the context's output vector, then pass it through the logistic sigmoid.
For an observed pair, training encourages a larger sigmoid value. For each sampled negative pair, it encourages a smaller value. The loss adds a negative log term for the positive label and negative log terms for the sampled negative labels. Only the relevant vectors need updating for that training instance.
The noise distribution and number of negative samples are design choices. The original work studied a distribution proportional to word frequency raised to the three-quarter power. This is a sampling choice, not a universal language law or a guarantee of optimal performance.

Three practical examples

A domain vocabulary project learns vectors from maintenance documents to suggest related terms for editorial review. Similarity provides candidates, not automatic synonym declarations.
A text-classification experiment uses SGNS vectors as input features and compares them with another representation on held-out documents. The downstream test determines whether the vectors help that task.
A corpus analyst studies nearest neighbors for a technical word. Inspection can reveal domain associations, but the analyst checks whether the neighborhood mixes senses or reflects repetitive source material.

Think of it this way

Think of learning useful coordinates by repeatedly asking which pairs belong to observed neighborhoods and which were sampled for contrast. The coordinates are the reusable result of that exercise. They are not a dictionary of definitions, and the binary training question is not the same as assigning probabilities to every possible next word.

Worked example

Use a two-dimensional center vector w = [1, 0]. Let an observed context vector be c+ = [ln(3), 1], and let two sampled negative context vectors be c1 = [−ln(3), 0] and c2 = [0, 2]. These are invented numbers.
The dot products are ln(3), −ln(3), and 0. With sigmoid σ(z) = 1/(1 + exp(−z)), their positive-label scores are 0.75, 0.25, and 0.50. For the observed pair, the desired-label factor is 0.75. For the two noise pairs, the desired-label factors are 1 − 0.25 = 0.75 and 1 − 0.50 = 0.50.
The sampled objective therefore has loss −ln(0.75) − ln(0.75) − ln(0.50), approximately 1.2685. Equivalently, the product of desired-label factors is 0.28125. Lowering this loss encourages better discrimination for these labeled pairs; it does not prove the embeddings encode the intended word senses.
Notice that the three positive-label scores add to 1.50. That is permitted because they are binary scores for different pairs, not a normalized distribution over context words. Treating them as next-word probabilities would be an error.

Illustrative case study

Fictional company ServiceLex trains SGNS vectors on equipment documentation to support glossary discovery. Editors find that some useful technical associations appear, but an ambiguous term has neighbors from several meanings. Repeated boilerplate also influences the neighborhoods.
The team reviews corpus composition and evaluates suggestions against its actual glossary task. It records the context window, sampling policy, vocabulary filtering, and vector-table choice so comparisons are reproducible. Editors keep association, similarity, and synonymy as different judgments.
This fictional case illustrates how to inspect learned representations. It provides no measured claim about training speed, glossary coverage, or downstream accuracy.

Common misconceptions and mistakes

SGNS is one word2vec method, not a name for every embedding model. Skip-gram can also be trained using other objectives, and continuous bag-of-words is a different word2vec architecture.
Negative samples are not verified semantic opposites. A sampled pair's label belongs to the training setup, so it should not be read as a universal statement that the words never co-occur.
Do not interpret binary sigmoid outputs as a vocabulary-wide softmax. Do not assume nearby embeddings have identical meanings or that analogy arithmetic always works. Corpus patterns can include ambiguity, stereotypes, and domain-specific associations. A compact vector is a useful representation, not independent factual knowledge.

Business use

SGNS can support experiments involving lexical features, vocabulary exploration, or representations learned from a domain corpus. Judge its value through the intended downstream task and appropriate comparison methods. Keep preprocessing and training settings with the vectors, and check stability when data or random initialization changes. For user-facing suggestions, retain human review where a distributional association could be mistaken for a definition or verified relationship.

Check your understanding

Why is the first noise pair's desired-label factor 0.75?

Answer: Its positive-label score is 0.25, so the negative-label score is 0.75.

Why may the three positive-label scores sum above one?

Answer: They answer separate binary pair-classification questions.

What does the sample loss measure?

Answer: Agreement with the one positive and two negative training labels under the supplied vectors, not general semantic understanding.

Related terms

Learn next

Compute a small SGNS loss by hand, then inspect how changing one dot product affects it. Compare nearest-neighbor judgments with an independently defined downstream task, keeping the training corpus and vector-table choice documented.

Sources

Mikolov and colleagues, Distributed Representations of Words and Phrases and their Compositionality, 2013, section 2.2, negative sampling: https://arxiv.org/pdf/1310.4546 Jurafsky and Martin, Speech and Language Processing, Chapter 5, section 5.5, SGNS classification and training: https://web.stanford.edu/~jurafsky/slp3/5.pdf The vectors, loss calculation, and ServiceLex scenario are original illustrations.

Previous: Set semantics Next: Shadow removal