Are the towers always identical?

No. Their structures and features can differ.

Must they share weights?

No. Weight sharing is a design choice.

Is retrieval the same as final ranking?

No. Retrieval often produces candidates for a later, more detailed scoring stage.

Can new items be represented?

Feature-based towers may support this, but coverage and quality still require evaluation.

AI Glossary / Retrieval and knowledge systems / Intermediate

Two-Tower Embedding Model

Definition

A Two-Tower Embedding Model uses two encoders to map two kinds of input, such as queries and documents or users and items, into compatible vector representations. A comparison function scores how well the outputs match. The architecture is especially useful for retrieval because candidate embeddings can often be computed in advance and searched efficiently when a new query arrives.

What it means

The two towers separate the processing of the two sides of a matching task. One might encode a user's current context and history, while the other encodes a product's characteristics. Another system might encode text on both sides, or text on one side and images on the other.
The towers need compatible outputs, but they do not necessarily require different modalities, identical architectures, or shared parameters. Equal vector length alone also does not align meaning. Training and the scoring design must make the two representations useful for matching.

How it works

Construct training examples that connect the two sides, such as a query with a relevant document or an observed user-item interaction. Define positive pairs and a strategy for comparison examples. An unobserved interaction is not automatically a true dislike; exposure and sampling affect the training signal.
Each encoder produces an embedding. A score such as a dot product or cosine similarity compares the outputs, and a training objective encourages appropriate pairs to score favorably relative to alternatives. Details such as normalization, negative sampling, and temperature influence the learned geometry.
For serving, encode candidates and store them in an index. At request time, encode the query and retrieve candidates with high scores, often using approximate nearest-neighbor search for scale. This separation avoids running a joint query-candidate encoder across the entire collection.
The retrieved set may then pass to a richer ranking model. A two-tower score is an affinity measure, not automatically a calibrated probability. Candidate vectors must be refreshed when relevant inputs or model versions change, and query and candidate embeddings must remain compatible. Eligibility filters, current availability, and access permissions also need appropriate handling in the retrieval workflow.

Three practical examples

A recommendation service retrieves candidate products using a user-context tower and an item-feature tower.
A knowledge-search system encodes questions and documents separately to find likely matches quickly.
A multimodal catalog aligns text descriptions and images so a text query can retrieve visually relevant items.

Think of it this way

Think of two specialists translating different kinds of descriptions into a shared coordinate system. Once every catalog item has a coordinate, a new request can be translated and compared with the catalog efficiently. The translators must learn compatible conventions; merely producing the same number of coordinates is insufficient.

Worked example

Consider an original scoring example with an invented query vector q=(1,2). Three candidate vectors are A=(2,1), B=(0,3), and C=(3,0). These vectors illustrate scoring and are not outputs from an actual trained model.
Their dot products with q are 1 times 2 plus 2 times 1 = 4 for A, 1 times 0 plus 2 times 3 = 6 for B, and 1 times 3 plus 2 times 0 = 3 for C. Dot-product retrieval ranks B first, then A, then C.
The query norm is sqrt(5). A also has norm sqrt(5), while B and C each have norm 3. Cosine similarities are therefore 4/5=0.8 for A, 6/(3*sqrt(5)), approximately 0.8944, for B, and 3/(3*sqrt(5)), approximately 0.4472, for C. The order happens to remain the same here.
Now replace A with A2=(20,10), which has the same direction as A but ten times its magnitude. Its dot product becomes 40, so it ranks above B under dot-product scoring. Its cosine similarity remains 0.8, so it still ranks below B under cosine scoring.
This shows why the index metric must match the model's intended comparison. Neither score is a probability: the dot product can exceed 1, and even a cosine value within a familiar range does not establish a likelihood of relevance. Normalization changes what information the score uses and should be consistent with training and evaluation.

Illustrative case study

Fictional case: Linden Courses needs to retrieve a manageable candidate set from a growing course catalog. One tower encodes learner context; the other encodes course information.
The team evaluates retrieval separately from final ranking, checking whether relevant courses enter the candidate set at all. It compares exact scoring with the approximate index to measure indexing losses. When course descriptions change, it refreshes their vectors and verifies that both towers and the index use compatible versions.
A later ranker considers additional information for the shortlisted courses. This hypothetical case illustrates a two-stage design, not a real implementation or a claim of measured gains.

Common misconceptions and mistakes

Two towers do not necessarily mean two different modalities. Query and document text can use the same modality with different roles.
A common embedding dimension is necessary for a simple dot product but is not sufficient for meaningful alignment. Independently trained spaces cannot be assumed interchangeable.
The architecture does not jointly inspect every query-candidate token interaction before initial retrieval. That efficiency can limit detailed matching compared with a joint encoder. Approximate indexing introduces another possible source of missed candidates, separate from errors in the embedding model.

Business use

Two-tower models can support scalable discovery when a business must select a small candidate set from a large catalog. Their practical advantage comes from separating reusable candidate computation from request-time query computation.
Evaluate candidate recall, response time, freshness, and performance for new users or items. A strong final ranker cannot recover an item omitted by every candidate source. Track model and index versions together, and test whether training interactions reflect exposure patterns that could distort recommendations.

Check your understanding

Why does scaling A change its dot-product rank?

Answer: Dot product depends on vector magnitude as well as direction.

Why does cosine remain unchanged?

Answer: Positive rescaling preserves direction.

Why can candidate embeddings be precomputed?

Answer: Their tower can often encode them without the current query, provided the required candidate features are available.

Related terms

Embedding
Candidate generation
Dot product
Cosine similarity
(1+ε)-approximate nearest neighbor search
Contrastive Learning
Dual encoder
approximate nearest-neighbor search
reranking

Learn next

Learn retrieval evaluation and vector similarity, then compare exact and approximate candidate search. Next study training-pair construction and negative sampling, paying attention to the difference between observed engagement and genuine preference.

Negative sampling

Sources

Primary implementation tutorial: TensorFlow Recommenders, Recommending movies: retrieval: https://www.tensorflow.org/recommenders/examples/basic_retrieval Supplied technical reference: Hopsworks, Two-Tower Embedding Model: https://www.hopsworks.ai/dictionary/two-tower-embedding-model

Previous: Two-stream self-attention Next: Twoway factorial design