--- title: "Embeddings in 3D: how models turn words into coordinates" description: "Type a sentence, watch each token land as a point in 3D space. Click any token to inspect its vector and see which other tokens it's nearest to." published: 2026-04-18 updated: 2026-06-14 canonical: https://protrailblazer.com/posts/embeddings-in-3d/ tags: [machine-learning, demo, embeddings] --- # Embeddings in 3D: how models turn words into coordinates **Answer:** An embedding is a list of numbers that represents a token's meaning. Models learn these numbers during training so words with related meanings end up close together in space. The explorer below tokenizes any sentence you type, projects the resulting vectors into 3D, and lets you click each token to inspect its dimensions and nearest neighbors. The interactive below tokenizes whatever sentence you give it, computes a synthetic 8-dimensional embedding for each token, and projects those vectors into 3D so you can walk around them. Click any token sphere (or its name in the left rail) to see the vector behind it and the four nearest neighbors by cosine similarity. *Interactive demo: [open the embedding space demo](https://protrailblazer.com/demos/embedding-space/)* ## What does the demo show? Each red sphere is a token from your sentence. Its position in 3D is a compressed projection of an 8-dimensional embedding vector. The closer two spheres are, the more similar their vectors by cosine similarity, where 1.0 means identical direction, 0 means unrelated, and -1 means opposite. A few things to try: - **Edit the sentence.** Change a word, swap word order, add punctuation. Watch how the cluster shifts. Punctuation tokens drift apart from word tokens. - **Compare similar words.** Try "cat dog wolf table". The animals tend to cluster, "table" sits further out. - **Click any token.** The right panel shows its 8D vector with each dimension drawn as a horizontal bar. Bigger absolute values mean stronger signal in that dimension. The bottom of the panel lists the four nearest neighbors. - **Watch the heatmap.** Bottom-right is a token-to-token cosine similarity matrix. Red squares mean similar tokens, dark squares mean unrelated. ## How do embeddings actually work? This demo uses a deterministic toy embedding (token character codes hashed into 8 dimensions, with a few hand-coded rules for common stopwords and punctuation). Real models do something fundamentally similar but with two big differences: 1. **Dimension count.** Production embeddings are typically 512 to 12,288 dimensions, not 8. The geometry is the same, just much higher-dimensional. We project down to 3D for visualization, the way you would project to 3D for plotting in TensorBoard or UMAP. 2. **Learned, not hand-coded.** Real embeddings are learned during pretraining. The model adjusts the numbers token by token across billions of examples, shaped by the same [gradient descent](https://protrailblazer.com/posts/gradient-descent-demo) that trains the rest of the network, until tokens that appear in similar contexts end up near each other in space. No human ever decides "this dimension means animal-ness." Why this matters: once words are coordinates, **meaning becomes geometry**. You can add and subtract vectors (the famous "king minus man plus woman lands near queen" property in well-trained spaces). You can find synonyms by nearest-neighbor search. You can compare entire sentences by averaging their token vectors. The whole field of vector search and [retrieval-augmented generation](https://protrailblazer.com/posts/retrieval-augmented-generation) rests on this idea. ## Frequently asked questions ### What is an embedding? An embedding is a list of numbers (a vector) that represents the meaning of a token, word, or piece of text. Models learn these numbers during training so that items with related meanings end up close together in the same space. Once meaning is stored as coordinates, you can measure similarity, search, and cluster with ordinary geometry. ### What is cosine similarity? Cosine similarity measures how closely two vectors point in the same direction, ignoring their length. It runs from 1.0 (identical direction, very similar) through 0 (unrelated) to -1 (opposite). It is the standard way to compare embeddings because direction, not magnitude, is what carries meaning. ### How many dimensions do real embeddings have? Production embeddings are typically 512 to 12,288 dimensions, far more than the 8 this demo uses and the 3 we project down to for viewing. The geometry works the same way at any dimension; high-dimensional space just has far more room to separate meanings. Tools like UMAP or t-SNE are used to project them down when you want to look at them. ### How are embeddings used in search and RAG? Search and retrieval-augmented generation both embed your query and your documents into the same space, then return the documents whose vectors are nearest the query. Because nearness means semantic similarity, this finds relevant text even when the exact words do not match. It is the retrieval half of every RAG pipeline. ## Key takeaways - Tokens are represented as points in a high-dimensional space. - Similar meaning means spatially close. Cosine similarity is the standard distance measure. - Real embeddings have hundreds to thousands of dimensions, not 8. We project to 3D only to look at them. - Embeddings are learned during pretraining, not hand-coded. The model finds the structure on its own.