---
title: "What does 7B mean? LLM parameter counts explained"
description: "The B in 7B counts the learned weights stored in a model. See how count, precision, and architecture determine file size in this interactive scale tunnel."
published: 2026-07-18
updated: 2026-07-18
canonical: https://protrailblazer.com/posts/what-does-7b-mean/
tags: [machine-learning, demo, llm, parameters, quantization, mixture-of-experts, model-size, fp16, int4]
---

# What does 7B mean? LLM parameter counts explained

**Answer:** The B in 7B stands for billion, and it counts the learned weights stored inside the model, not facts, tokens processed, or neurons in the biological sense. A 7B model holds exactly 7,000,000,000 floating-point numbers that were adjusted during training to minimize prediction error. Those numbers take up memory proportional to how many bits each one is stored with: at FP16 (16 bits per weight) a 7B model occupies about 14 GB just for the weights, and switching to INT4 (4 bits) drops that to roughly 3.5 GB without changing the count at all. Parameter count is a measure of size, not capability.

The interactive below draws six models from 1B to 400B parameters as tile fields on a shared canvas, where tile area is exactly proportional to parameter count. Pick a model, change the precision, switch to MoE, or run the story presets to see how count, bits, and architecture combine into a weight file size.

*Interactive demo: [open the llm parameter counts demo](https://protrailblazer.com/demos/llm-parameter-counts/)*

## What you're seeing

The canvas places all six model sizes on a shared horizontal baseline. Each model is a rectangular tile field whose area is proportional to its parameter count, so 7B is literally 7 times the area of 1B. Same-size individual tiles hold the same number of weights wherever they appear in the scene. A count sweep ticks through the selected model's weights in real time, and the number in the HUD top-left (TOTAL PARAMS) shows the exact integer, not a rounded approximation.

Below the tile fields runs a memory ruler. It converts the selected model's parameter count and the current precision setting into an exact weight file size using the formula: params x bits per weight / 8. Reference ticks for consumer and server hardware appear along the ruler as labeled guides. The formula is exact (1 GB = 10^9 bytes, as stated on the ruler). Running a model needs more memory than the weight file alone (the KV cache, activations, and buffers all add on top), but the weight file is the anchor.

The HUD top-right tracks seven live values: total parameters, active parameters per token (which differs from total only in MoE mode), bits per weight, weight file size, relative scale versus 1B, dense or MoE architecture, and the count sweep progress. The NOT strip in the bottom-left panel crosses out FACTS, TOKENS, NEURONS, and LINES OF CODE to make explicit what the number does not measure.

A few things to try:

- **Run the "Quantize 7B" story preset.** Watch TOTAL PARAMS stay fixed at 7,000,000,000 while WEIGHT FILE collapses from 14 GB (FP16) down to 3.5 GB (INT4). The tile field does not change size. Only the bits column shrinks.
- **Switch to "MoE 400B" and compare TOTAL PARAMS with ACTIVE / TOKEN.** The 400B MoE stores all expert weights but routes each token through roughly 32.5% of them (8 experts, top-2 routing, illustrative). The tile field shows the full 400B footprint; the HUD shows how much of it fires per forward pass.
- **Run "70B vs 7B" and use Fit All to see both models at once.** The 70B tile field is 10 times the area of 7B. Then change the 70B to INT4 and the 7B to FP32 and read the memory ruler: the 70B at INT4 (35 GB) is smaller than the 7B at FP32 (28 GB). File size is not the same as parameter count.
- **Toggle the Training Memory band on.** The x3 to x8 range above the weight file shows the rough envelope of training memory (gradients, optimizer states, activations). This band is labeled ILLUSTRATIVE because the real multiplier depends heavily on batch size, optimizer, and distributed sharding strategy.
- **Use the zoom controls or drag the canvas to read tile counts at different abstraction levels.** Zooming in shifts the granularity so each tile represents fewer parameters. At the finest zoom, the matrix microscope at the tunnel's far end resolves into individual stored numbers with values labeled ILLUSTRATIVE (they re-encode when precision changes, and the count stays unchanged).

## How parameter counts actually work

A language model is, at its core, a large collection of transformer layers, each containing matrices of floating-point numbers. During pretraining, those numbers are updated repeatedly via backpropagation to minimize the error the model makes when predicting the next token. The total count of those numbers is the parameter count.

The count is fixed after training. Nothing about running the model (the inputs you give it, the length of the conversation, the temperature setting) changes the number of weights stored in the file. What changes at inference is how those weights are read: during a forward pass, each weight is loaded from memory, multiplied with an activation, and discarded from the compute unit. The memory pressure during inference comes from holding all the weights in VRAM at once, not from repeatedly writing new ones.

The file size formula is arithmetic: weight file size in bytes = parameter count x bits per weight / 8. A 7B model at FP16 (16 bits) uses 7,000,000,000 x 16 / 8 = 14,000,000,000 bytes = 14 GB. At INT4 (4 bits) the same model uses 3.5 GB. At FP32 (32 bits) it uses 28 GB. The number 7,000,000,000 appears in all three equations. Only the bits column changes. For a walkthrough of what happens to those weights once the model starts generating text, see [how LLMs generate text](https://protrailblazer.com/posts/how-llms-generate-text).

## Why MoE models report two different numbers

A mixture-of-experts architecture replaces each dense feed-forward layer with a set of expert sub-networks and a routing mechanism. The router selects a small subset of experts to process each token. The headline parameter count reflects all experts stored in the file. The active parameter count reflects only the experts that fire on any given token.

The demo models this with an illustrative 8-expert, top-2 routing configuration plus a shared expert layer. At 10% of parameters shared and the top-2 of 8 specialized experts active, roughly 32.5% of the total parameter count is used per token. This is labeled ILLUSTRATIVE because real MoE models vary in expert count, top-k, and share ratio by architecture and layer.

The practical consequence is that sparse activation lets a MoE model store far more total capacity than a dense model of the same compute budget. A 400B MoE may activate only as many weights per token as a 130B dense model, while having access to the full 400B of trained knowledge across the expert pool. But you still need enough VRAM to hold all 400B weights if the model is not split across multiple devices. The [mixture-of-experts explainer](https://protrailblazer.com/posts/mixture-of-experts-explained) covers routing strategies and real-world tradeoffs in more depth.

## Frequently asked questions

### Does a 70B model know 10 times as many facts as a 7B model?

Not in any direct sense. Parameter count is a capacity measure, not a knowledge inventory. A larger model has more degrees of freedom to fit training data, but what it retains depends on the training data itself, the architecture, the learning rate schedule, and how the model was aligned or fine-tuned. Two models with identical counts can differ substantially in what they know or can do.

### Why does the parameter count stay the same when I change precision?

Precision determines how many bits each parameter is stored with, not how many parameters there are. Switching from FP16 to INT4 changes the number from 16 bits wide to 4 bits wide, but there are still 7,000,000,000 of them. The file gets smaller because each one takes less space, not because any are removed.

### What is the difference between total parameters and active parameters per token?

In a dense model they are the same: every weight participates in every forward pass. In a mixture-of-experts model, the architecture stores multiple expert sub-networks but only routes each token through a subset at inference time. The total count is what you store; the active count is what you compute with per token. A 400B MoE might activate only 130B of its parameters per token depending on the routing configuration.

### How much memory does it actually take to run a model?

The weight file is the floor, not the ceiling. Running a model also requires the KV cache (grows with context length), intermediate activations, and framework overhead. A rough rule for inference is 1.2 to 1.5 times the weight file for short contexts. Longer contexts expand the KV cache further. Training adds optimizer states and gradients on top, which is why the demo's training band spans x3 to x8 of the weight file.

### Why do some model cards list two parameter counts?

Usually because the model is a mixture-of-experts architecture. The first number is total parameters (everything stored in the file). The second number is active parameters per forward pass (what actually computes during inference). If you see something like "400B total, 52B active," the model stores 400B weights but only uses about 52B per token.

## Key takeaways

- Parameter count and weight file size are linked by arithmetic, not heuristics: multiply the count by bits per weight, divide by 8, and you have the exact weight storage in bytes. A 7B FP16 model is 14 GB; a 70B INT4 model is 35 GB. No estimation needed.
- Quantization changes bits per weight and therefore file size, but the parameter count printed on the model card never moves. The HUD in the demo shows this directly: TOTAL PARAMS stays fixed while WEIGHT FILE collapses as you drop from FP16 to INT4.
- MoE models report total parameters across all experts but activate only a fraction per token. A 400B MoE with 8 experts and top-2 routing computes with roughly 32.5% of its weights per token, which is why a 400B MoE can run faster than a 70B dense model at similar quality points despite the headline count.
- The parameter count on the model card says nothing about what the model can do. Two models with identical counts can differ wildly in capability depending on architecture, training data, fine-tuning, and the precision of activations and the KV cache at inference.
- Running a model costs more memory than storing its weights. The KV cache, activations, and optimizer states during training all add on top. The demo's training band (an illustrative x3 to x8 multiplier) shows how far training memory can exceed the raw weight file.