LLM Parameters vs Weights: What the Difference Really Is
LLM parameters vs weights: parameters are every learned number in a model (weights, biases, norm scales, embeddings), while weights are the multiplicative matrices that make up almost all of them, so the terms are often used interchangeably.
- A parameter is any number that starts random and is updated by gradient descent to reduce loss; weights are the main kind, and biases, LayerNorm gains and embedding vectors are the rest.
- Modern LLMs such as Llama drop most biases, so nearly 100% of parameters are weight matrices and the two words describe almost the same set.
- A standard transformer block stack holds about 12 x layers x d_model^2 parameters, plus vocab x d_model for embeddings; GPT-2 small checks out at about 124M.
- Memory for weights is parameters x bytes per parameter: 7B is about 14 GB in fp16 and roughly 3.5 to 4 GB at 4-bit, before KV cache and activations.
- Parameter count alone does not set quality: training tokens, compute and architecture matter, and mixture-of-experts models separate total from active parameters.
Are LLM parameters and weights the same thing?
Almost, but not strictly. A parameter is any number in the model that is learned during training: it is initialised randomly (or from a scheme like Xavier or scaled normal init) and then nudged by gradient descent to reduce the loss. Weights are the most important type of parameter, the entries of the matrices that multiply inputs.
Parameters = weights + biases + other learned values. The other values include LayerNorm or RMSNorm gain (and shift) vectors and the token embedding vectors. Embeddings are technically a lookup table, but they are learned the same way, so they count.
People use the words interchangeably because in LLMs the gap is tiny. Llama-style models have no bias terms on their linear layers and use RMSNorm without a shift, so the only non-matrix parameters are a few thousand-length vectors per layer. Those are a rounding error next to billions of matrix entries.
Headlines say 'parameters' because it covers everything learned, it is the general term in statistics and machine learning, and it is the standard unit of model size. 'Weights' survives in phrases like 'open weights' and 'load the weights', where it means the whole saved set of learned values.
Where do the parameters live in a transformer?
A decoder-only transformer has five places where learned numbers sit. Two of them, the attention and feed-forward (MLP) blocks, hold the great majority.
In a classic block with hidden size d_model and an MLP expansion of 4x, attention has four d x d matrices (4d^2) and the MLP has two matrices of d x 4d (8d^2). That gives 12d^2 per layer, so the MLP holds about two thirds and attention about one third.
| Component | What it holds | Approx. parameters |
|---|---|---|
| Token embedding | One learned vector per vocabulary token | vocab x d_model |
| Attention Q, K, V, output projections | Four weight matrices per layer | 4 x d_model^2 per layer |
| Feed-forward (MLP) | Up and down projections (three matrices in SwiGLU variants) | 8 x d_model^2 per layer (classic 4x MLP) |
| Normalisation | Gain (and sometimes shift) vectors | 2 x d_model per norm, tiny |
| Output head | Projects hidden state to vocabulary logits | vocab x d_model, or shared with embedding |
How do I count the parameters of a transformer?
Use the rule of thumb: non-embedding parameters ~ 12 x n_layers x d_model^2, then add vocab x d_model for embeddings (and position embeddings if the model has learned ones).
Worked check with GPT-2 small: 12 layers, d_model 768, vocab 50,257, context 1,024. The rule gives 12 x 12 x 768^2 = 84.9M for the blocks. Embeddings add 50,257 x 768 = 38.6M, positions add 1,024 x 768 = 0.8M. The total is about 124M, matching the published figure. The exact count including biases and norms is 124,439,808.
The rule changes with architecture. Llama 2 7B uses d_model 4096, 32 layers, a SwiGLU MLP with hidden size 11,008 (three matrices), and untied embeddings with a 32,000 vocabulary: attention 4 x 4096^2 = 67.1M, MLP 3 x 4096 x 11008 = 135.3M, so about 202M per layer, x32 = 6.48B, plus 2 x 131M for embeddings and head, giving about 6.74B.
def count_params(layers, d, vocab, ctx=0, mlp_mult=4, tied=True):
attn = 4 * d * d # Q, K, V, output (no biases)
mlp = 2 * mlp_mult * d * d # up + down projection
blocks = layers * (attn + mlp)
emb = vocab * d * (1 if tied else 2)
pos = ctx * d
return blocks + emb + pos
print(count_params(12, 768, 50257, ctx=1024)) # ~124.3M, close to GPT-2 small's 124.4M
# With PyTorch, count the real thing:
# sum(p.numel() for p in model.parameters())What does 7B parameters mean in memory?
Each parameter is one number, so weight memory is parameters x bytes per parameter. fp32 is 4 bytes, fp16 and bf16 are 2, int8 is 1, and 4-bit quantisation is about 0.5 bytes (a little more once scale factors are stored).
These figures are for weights only. Running the model also needs a KV cache and temporary activations, which you must add on top.
| Model size | fp16 / bf16 | int8 | 4-bit |
|---|---|---|---|
| 124M (GPT-2 small) | ~0.25 GB | ~0.12 GB | ~0.06 GB |
| 7B | ~14 GB | ~7 GB | ~3.5 to 4 GB |
| 13B | ~26 GB | ~13 GB | ~6.5 to 7 GB |
| 70B | ~140 GB | ~70 GB | ~35 to 40 GB |
How are parameters different from hyperparameters, activations and the KV cache?
Parameters are learned and saved. Hyperparameters are chosen by people before or during training and are not learned by gradient descent. Activations are the intermediate outputs each layer computes for a particular input, and they are not part of model size.
What happens to activations depends on the mode. At inference they can be discarded once the next layer has used them. During training they are kept until the backward pass uses them to compute gradients, which is why training memory grows with batch size and sequence length (and why techniques like activation checkpointing exist).
The KV cache is a special case of activations that is deliberately kept during generation: the keys and values for every past token are stored so they are not recomputed. Its size is 2 x layers x kv_heads x head_dim x tokens x bytes. For Llama 2 7B in fp16 (32 layers, 32 heads, head dim 128) that is 0.5 MB per token, so a 4,096-token context costs about 2 GiB per sequence.
| Thing | Set by | Saved with model? | Counts in '7B'? | Examples |
|---|---|---|---|---|
| Parameters | Gradient descent | Yes | Yes | Weight matrices, biases, norm gains, embeddings |
| Hyperparameters | Humans | Only as config | No | Learning rate, layers, d_model, heads, context length, batch size |
| Activations | The input at run time | No (discarded after inference; held until the backward pass in training) | No | Hidden states, attention scores |
| KV cache | Generation at run time | No | No | Cached keys and values per token |
What do 'open weights' and mixture-of-experts mean for the count?
'Open weights' means the trained parameters are published, so anyone can download and run the model. It does not by itself mean the training data, training code or a permissive licence are released, which is why it differs from 'open source'.
In a mixture-of-experts (MoE) model, the MLP is split into several experts and a router sends each token to only a few. Total parameters set the memory you need, because all experts must be loaded. Active parameters per token set the compute per step. Mixtral 8x7B has about 46.7B total but uses about 12.9B per token, so it needs memory like a ~47B model but runs at roughly the speed of a ~13B one.
Does more parameters mean a better model?
Not automatically. Scaling-law work, notably the Chinchilla results, showed that for a fixed compute budget, parameters and training tokens should grow together, and many earlier large models were undertrained. A smaller model trained on far more tokens can beat a larger one, and recent 7B to 8B models routinely outperform older models many times their size.
Parameters are also not facts stored one-to-one. Knowledge is spread across many weights in a distributed way, and no single number corresponds to a sentence or fact. Parameter count is best read as capacity, not as knowledge or quality.
Data quality, architecture, post-training (instruction tuning, preference tuning), context length and quantisation all change real-world results. Compare models on tasks you care about, using size mainly to predict cost and hardware needs.
Common questions
What is a parameter in an LLM, in one sentence?
It is a single learned number, such as one entry in a weight matrix, that starts at a random value and is adjusted by gradient descent during training. The model's behaviour is entirely determined by the full set of these numbers plus its fixed architecture.
Are biases counted as parameters?
Yes. Biases are learned, so they count as parameters even though they are not weights. Many modern LLMs omit them on linear layers, which is one reason people treat 'parameters' and 'weights' as the same.
What are billions of parameters doing in a language model?
They encode the transformations that turn token embeddings into a probability distribution over the next token. Most sit in attention and MLP matrices that mix and reshape information across dozens of layers, with no single parameter holding a readable fact.
Are weights vs biases vs hyperparameters really three separate groups?
Weights and biases are both learned, so both are parameters. Hyperparameters are a different category: they are configuration values chosen by people, such as learning rate or number of layers, and they are not updated by gradient descent.
Why is the downloaded file for a 7B model about 14 GB?
The file stores 7 billion numbers at 2 bytes each in fp16 or bf16, which is about 14 GB. A 4-bit quantised version of the same model shrinks to roughly 4 GB because each number takes about half a byte.