Parameters, Weights and Layers: Counting What a 7B Model Means
Where seven billion numbers live, and why the label on a model tells you its memory bill

Contents
Model names come with a number attached: 7B, 70B, 405B. The letter is a billion, and the thing being counted is parameters. People quote the figure as a stand-in for capability, price and hardware requirements all at once, which is a lot to ask of one digit.
The number does carry real information, and you can check it yourself. A parameter count is a sum, and the sum comes from a handful of choices that a model’s designers wrote down in a config file. In this post I will total a 7B model from scratch with arithmetic you can do on paper, then use the result to explain memory, file sizes and why the count alone misleads.
If the phrase “file of numbers” is new, start with the opener of the series, What a Model Actually Is. This post assumes only that.
Parameter, weight: the vocabulary
In everyday use parameter and weight are interchangeable: both mean one learnable number inside the model. Strictly, “weights” often refers to the multiplicative values in the big grids and “biases” to the added offsets, and “parameters” covers both. Modern language models mostly drop biases, so in practice nearly all parameters are weights.

A layer is one repeated block of the network. A tensor is a named grid of parameters inside it. The model is a stack of identical layers, each holding the same set of tensors with its own learned values. That repetition is what makes the count computable: figure out one layer, multiply by the depth, add the entry and exit pieces.
The config that fixes the count
Mistral 7B is a convenient case, because its paper states the sizes in a table. The ones that matter are:
- model width (the hidden dimension): 4,096
- layers: 32
- feed-forward hidden size: 14,336
- attention heads: 32, with 8 key-value heads
- vocabulary: 32,000 tokens
These come from the paper’s architecture table, and the sums below reproduce the published headline of seven billion, which is the check that matters. The config.json in the model’s repository on the hub holds the same fields, so you can verify them against the real file.
Adding it up
Each layer has two sub-blocks: attention and the feed-forward network.
Attention. Four grids. The query and output projections are each 4,096 by 4,096, about 16.8 million numbers apiece. With grouped-query attention, the key and value projections are narrower, because there are only 8 key-value heads of 128 dimensions each, giving 4,096 by 1,024: about 4.2 million apiece. The attention block totals roughly 41.9 million.
Feed-forward. This block widens the vector, applies a non-linearity, and narrows it again. The gated variant used by Mistral and Llama has three grids of 4,096 by 14,336, which is about 58.7 million each, so 176 million for the block.
One layer is therefore about 218 million parameters, and the feed-forward block is the larger share by a wide margin. That ratio, roughly four-fifths of each layer in the feed-forward part, holds for most models in this family and is worth remembering. People sometimes assume attention, the famous part, holds most of the numbers. It holds a fifth.
The ten lines of arithmetic, as Python:
| |
The output is 218M per layer and 7.24B in total. Small normalisation vectors add a few hundred thousand more, which is rounding noise at this scale. The sum agrees with the “7 billion” in the paper’s abstract, which is why the model gets called 7B even though the exact figure is a little over.
Notice the two ends. The embedding table, which turns a token ID into a vector, holds 131 million parameters: 32,000 rows of 4,096 numbers. Vectors and Embeddings covers what those rows mean. The output projection mirrors it. Models with larger vocabularies spend noticeably more here, which is one reason small models with big vocabularies can feel top-heavy.
One wrinkle worth knowing: some models tie the two ends together, reusing the embedding table as the output projection. That saves 131 million parameters here and matters more in small models, where the vocabulary tables are a large share of the total. Mistral keeps them separate, which the script above reflects, and a model’s config usually has a flag (often named tie_word_embeddings) that says which choice was made. If your own sum lands a few hundred million off the published figure, tying and rounding are the first places to look.
From parameters to gigabytes
Every parameter is stored as a number with a fixed width, so memory is multiplication.
| Precision | Bytes per number | 7B model | 70B model |
|---|---|---|---|
| 16-bit (bf16/fp16) | 2 | ~14.5 GB | ~140 GB |
| 8-bit | 1 | ~7 GB | ~70 GB |
| 4-bit | ~0.5 | ~3.6 GB plus overhead | ~35 GB plus overhead |
Those are arithmetic from the counts, with real quantised files landing a little above the 4-bit line because some tensors and the scale factors are kept at higher precision. Treat them as floors.
Two things get added on top at run time. The first is the KV cache, the stored keys and values of every token processed so far, which grows with the length of the conversation. The context window cost post covers it. The second is working memory for the runtime itself. A model that fits on paper may not fit once you ask it to read a long document.
This is why 7B is the size people reach for first at home. A 14 GB file runs on a typical gaming-class GPU or a Mac with enough unified memory, and a 4-bit version fits on far less. I’ve run Mistral and Llama this way on an Apple Silicon Mac and on an NVIDIA GPU box, and the ability to pick precision to match the machine is the whole trick.
Where the famous models sit
The chart plots published parameter counts on a logarithmic axis, because the range is too wide for a linear one: the largest bar is over a thousand times the smallest.
A few things stand out.
The climb was fast. BERT-large had 340 million parameters in 2018. GPT-2 reached 1.5 billion in 2019. GPT-3 arrived in 2020 at 175 billion, which its authors described as ten times more than any previous non-sparse language model. Three years took the field from hundreds of millions to hundreds of billions.
Bigger stopped being the only goal. The Chinchilla result in 2022 found that models of the era were significantly undertrained; its 70-billion-parameter model was trained on far more tokens than comparable larger ones. Parameter count describes capacity. How well that capacity got filled depends on the data.
Small models caught up. Mistral’s paper states that its 7B model outperforms Llama 2 13B across all evaluated benchmarks, which is a claim that a smaller count trained well can beat a larger one trained less well. Llama 2 itself spans 7 to 70 billion, and Llama 3.1 pushed the top to 405 billion with a 128K context window.
Total and active counts diverge. Mixtral 8x7B holds 47 billion parameters but uses 13 billion per token, since a router sends each token to only two of eight expert feed-forward blocks. You pay disk and memory for 47 billion and arithmetic for 13 billion. The name “8x7B” misleads here: eight times seven would be 56, but the experts share the attention layers and other parts, so the true total is lower.
What the count does and does not tell you
The count predicts cost. It tells you about how much memory the file needs, how much arithmetic each token takes (roughly two operations per active parameter) and therefore how fast the model can run on given hardware.
It predicts quality only loosely. Within a single family trained the same way, bigger models do better. Across families, the training data, the training duration, the architecture tweaks and the post-training all move the result. Mistral’s own result, a 7B model ahead of Llama 2 13B on every benchmark the paper ran, shows the effect, so “70B beats 7B” holds within a generation of models and may fail across generations.
It also says nothing about what the parameters are good at. Two models with the same count can differ in how much of that capacity goes to code, to other languages or to long documents.
Troubleshooting the number
A few situations where the figure confuses people.
The file is bigger than the label implies. A “7B” file at 14.5 GB is correct for 16-bit. If you see 29 GB, the weights are stored in 32-bit, common for training checkpoints. Download the half-precision release instead.
The count differs between sources. Some quote the total, some the active count, some round down. Mixtral and other mixture-of-experts models are the usual culprits. Check which figure a source means.
The model fits but crashes. The weights fit; the cache and runtime overhead did not. Shorten the context or pick a lower-precision file.
Two “7B” models give different file sizes. Their exact counts differ (6.7B, 7.2B, 8B all get rounded to “7B” or “8B”), and vocabulary size alone can move the total by hundreds of millions.
The verdict
Learn to read “7B” as a sentence: roughly seven billion numbers, about 14 GB at full precision, about 4 GB squeezed, arithmetic proportional to seven billion per token. It is a cost label first and a capability hint second. Attention gets the attention, yet each layer’s feed-forward block holds most of the numbers, and the two ends of the network hold a surprising amount more.
Next in the series: Vectors and Embeddings, the entry point where words turn into the numbers all this arithmetic then operates on.
Sources: Mistral 7B (Jiang et al., 2023) · BERT repository, Google Research · GPT-2 model card, OpenAI · Language Models are Few-Shot Learners (Brown et al., 2020) · Chinchilla (Hoffmann et al., 2022) · Llama 2 and Llama 3 papers · Mixtral of Experts




