BERT CLS Token Purpose: Why Position 0 Summarises Text
The BERT CLS token purpose is to give the model a neutral slot at position 0 that attends to every token and ends up holding a sequence-level summary, which a classification head then reads. Its position does not limit what it sees.
- [CLS] is prepended to every input (id 101 in bert-base-uncased), but bidirectional self-attention lets it see all tokens at every layer.
- Because [CLS] has no word of its own, its final vector is free to aggregate sentence-level information instead of encoding one token.
- Classification uses last_hidden_state[:, 0, :] (shape [batch, 768]); BertForSequenceClassification feeds pooler_output into dropout and a linear layer.
- Raw [CLS] from a non-fine-tuned model is a weak sentence embedding; mean pooling with the attention mask is usually better for frozen similarity work.
- Decoder-only models classify from the last token, so never assume index 0 for every model or padding side.
What is the CLS token in BERT?
[CLS] (short for classification) is a special token that the BERT tokenizer prepends to every input. In bert-base-uncased it has id 101 and always sits at position 0. [SEP] (id 102) marks the end of a sentence, or separates sentence A from sentence B in a pair.
A single sentence becomes `[CLS] tokens [SEP]`. A pair becomes `[CLS] A tokens [SEP] B tokens [SEP]`, with token type ids 0 for the first segment and 1 for the second.
The token exists purely as a placeholder. Its embedding is a learned vector that is identical for every input before any attention happens, so it carries no information about your text at the input layer.
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained('bert-base-uncased')
enc = tok('I loved this film', 'It was great')
print(enc['input_ids'])
# [101, 1045, 3866, 2023, 2143, 102, 2009, 2001, 2307, 102]
print(tok.convert_ids_to_tokens(enc['input_ids']))
print(enc['token_type_ids'])If CLS is at the start, how does it know about the rest of the sentence?
Position is irrelevant to what a token can see in BERT. The encoder uses full bidirectional self-attention with no causal mask, so every token attends to every other token (except padding, which the attention mask blocks).
The walkthrough looks like this:
- Layer 0: [CLS] is just the same learned embedding plus a position embedding for slot 0. It knows nothing about the text.
- Layer 1: attention lets [CLS] take a weighted mix of the value vectors of all tokens, and all other tokens also take a weighted mix that includes [CLS].
- Layers 2 to 12: each layer re-mixes already-contextualised vectors, so information propagates further and is combined non-linearly through the feed-forward blocks.
- Final layer: the 768-dim hidden state of [CLS] (bert-base) is a context-dependent blend of the whole sequence.
Why does BERT use the CLS token for classification?
A normal token's output vector mostly encodes that word in its context, since its residual stream started from that word's embedding. Pooling from one real word would bias the summary toward it.
[CLS] has no lexical identity, so it is a neutral slot. Nothing pulls it toward a particular word, so training can shape it into an aggregator of sequence-level information.
Pretraining gave it that job. In Next Sentence Prediction (NSP), a classifier on the final [CLS] vector predicted whether sentence B actually followed sentence A, forcing [CLS] to hold sequence-level signals.
NSP is not essential, though. RoBERTa dropped NSP and still classifies well using its `<s>` token (the equivalent of [CLS]), because fine-tuning teaches the slot what it needs.
How does fine-tuning use the CLS vector, and what tensor shapes are involved?
The encoder output is a tensor of shape [batch, seq_len, hidden], for bert-base [batch, seq_len, 768]. Think of it as a cube. Selecting [CLS] is a slice at index 0 along the sequence axis: `[:, 0, :]`, giving [batch, 768].
For classification, a dropout plus a linear layer (768 to num_labels) is applied to that slice. The cross-entropy loss backpropagates through the head and all 12 layers, so [CLS] learns task-specific summarisation. The other token vectors are discarded for classification, but token-level tasks such as NER and QA use them directly.
This is why raw [CLS] from a model that was never fine-tuned is a mediocre sentence embedding: nothing ever trained it to be comparable under cosine similarity.
How do I get the CLS embedding from BERT in Hugging Face?
The snippet below shows the three common representations side by side. The mean pooling uses the attention mask so padding tokens do not dilute the average.
import torch
from transformers import AutoTokenizer, AutoModel
name = 'bert-base-uncased'
tok = AutoTokenizer.from_pretrained(name)
model = AutoModel.from_pretrained(name).eval()
texts = ['A short sentence.', 'A noticeably longer sentence with more tokens in it.']
enc = tok(texts, padding=True, truncation=True, return_tensors='pt') # add_special_tokens=True by default
with torch.no_grad():
out = model(**enc)
h = out.last_hidden_state # [batch, seq_len, 768]
cls = h[:, 0, :] # [batch, 768] raw [CLS]
pooled = out.pooler_output # [batch, 768] dense + tanh over [CLS]
mask = enc['attention_mask'].unsqueeze(-1).float() # [batch, seq_len, 1]
mean = (h * mask).sum(1) / mask.sum(1).clamp(min=1e-9) # [batch, 768]
print(cls.shape, pooled.shape, mean.shape)BERT pooler_output vs last_hidden_state CLS: what is the difference?
`last_hidden_state[:, 0, :]` is the raw final-layer vector. `pooler_output` passes that vector through an extra dense layer (768 to 768) and tanh. That pooler was trained during NSP.
`BertForSequenceClassification` uses `pooler_output`, then dropout, then a linear classifier. Models such as DistilBERT keep [CLS] but have no pooler and no NSP, so their classification heads work from the raw hidden state with their own added layers.
Note that if you load a checkpoint that lacks pooler weights, the pooler is randomly initialised, so its output is not meaningful until fine-tuned.
BERT CLS token vs mean pooling: which should I use?
Mean pooling averages all non-padding token vectors, so every token contributes. For frozen embeddings and semantic similarity it usually beats raw [CLS]. Sentence-BERT uses mean pooling by default, and sometimes max pooling. When you fine-tune end to end, the gap is usually small.
Choose using the table below.
| Situation | Use | Why |
|---|---|---|
| Fine-tuning for classification | [CLS] (or pooler_output) plus linear head | Loss shapes [CLS] into a task summary |
| Frozen BERT, similarity or clustering | Mean pooling with attention mask | Raw [CLS] was never trained for cosine geometry |
| Semantic search or retrieval | Purpose-trained embedding model | Trained contrastively for this exact use |
| NER, QA, token tagging | Per-token last_hidden_state | Each token needs its own vector |
| Decoder-only model classification | Last non-pad token | Causal attention: only the last token sees everything |
What mistakes do people make with the CLS token?
These are the pitfalls most tutorials skip:
| Mistake | Effect | Fix |
|---|---|---|
| Cosine similarity on raw [CLS] without fine-tuning | Poor, noisy similarity scores | Use mean pooling or a sentence-embedding model |
| add_special_tokens=False | No [CLS] or [SEP]; index 0 is a real word | Keep the default True |
| Averaging over padding | Embeddings depend on batch padding length | Multiply by attention_mask and divide by its sum |
| Assuming index 0 for every model | Wrong vector with left padding or decoders | Use the model's convention; for GPT-style, the last non-pad token |
| Using pooler_output from a checkpoint without trained pooler | Random-layer output | Use last_hidden_state[:, 0] or fine-tune |
Common questions
Is the CLS token at the beginning of the sentence in BERT?
Yes, the tokenizer always places it at position 0, before the first word. For a sentence pair it appears only once, at the very start, while [SEP] appears after each segment.
Does the CLS token have its own meaning?
No. Its input embedding is the same learned vector for every input, so it encodes no word. Any meaning in its output comes from attention over the actual tokens plus what pretraining and fine-tuning taught it.
Do I need the CLS token if I use mean pooling?
The tokenizer still adds it and the model still processes it, and it takes part in attention. When mean pooling, you can include or exclude it; both are common, and the difference is typically small.
Why did RoBERTa drop NSP if CLS was trained with it?
Experiments in the RoBERTa work found that removing NSP did not hurt, and could help, downstream results. The <s> token still becomes a good summary because fine-tuning supplies the supervised signal.
Can I fine-tune BERT with mean pooling instead of CLS?
Yes. Compute the masked mean of last_hidden_state and put your dropout and linear head on it. Results are usually close to the [CLS] approach, so pick whichever matches your downstream embedding needs.