Why BERT Has No Decoder and What to Use for Generation
Why BERT has no decoder: it is an encoder-only Transformer built to understand text, not generate it. Predictions come from small task heads on its outputs, such as a fill-in-the-blank MLM head or a classifier on [CLS].
- BERT keeps only the Transformer encoder stack (12 layers base, 24 large) and drops the decoder and cross-attention.
- Predictions come from task heads on a [batch, seq, 768] tensor: an MLM head scores the vocabulary at [MASK] positions, a classifier reads [CLS], and a QA head predicts start and end logits.
- Bidirectional attention lets every token see its right context, so BERT cannot generate left to right the way a causal decoder does.
- For generation use a decoder-only LLM, T5 or BART, or fine-tune a BERT2BERT EncoderDecoderModel if you must start from BERT weights.
- In practice the best pattern is often an encoder for retrieval and classification paired with a decoder model for the text it writes.
Is BERT encoder only, and why is there no decoder?
Yes. BERT stands for Bidirectional Encoder Representations from Transformers, and it is exactly the encoder stack of the 2017 Transformer: 12 layers in BERT-base (hidden size 768, 12 heads) and 24 layers in BERT-large (hidden size 1024). There is no decoder stack and no cross-attention block, because nothing in BERT's training needs one.
In the original encoder-decoder design, the decoder exists to emit a variable-length output one token at a time. It uses causal masking so position t cannot see positions after t, and cross-attention so it can read the encoder's representation of the input. That machinery is only useful when the output is a new sequence.
BERT's goal was different: produce good contextual representations of an existing sequence, then reuse them for many downstream tasks. Its pretraining tasks (masked language modelling and next sentence prediction) predict things about the input itself, so a decoder would add cost without adding capability.
The Transformer did not abolish decoders. The original design had two halves, and the families that followed picked one or both: BERT, RoBERTa and DeBERTa took the encoder, GPT and Llama took the decoder, and T5 and BART kept both.
What actually comes out of BERT?
BERT returns one vector per input token, not text. For a batch of 2 sentences padded to 16 tokens, BERT-base outputs a last hidden state of shape [2, 16, 768]. Each 768-dimensional vector is that token's meaning in context, so "bank" gets different vectors in "river bank" and "bank loan".
It also returns a pooled output of shape [2, 768], which is the [CLS] hidden state passed through a dense layer and tanh. That vector was trained for next sentence prediction and is often a mediocre sentence embedding unless you fine-tune.
These vectors are not reversible into words. Nothing maps a hidden state back to a token except a head that projects onto the vocabulary, and that head only exists for the masked language modelling task.
from transformers import AutoTokenizer, AutoModel
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")
batch = tok(["The bank was closed.", "We sat on the river bank."],
padding="max_length", max_length=16, return_tensors="pt")
out = model(**batch)
print(out.last_hidden_state.shape) # torch.Size([2, 16, 768])
print(out.pooler_output.shape) # torch.Size([2, 768])How does BERT make predictions without a decoder?
Every prediction comes from a small task head stacked on the encoder output. The encoder does the heavy lifting; the head turns vectors into the answer format you need. Each head is a few linear layers, so swapping tasks means swapping the head, not adding a decoder.
The MLM head is the one that comes closest to "decoding". It applies a linear layer, GELU, LayerNorm, then a projection to the vocabulary (30,522 tokens for bert-base-uncased) whose weights are tied to the input embedding matrix. A softmax at each [MASK] position gives a distribution over tokens, and you take the argmax or top-k, then call tokenizer.decode on the ids.
| Task | Head | Reads from | Output |
|---|---|---|---|
| Masked language modelling | Linear + GELU + LayerNorm + vocab projection (tied embeddings) | Hidden state at each [MASK] | [batch, seq, 30522] logits |
| Sentence or pair classification, NSP | Dropout + linear on pooled [CLS] | [CLS] vector | [batch, num_labels] logits |
| Token classification (NER, POS) | Linear per token | Every token vector | [batch, seq, num_labels] |
| Extractive QA | Linear to 2 outputs per token | Every token vector | start and end logits, each [batch, seq] |
How does masked language modelling work, and why is it not generation?
During pretraining, 15% of tokens are selected for prediction. Of those, 80% are replaced with [MASK], 10% with a random token and 10% left unchanged. The mixed recipe stops the model from learning that only [MASK] positions matter, since [MASK] never appears at fine-tuning time.
The loss is computed only at the selected positions, and the model sees the whole sentence in one forward pass. It is fill-in-the-blank, not writing: there is no loop in which each output feeds the next step, so no autoregressive decoding is involved.
This is the key architectural reason. BERT uses bidirectional self-attention, with no mask on future positions. A decoder-only model uses a causal mask, so token t only attends to tokens up to t, which makes next-token prediction well defined. If a token can already see its right neighbours, there is nothing left to "generate" and no natural left-to-right order.
from transformers import pipeline, AutoTokenizer, AutoModelForMaskedLM
import torch
# Quick route
fill = pipeline("fill-mask", model="bert-base-uncased")
print(fill("The capital of France is [MASK].")[:3])
# Manual route: this is the only 'decoding' BERT does
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
mlm = AutoModelForMaskedLM.from_pretrained("bert-base-uncased")
enc = tok("The capital of France is [MASK].", return_tensors="pt")
with torch.no_grad():
logits = mlm(**enc).logits # [1, seq, 30522]
mask_pos = (enc.input_ids == tok.mask_token_id).nonzero()[0, 1]
top_id = logits[0, mask_pos].argmax().item()
print(tok.decode([top_id])) # 'paris'How do I use BERT for classification, tagging and question answering?
Use the Auto class that attaches the right head. The head is randomly initialised, so you must fine-tune it on labelled data; the warning "Some weights were not initialized" is expected, not an error.
Typical fine-tuning settings for BERT are a learning rate of 2e-5 to 5e-5, 2 to 4 epochs, batch size 16 or 32, and a maximum length of 128 to 512 tokens. BERT's position embeddings stop at 512, so longer documents need chunking.
- Pick the class: AutoModelForSequenceClassification for labels per text, AutoModelForTokenClassification for NER, AutoModelForQuestionAnswering for span extraction.
- Load it with num_labels set where relevant, for example from_pretrained('bert-base-uncased', num_labels=3).
- Tokenise with truncation=True and a max_length, and align word-level labels to subword tokens for tagging, usually labelling the first subword and ignoring the rest with -100.
- Fine-tune with the Trainer API or a plain PyTorch loop using AdamW and linear warmup.
- For QA, decode the answer by taking argmax of start and end logits and slicing the input ids between them.
from transformers import (AutoModelForSequenceClassification,
AutoModelForTokenClassification,
AutoModelForQuestionAnswering)
cls = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=3)
ner = AutoModelForTokenClassification.from_pretrained("bert-base-uncased", num_labels=9)
qa = AutoModelForQuestionAnswering.from_pretrained("bert-base-uncased")How can I generate text with BERT, and which model should I use instead?
Usually you should not. BERT can be forced to generate by repeatedly appending [MASK] and filling it, or by Gibbs-style resampling, and research variants such as Mask-Predict refine all positions over several rounds. In practice, vanilla BERT produces repetitive, less fluent text, and it was never trained to choose a length or stop.
If you need generation, switch model family. Choose by the shape of your task, not by brand.
| Family | Examples | Attention | Best for |
|---|---|---|---|
| Encoder-only | BERT, RoBERTa, DeBERTa | Bidirectional | Classification, NER, QA spans, embeddings for search |
| Decoder-only | GPT, Llama, most chat LLMs | Causal | Open-ended generation, chat, code, in-context tasks |
| Encoder-decoder | T5, BART, mBART | Bidirectional encoder, causal decoder with cross-attention | Translation, summarisation, structured rewriting |
Can I use BERT as a decoder (BERT2BERT)?
Yes, with caveats. Hugging Face's EncoderDecoderModel can initialise both sides from BERT checkpoints. The decoder copy gets a causal mask and newly added cross-attention layers, and those cross-attention weights are random, so you must fine-tune on paired data such as article and summary.
Expect it to work as a baseline, but T5, BART or a modern LLM typically gives better quality for the same effort, because their decoders were pretrained to generate. Use BERT2BERT mainly when you need to reuse a specific language or domain BERT checkpoint, for example a monolingual model with no good generative equivalent.
from transformers import EncoderDecoderModel, AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
model = EncoderDecoderModel.from_encoder_decoder_pretrained(
"bert-base-uncased", "bert-base-uncased" # decoder gets is_decoder=True + cross-attention
)
model.config.decoder_start_token_id = tok.cls_token_id
model.config.pad_token_id = tok.pad_token_id
model.config.eos_token_id = tok.sep_token_id
# Fine-tune on (source, target) pairs first, then:
ids = tok("Long input text ...", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=48, num_beams=4)
print(tok.decode(out[0], skip_special_tokens=True))Common questions
Can BERT embeddings be turned back into text?
Not in general. Hidden states are contextual vectors with no inverse mapping to tokens, and [CLS] or pooled vectors compress the whole sequence into 768 numbers. Only the MLM head projects states at masked positions to vocabulary logits, and you would need a trained decoder to reconstruct full sentences.
Is the [CLS] token a summary I can decode?
No. [CLS] is a learned position whose final state is shaped by the task head, originally next sentence prediction. It works as input to a classifier, but it is not designed to be reversible, and unfine-tuned it is a weak sentence embedding compared with models trained for that purpose.
Why did GPT choose a decoder while BERT chose an encoder?
They optimise different objectives. GPT predicts the next token, which needs a causal mask so it cannot peek ahead, while BERT predicts masked tokens using both sides, which needs unmasked bidirectional attention. The objective dictates the attention pattern, and the attention pattern dictates the architecture.
Is BERT better than GPT?
Neither is better overall. Encoder models like BERT are smaller, cheaper and strong for classification, tagging and embeddings, while decoder LLMs are the right tool for generation and flexible instruction following. Pick based on whether your output is a label or span, or new text.
Does a decoder-only model like GPT also have no cross-attention?
Correct. GPT-style models keep only causal self-attention layers and treat the prompt as a prefix of the same sequence, so there is no separate encoder to attend to. Cross-attention exists only in encoder-decoder models such as T5 and BART.