Llama 2 Repeats Context and Newlines: Causes and Fixes

Why does Llama 2 copy the context and then print newlines?

Two separate problems produce this one symptom. The first is the model: meta-llama/Llama-2-7b-hf is the pretrained base model. It was trained only to predict the next token, so given a prompt that ends with a retrieved chunk and a question, the most likely continuation is more document-like text, often the chunk itself.

The second is stopping. A base model has no learned habit of ending a reply. If it never produces the end-of-sequence token (</s>, id 2), generate() keeps going until max_new_tokens, and the cheapest filler is newline tokens. That is the trailing wall of blank lines.

Both are fixable without touching temperature. Tuning sampling on the base model only changes which wrong continuation you get. The fix is the right checkpoint, the right prompt format, and explicit stop settings.

Quick diagnostic: symptom, cause, fix

Use this table to map what you see to the likely cause before changing anything.

How do I use the right model and the Llama 2 chat prompt format?

Llama-2-7b-chat-hf was fine-tuned with supervised data and RLHF on a specific format, and it learned to end its turn with </s>. The format is:

<s>[INST] <<SYS>>\nsystem prompt\n<</SYS>>\n\nuser message [/INST]

Do not hand-write this. Spacing mistakes quietly degrade quality, and the most common bug is a doubled <s>: the template already inserts the BOS token, and the tokenizer adds another if you tokenize the rendered string with default settings. Either let apply_chat_template tokenize directly, or pass add_special_tokens=False when encoding the rendered text.

The model access is gated on Hugging Face, so you must accept Meta's license and authenticate before downloading.

from transformers import AutoTokenizer

name = "meta-llama/Llama-2-7b-chat-hf"
tok = AutoTokenizer.from_pretrained(name)

messages = [
    {"role": "system", "content": "You answer using only the provided sources."},
    {"role": "user", "content": "What is MMR?"},
]
# Inspect the exact string once:
print(tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))
# Real use: tokenize=True gives ids with ONE bos
input_ids = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
)

How do I make Llama 2 stop generating?

Generation ends when the model emits any id in eos_token_id, or when a stopping criterion fires. Set the ids explicitly so nothing depends on config defaults, and give the tokenizer a pad id, because Llama-2 ships without one. For inference, pad equal to eos or unk is fine; do not use eos as pad during fine-tuning, because the loss mask then hides the real end-of-sequence labels and the model learns never to stop.

Also remove the echo of the prompt. A pipeline returns prompt plus completion by default, which looks like the model repeating its input. Use return_full_text=False, or when calling generate() decode only the tokens after the input length.

If a model still rambles after its answer, add stop strings. Recent transformers versions accept stop_strings=[...] together with tokenizer=tok in generate(); older ones need a custom StoppingCriteria.

import torch
from transformers import AutoModelForCausalLM, StoppingCriteria, StoppingCriteriaList

model = AutoModelForCausalLM.from_pretrained(
    name, torch_dtype=torch.float16, device_map="auto"
)
tok.pad_token_id = tok.unk_token_id  # or tok.eos_token_id for inference

class StopOnText(StoppingCriteria):
    def __init__(self, tok, text, start):
        self.tok, self.text, self.start = tok, text, start
    def __call__(self, ids, scores, **kw):
        tail = self.tok.decode(ids[0][self.start:], skip_special_tokens=True)
        return self.text in tail

ids = input_ids.to(model.device)
out = model.generate(
    ids,
    max_new_tokens=400,
    do_sample=True, temperature=0.3, top_p=0.9,
    repetition_penalty=1.05,
    eos_token_id=tok.eos_token_id,
    pad_token_id=tok.pad_token_id,
    stopping_criteria=StoppingCriteriaList([StopOnText(tok, "[INST]", ids.shape[1])]),
)
answer = tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True)

Which sampling settings actually help, and why doesn't repetition_penalty fix copying?

For RAG on a 7B model, start with do_sample=True, temperature 0.1 to 0.7 (lower for factual answers), top_p 0.9, and repetition_penalty between 1.0 and 1.15. Temperature and top_p are ignored when do_sample=False, so setting temperature=0.2 with greedy decoding changes nothing and only prints a warning.

The penalty divides the scores of every token already in the sequence, and that includes your prompt. In RAG the correct answer legitimately reuses names, numbers and terms from the retrieved text, so a penalty of 1.3 or more pushes the model away from the very tokens it needs, producing paraphrase gibberish. It also does not address the cause: a base model copies because continuing the document is its job, not because of sampling.

Illustrative example, not a benchmark result: these are patterns to expect, not measurements from a controlled test, and exact output varies with prompt, model version and sampling seed. With a base model and penalty 1.0, the output often repeats the chunk and then trails off into newlines. Raising the penalty to around 1.4 on the same base model tends to give a distorted version of the chunk, still followed by newlines. A chat model with a proper template and a penalty near 1.05 is far more likely to give a short synthesized answer that ends cleanly with </s>. Run the comparison on your own prompts to confirm.

How should I structure a RAG prompt and retrieval for Llama 2?

Plain similarity search returns the k nearest chunks, which for a summary-style question are often near duplicates from overlapping windows. The model sees the same text three times and copies it. Use MMR (maximal marginal relevance), which fetches a larger candidate pool and picks results that are relevant yet different from each other. Then tune chunk size (500 to 1000 characters is common for 7B) and overlap (10 to 15 percent of chunk size).

In the prompt, put the instruction in the system message, label each chunk, and place the question last, since small models weight the end of the prompt heavily. Tell it explicitly to synthesize and to admit when the context lacks the answer.

Here is a complete example with LangChain and Chroma. Package paths vary between LangChain versions, so adjust imports if yours differ.

from langchain_chroma import Chroma

# vectordb = Chroma(persist_directory="db", embedding_function=emb)
retriever = vectordb.as_retriever(
    search_type="mmr",
    search_kwargs={"k": 4, "fetch_k": 20, "lambda_mult": 0.5},
)

SYSTEM = (
    "You are a careful assistant. Answer using ONLY the sources below. "
    "Synthesize across all relevant sources in your own words; do not copy "
    "passages. If the answer is not in the sources, say 'I don't know based "
    "on the provided documents.'"
)

def ask(question, max_new_tokens=400):
    docs = retriever.invoke(question)
    ctx = "\n\n".join(
        f"### Source {i+1}\n{d.page_content}" for i, d in enumerate(docs)
    )
    messages = [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": f"{ctx}\n\n### Question\n{question}"},
    ]
    ids = tok.apply_chat_template(
        messages, add_generation_prompt=True, return_tensors="pt"
    ).to(model.device)
    assert ids.shape[1] + max_new_tokens <= 4096, "prompt too long"
    out = model.generate(
        ids, max_new_tokens=max_new_tokens,
        do_sample=True, temperature=0.2, top_p=0.9, repetition_penalty=1.05,
        eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id,
    )
    return tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True).strip()

Context window, loading, and LangChain wiring

Llama 2 has a 4096-token window shared by prompt and output. Four chunks of 1000 characters is roughly 1000 tokens, which is safe, but k=10 with large chunks is not. If the total exceeds the limit and your pipeline truncates from the left, the system instruction is the part that gets cut, and the model reverts to copying. Count tokens with the tokenizer and reserve room for max_new_tokens.

Load in float16 with device_map='auto'; the 7B model needs about 14 GB of VRAM. On smaller GPUs use bitsandbytes 4-bit or 8-bit via BitsAndBytesConfig (for example load_in_4bit=True with bnb_4bit_compute_dtype=torch.float16). Quantization saves memory but can slightly lower answer quality, so test on your own questions before blaming the prompt.

In LangChain, HuggingFacePipeline wraps a transformers pipeline; pass generation options through pipeline_kwargs (max_new_tokens, do_sample, temperature, eos_token_id, return_full_text=False). With a raw pipeline you must supply the formatted string yourself, so render it with apply_chat_template(tokenize=False) and feed that, ensuring nothing else adds [INST] or a second BOS. ChatHuggingFace applies the model's template for you, so give it plain messages instead.

If you must stay on the base model, add a few worked examples (context, question, answer) and end the prompt with 'Answer:'. Combine this with a stop string such as '\n\n' or 'Question:'. It is fragile, and the chat model is a better use of your time.

What if answers are still poor?

A 7B model from 2023 is weak at multi-document synthesis, even when correctly prompted. Verify the pipeline first: print the final decoded prompt, confirm one <s>, confirm the question is last, confirm the output ends via EOS rather than the token limit (check whether the last generated id equals 2).

If quality is still low, upgrade rather than tune. Instruct models such as Llama 3.x Instruct or Mistral Instruct follow RAG prompts and long contexts far better, and larger Llama-2 variants (13B, 70B) improve synthesis. Keep the same code, but use the new model's own chat template.

1. Switch to the chat checkpoint and load it in float16.
2. Render prompts with apply_chat_template and print one to inspect it.
3. Set eos_token_id, pad_token_id and max_new_tokens; slice off the prompt.
4. Switch retrieval to MMR and label sources.
5. Check token counts and test with a question whose answer is absent.