LangChain ConversationalRetrievalChain Custom Prompt and Memory

Which chain should I use: RetrievalQA, ConversationalRetrievalChain or a history-aware chain?

The confusion comes from three generations of API that all look similar. The choice depends on whether you need follow-up questions to work and whether you are writing new code.

Short version: use the history-aware retrieval chain for anything new, keep ConversationalRetrievalChain only to maintain existing code, and use RetrievalQA only when every question stands alone.

Why do I get memory or a custom prompt, but not both?

RetrievalQA is stateless: one query goes in, it retrieves, stuffs documents into a prompt, and returns an answer. It has no notion of earlier turns, so a custom prompt works but nothing is remembered.

ConversationalRetrievalChain adds a first step. An LLM call takes the chat history plus the new question and produces a standalone question, which is then used for retrieval. Only then does the answering step run.

That means two prompts exist, and people often edit the wrong one. The condense_question_prompt uses {chat_history} and {question} and only rewrites the query. The QA prompt (the combine-docs prompt) uses {context} and {question} and produces the visible answer. Your tone, language and 'only answer from context' rules belong in the QA prompt.

How do I add memory and a custom prompt to ConversationalRetrievalChain?

This is the legacy fix and it still works in versions that ship the old classes. In newer releases the legacy chains may live in the langchain-classic package, so adjust imports if you see ImportError.

The QA prompt must use exactly {context} and {question} for the default stuff chain. Other names raise a validation error about missing or extra input variables, or the context silently never reaches the model.

from langchain.chains import ConversationalRetrievalChain
from langchain.memory import ConversationBufferMemory
from langchain.prompts import PromptTemplate

QA_PROMPT = PromptTemplate(
    input_variables=['context', 'question'],
    template=(
        'You are a support assistant. Answer ONLY from the context below. '
        'If the answer is not in the context, say "I don\'t know" and do not guess.\n\n'
        'Context:\n{context}\n\nQuestion: {question}\nAnswer:'
    ),
)

memory = ConversationBufferMemory(
    memory_key='chat_history',   # must match the condense prompt
    return_messages=True,
    output_key='answer',         # needed when returning source documents
)

qa = ConversationalRetrievalChain.from_llm(
    llm,
    retriever=vectorstore.as_retriever(search_kwargs={'k': 4}),
    memory=memory,
    combine_docs_chain_kwargs={'prompt': QA_PROMPT},
    return_source_documents=True,
    verbose=True,
)

print(qa.invoke({'question': 'What is the refund window?'})['answer'])
print(qa.invoke({'question': 'And for digital items?'})['answer'])

What is the current way: create_history_aware_retriever with a custom prompt?

The modern pattern splits the same two steps into explicit pieces. create_history_aware_retriever holds the rewrite prompt, create_stuff_documents_chain holds the QA prompt, and create_retrieval_chain joins them. Chat history is just a list of messages.

The QA prompt needs {context} (filled with documents) and {input} (the user message), plus a MessagesPlaceholder for history. The rewrite prompt needs a placeholder for history and {input}.

from langchain.chains import create_history_aware_retriever, create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.runnables.history import RunnableWithMessageHistory

retriever = vectorstore.as_retriever(search_kwargs={'k': 4})

# 1) rewrite prompt: follow-up -> standalone question
condense_prompt = ChatPromptTemplate.from_messages([
    ('system', 'Given the chat history and the latest user message, rewrite it as a '
               'standalone question. Do NOT answer it. Return it unchanged if already standalone.'),
    MessagesPlaceholder('chat_history'),
    ('human', '{input}'),
])
history_aware_retriever = create_history_aware_retriever(llm, retriever, condense_prompt)

# 2) QA prompt: your custom system prompt
qa_prompt = ChatPromptTemplate.from_messages([
    ('system', 'Answer only from the context. If it is not there, say "I don\'t know".\n\n{context}'),
    MessagesPlaceholder('chat_history'),
    ('human', '{input}'),
])
qa_chain = create_stuff_documents_chain(llm, qa_prompt)
rag = create_retrieval_chain(history_aware_retriever, qa_chain)

# 3) memory keyed by session
store = {}
def get_history(session_id: str):
    return store.setdefault(session_id, InMemoryChatMessageHistory())

chat = RunnableWithMessageHistory(
    rag, get_history,
    input_messages_key='input',
    history_messages_key='chat_history',
    output_messages_key='answer',
)

cfg = {'configurable': {'session_id': 'user-42'}}
chat.invoke({'input': 'What is the refund window?'}, config=cfg)
res = chat.invoke({'input': 'And for digital items?'}, config=cfg)
print(res['answer'])

Can I add memory to RetrievalQA, and what goes wrong?

Yes, mechanically. Put {history} in the prompt and pass the memory through chain_type_kwargs. The input_key must be 'question' so the memory knows which field is the user turn.

The catch is retrieval. RetrievalQA embeds the raw user text. A follow-up such as 'what about the second one?' has no searchable meaning on its own, so the retriever returns irrelevant chunks and the model, which can see the history but not the right documents, either hallucinates or says it does not know.

Memory in the prompt fixes how the answer reads, not what gets retrieved. Only a rewrite step fixes retrieval, which is exactly what ConversationalRetrievalChain and the history-aware retriever add.

from langchain.chains import RetrievalQA
from langchain.memory import ConversationBufferMemory
from langchain.prompts import PromptTemplate

prompt = PromptTemplate(
    input_variables=['history', 'context', 'question'],
    template='History:\n{history}\n\nContext:\n{context}\n\nQuestion: {question}\nAnswer:',
)
qa = RetrievalQA.from_chain_type(
    llm, retriever=retriever,
    chain_type_kwargs={
        'prompt': prompt,
        'memory': ConversationBufferMemory(input_key='question', memory_key='history'),
    },
)

What do the common errors mean and how do I fix them?

Most failures come from variable names and output keys, not from the model.

How should I choose a memory type and debug the final prompt?

Every stored turn is re-sent on each call, so cost and latency grow with conversation length, and eventually you hit the context window. Pick the memory that fits the conversation shape.

Buffer keeps everything: simplest and exact, but unbounded. Window keeps the last k turns (k of 3 to 6 is a common starting point): predictable cost, but loses early facts. Summary compresses old turns with an extra LLM call: bounded size, but adds cost and can drop details. In the modern API, windowing is just slicing the message list or using trim_messages before the call.

To debug, first print what the model actually sees. In the legacy chain set verbose=True. In the modern chain, attach a callback or use LangSmith tracing, or run the pieces separately.

1. Invoke the history_aware_retriever alone with a sample history and a vague follow-up, and read the documents returned.
2. Check that the rewritten standalone question names the subject explicitly; if it does not, tighten the condense prompt with 'do not answer, only rewrite'.
3. Print the formatted QA prompt (qa_prompt.invoke({...})) and confirm {context} is populated.
4. Open two sessions with different IDs and confirm neither sees the other's turns.
5. Lower temperature to 0 for the rewrite step if the standalone questions drift.