LangChain CharacterTextSplitter chunk_size Not Working: Fix

Why is LangChain CharacterTextSplitter chunk_size not working?

It is not ignoring it, but chunk_size does not mean what most docs summaries and chatbots say. CharacterTextSplitter first splits the text on one separator, then merges neighbouring pieces until adding the next piece would go past chunk_size. It never splits inside a piece.

So if a piece is already bigger than chunk_size, nothing in the algorithm can shorten it. It is emitted whole. The setting limits how much gets merged, not how long a finished chunk can be.

The most common trigger is a text that contains no occurrence of the separator. The default separator is "\n\n" (a blank line), so single-line text, minified JSON or HTML, PDF extractions with single newlines, and logs all become one huge piece.

How does the merge algorithm work, step by step?

The logic lives in the shared merge routine. Here is what happens for text split into pieces, with a running total that counts piece lengths plus separator lengths:

1. Split the text on the separator (escaped to a plain string unless is_separator_regex=True). Whitespace around chunks is stripped by default.
2. Take the next piece. If total + len(piece) + separator length would exceed chunk_size, close the current chunk first.
3. If the group being closed is already longer than chunk_size, log the warning "Created a chunk of size X, which is longer than the specified Y".
4. Join the closed group with the separator and emit it.
5. Drop pieces from the front of the group while its length is above chunk_overlap (or it still would not fit the next piece). The survivors become the overlap.
6. Add the new piece and repeat. At the end, emit whatever remains.

What does a minimal reproducible example look like?

Three 30-character paragraphs with chunk_size=70 show the merge behaviour. A+B joins to 62 characters (30 + 2 + 30). Adding C would make 94, so the chunk closes there. The second example has no blank lines at all, so the whole text is one piece.

# pip install langchain-text-splitters
from langchain_text_splitters import CharacterTextSplitter

sp = CharacterTextSplitter(separator="\n\n", chunk_size=70, chunk_overlap=0)

text = "A"*30 + "\n\n" + "B"*30 + "\n\n" + "C"*30
print([len(c) for c in sp.split_text(text)])
# [62, 30]   (A+B merged, C alone)

# No "\n\n" anywhere: one piece, nothing to merge or cut
flat = "word " * 100          # 500 chars, single line
sp = CharacterTextSplitter(chunk_size=100, chunk_overlap=0)
print([len(c) for c in sp.split_text(flat)])
# [499]   (trailing space stripped)

# Oversized piece followed by another piece triggers the warning
mixed = flat + "\n\n" + "tail"
print([len(c) for c in sp.split_text(mixed)])
# WARNING: Created a chunk of size 500, which is longer than the specified 100
# [499, 4]

How do separator, chunk_overlap and length_function change the result?

Separator: it is a plain string by default, so separator="\n" or separator=" " works directly. A regex such as r"\n{2,}" or r"(?<=\.) " only behaves as a regex if you pass is_separator_regex=True. Otherwise it is escaped and searched for literally, which usually matches nothing and gives you one big chunk.

chunk_overlap: overlap is built from the trailing pieces of the chunk just closed, not from an arbitrary character window. Pieces are kept only while their total is at most chunk_overlap, so if every piece is longer than the overlap, you get zero overlap. It should be smaller than chunk_size: the constructor raises a ValueError ("Got a larger chunk overlap ... than chunk size ...") only when chunk_overlap is greater than chunk_size, so equal values are accepted but rarely make sense.

length_function: the default is len, so units are characters. chunk_size and chunk_overlap are in whatever unit this function returns. Pass a token counter and both numbers become tokens, but the splitter still cannot cut inside a piece.

How do I guarantee chunks stay under the limit?

Use RecursiveCharacterTextSplitter. It tries separators in order, by default ["\n\n", "\n", " ", ""]. Any piece still larger than chunk_size is re-split with the next separator, down to single characters, so the size limit holds except in rare edge cases. It also keeps paragraphs and sentences together whenever they fit.

The old import path langchain.text_splitter is deprecated in current releases. Import from the separate langchain_text_splitters package instead.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,        # characters
    chunk_overlap=150,      # about 15%
    separators=["\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_text(text)

# verify
print(len(chunks), max(len(c) for c in chunks))
assert max(len(c) for c in chunks) <= 1000

How do I split by token count in LangChain?

Models limit tokens, not characters. English text averages roughly 4 characters per token, so chunk_size=1000 characters is around 250 tokens, but code, URLs and non-English text can be far denser. If you need a real token budget, count tokens.

from_tiktoken_encoder wires a tiktoken counter in as length_function. Prefer it on the recursive splitter, because the same class on CharacterTextSplitter still passes oversized pieces through. TokenTextSplitter slices strictly by token windows, which gives exact sizes but can cut mid-word or mid-sentence.

# pip install tiktoken langchain-text-splitters
import tiktoken
from langchain_text_splitters import (
    RecursiveCharacterTextSplitter, TokenTextSplitter)

splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name="cl100k_base",
    chunk_size=400,         # tokens
    chunk_overlap=60,
)
chunks = splitter.split_text(text)

enc = tiktoken.get_encoding("cl100k_base")
print(max(len(enc.encode(c)) for c in chunks))

# strict token windows (may cut mid-sentence)
hard = TokenTextSplitter(encoding_name="cl100k_base",
                         chunk_size=400, chunk_overlap=60)

Which splitter should I choose, and what chunk size works for RAG?

Pick by how reliable your delimiters are and whether the limit is hard.