How to chunk a large text file for LLM processing

The short answer. Even with 1M+ token context windows now common (as of 2026-05: Claude Sonnet 4.6 hit 1M tokens in February 2026, GPT-5.5 reached 1M via API in April 2026, Gemini 3.1 Pro sits at 1M tokens — verify these against each provider's current docs, the model lineup turns over every quarter), chunking still matters for cost, quality, and retrieval. For a quick browser split — paste, pick CHARS mode, set a codepoint count, download the zip — use the text file splitter. For token-accurate chunking that matches OpenAI's billing, use Python with tiktoken. For RAG pipelines, LangChain's RecursiveCharacterTextSplitter.from_tiktoken_encoder handles the recursion + overlap correctly. The catch: characters ≠ tokens, and a "10K character" chunk isn't a "10K token" chunk. The table at the bottom maps your situation to the right method.

Why chunking still matters in 2026

Long context windows have changed what's possible — you can now paste an entire book into Claude or Gemini and ask questions about it. But four reasons keep chunking necessary:

• Cost. Google applies a 2x input / 1.5x output surcharge above 200K tokens on Gemini 3.x Pro / 2.5 Pro as of mid-2026. Anthropic charged a similar surcharge on Claude Sonnet 4.5's 1M-beta but dropped it for Sonnet 4.6+ — the full 1M now runs at standard pricing. The pricing model varies fast; verify against the provider's pricing page before relying on long-context economics in production. Chunking still helps when you target models that have the surcharge.
• Quality degradation. Stated context limit ≠ usable context limit. Research consistently shows performance degrades well before the advertised number — the "lost in the middle" effect, where content in the middle of a long prompt gets ignored. A 100K-token chunk often gets better attention than a 1M-token blast.
• Retrieval-augmented generation (RAG). RAG works by embedding small chunks, retrieving the relevant few, then sending only those to the LLM. The chunk size affects retrieval quality — too small loses context, too big dilutes the embedding.
• API streaming and latency. Smaller prompts come back faster. For interactive use, three 30K chunks often beats one 90K chunk.

The right chunk size depends on what comes next. For RAG retrieval, 200–1000 tokens per chunk is the current sweet spot — most modern embedding models (OpenAI text-embedding-3-large, Cohere, Voyage) reach their best signal-to-noise around 512 tokens. For direct prompting with no retrieval, you can go much bigger — up to whatever fits under the cost-quality sweet spot, often 50–200K tokens.

How do you chunk in the browser, instantly?

Open the text file splitter. Paste the content or drop the file. Pick the right mode:

• CHARS — splits by codepoint count. 1 codepoint = 1 character you see on screen for most text (Latin, common CJK), but emoji and Hindi/Thai cluster differently. Closest browser-tool approximation to a token, but not identical — see the caveat below.
• LINES — if your text is naturally line-oriented (logs, lists), split by N lines per chunk for clean boundaries.
• BYTES — for size-driven limits (API payload caps), use byte mode. UTF-8 safe — the tool doesn't split inside a multi-byte character.
• PARTS — "give me N equal chunks" without thinking about size. Useful for parallel processing.

Click Split. Each chunk appears as its own panel; the Download ZIP button packages them. Nothing uploads — the split happens in the tab.

Honest limitation: CHARS ≠ tokens. The tool counts codepoints, not LLM tokens. As a rough rule, 1 English token ≈ 4 characters in GPT-style tokenizers — so 10,000 characters ≈ 2,500 tokens. For programming-heavy or non-English text, that ratio can be 2–6. If your downstream consumer charges by token (every commercial API), use tiktoken to get exact counts; the browser tool is the right pick for "rough chunking, no install".

How do you chunk with tiktoken (token-accurate)?

The OpenAI-released tokenizer that matches their billing. pip install tiktoken.

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")
tokens = enc.encode(open("long_doc.txt").read())

chunk_size = 8000  # tokens
chunks = [
    enc.decode(tokens[i:i + chunk_size])
    for i in range(0, len(tokens), chunk_size)
]

for i, chunk in enumerate(chunks):
    with open(f"chunk_{i:04d}.txt", "w") as f:
        f.write(chunk)

Five lines, exact token boundaries. The encoder is model-specific. encoding_for_model("gpt-4o") returns the o200k_base encoding; encoding_for_model("gpt-4") returns cl100k_base; you can also load encodings directly with tiktoken.get_encoding("o200k_base"). Use whichever matches the model you'll send the tokens to — counts diverge meaningfully across encoders.

For Claude, Anthropic doesn't publish an open tokenizer, but the anthropic Python package (SDK 0.40+) exposes client.messages.count_tokens(model="claude-sonnet-4-6", messages=[{"role": "user", "content": text}]) for exact counts (server-side). For Gemini, the SDK exposes a count_tokens() method too. If you don't want a server round-trip just to count, tiktoken's o200k_base is the closest open approximation for most current commercial tokenizers — usually within 10–20% of the actual count.

How do you chunk with LangChain (RAG-ready)?

The default for production RAG pipelines — chunk size, overlap, recursive splitting, and tokenizer integration all in one class.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    model_name="gpt-4o",
    chunk_size=500,        # tokens
    chunk_overlap=50,      # tokens
)

chunks = splitter.split_text(open("long_doc.txt").read())

What the recursive splitter does that simple slicing doesn't:

• Respects natural boundaries. Tries to split on paragraph breaks first, then on sentence breaks, then on word breaks, then on character breaks — in that order. A chunk almost never ends mid-word.
• Overlap. chunk_overlap=50 means each chunk repeats the last 50 tokens of the previous chunk. Helps when a single semantic unit (a definition, a quote, a procedure) straddles the boundary — it appears in full in at least one chunk.
• Token-accurate counting. Uses tiktoken under the hood; chunk size is in actual tokens, not characters.

The from_tiktoken_encoder variant is the right pick for any pipeline that goes through OpenAI APIs. For Anthropic or Gemini, use the plain character-based splitter and approximate token count with a 4:1 ratio — or use the provider's SDK for exact counts.

When semantic chunking beats fixed-size

Fixed-size chunking (every chunk is exactly N tokens / chars) is fine when the text is uniform — logs, news articles, plain prose. Semantic chunking (each chunk is one semantic unit) is better when the document has natural structure:

• Markdown / HTML. Split at headers; each section becomes a chunk.
• Source code. Split at function or class boundaries; never split inside a function.
• FAQ / Q&A. Split at each Q&A pair.
• Legal / structured contracts. Split at numbered clauses.

LangChain has dedicated splitters for these: MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter(separators=["\n\nclass ", "\n\ndef "]) for Python code, and so on. For one-off work, you can do semantic chunking with the joinlines splitter in DELIM or REGEX mode — pick the delimiter that separates your semantic units.

Which method should you use?

Frequently asked questions

What's the right chunk size for RAG?

Most guides land in 200–1500 tokens, with 10–20% overlap. Smaller chunks (200–500) work better for fact-retrieval queries; larger chunks (1000–1500) work better when the answer needs surrounding context. We see teams default to 500-token chunks with 50-token overlap and tune from there. The right answer depends on the embedding model and the query types — no universal default works for every dataset.

Why does my "10K character" chunk show up as different token counts in different models?

Because each model has its own tokenizer. GPT-4o uses o200k_base; GPT-4 uses cl100k_base; Claude uses an unpublished tokenizer; Llama uses its own SentencePiece encoding. For English text, all are roughly 4 chars per token; for code, closer to 3; for CJK text, sometimes 1-2 chars per token. tiktoken covers OpenAI's family; for others, use the provider's SDK to count.

Should I use overlap between chunks?

For RAG: yes, almost always. 10–20% overlap reduces the "answer split across chunks" failure mode. For direct prompting, depends on which sub-case: if you concatenate all chunks into one continuous conversation, overlap duplicates context the model has already seen and is wasted tokens. If each chunk is sent as its own independent prompt (separate sessions, parallel processing), overlap still helps — it prevents semantic units from being split across the boundary.

How do I chunk a PDF / Word document for an LLM?

Extract to plain text first, then chunk the text. PDF extraction is its own problem — pdftotext on Linux, pypdf in Python, pdf-parse in Node. For PDFs with tables and figures, semantic-aware extractors like unstructured or commercial tools like LlamaParse give better results. Once you have clean text, chunk with any of the recipes above.

Will a 1M-token context replace chunking?

Not for cost-aware production work. Google still applies a 2x input surcharge above 200K on Gemini 3.x Pro, and even on providers that have dropped the surcharge (Anthropic Sonnet 4.6+), the quality-degradation effect makes giant prompts unreliable. For interactive one-off use — "summarise this book" in Claude.ai or Gemini — the long context does replace chunking for that single query. For repeated queries against the same long document, RAG with chunked embeddings beats re-sending the full context every time. Model specs change fast — check the provider's current pricing page before relying on any specific number in this article.

Is it safe to paste a sensitive document into an online chunker?

Most "free online text chunker" tools upload the document to their server. For sensitive content (legal docs, internal data, drafts), use the browser-based joinlines splitter (verifiable in DevTools → Network, zero outbound requests) or a local Python script. Never paste confidential content into a SaaS chunker.

— the joinlines team, 2026-05-19