How Chatbots Answer Using Your Documents

Many people picture a chatbot as a machine that somehow knows everything about a business from day one. The reality is both simpler and more interesting: a good bot memorises nothing. Every time a question arrives, it first searches for the most relevant part of your documents, then composes an answer from that part alone.

The technical name for this is retrieval. Understanding it makes a lot of otherwise puzzling chatbot behaviour suddenly make sense.

Documents are split into chunks

A single product guide can run to dozens of pages. Feeding all of it to a language model on every question is not only expensive, it actively hurts accuracy: the more irrelevant material you include, the more room there is for the answer to drift.

So documents are split into small chunks first. Good chunking tries to keep one idea whole inside one chunk. If the widget installation procedure is cut in half, the first chunk holds steps without a conclusion and the second holds steps without context. Both end up half useful.

Each chunk gets coordinates for meaning

Every chunk is then converted into a list of numbers representing its meaning, commonly called an embedding. Think of it as giving each chunk a position on a map. What makes this map useful is that two passages with similar meaning land close together, even when they share no words at all.

Search runs down two paths at once

Meaning-based search alone turns out to be insufficient. A whole class of questions depends on literal matching instead: product names, order codes, technical terms, the exact label of a menu in your app. For those, direct word matching through a full-text index is far sharper than semantic proximity.

Mature systems therefore use both. The semantic path catches questions worded differently from the source. The lexical path catches questions that hinge on an exact term. They answer different needs, and neither replaces the other.

The model composes the answer, it does not invent it

The selected chunks are handed to the language model together with the original question, under a firm instruction: answer only from this material. The model is not asked to remember anything. It is asked to summarise and to phrase clearly.

This is also where citations come from. The bot can point at a source page because it genuinely read that chunk in order to answer, not because it claims to have read it once.

What this means for you

The practical conclusion often surprises business owners: answer quality depends more on the quality of your source documents than on the sophistication of the model. When a bot keeps missing, the first thing to inspect is not the model but whether the answer actually exists in your sources, stated plainly and not scattered across several places.

Tidy, specific documentation almost always produces a bigger jump in quality than swapping the model.