Stop Your AI Inventing Facts: Give It Sources It Must Use

You asked it a question about your own company and it answered instantly, confidently, and wrong.

Not vaguely wrong. Specifically wrong. It gave you a figure, in a whole sentence, with the tone of somebody reading off a document. The real figure had changed a month earlier and applied to one team only.

If that has happened to you, you are not the only one, and the reason it unsettles people is not the error. It is the confidence. A hesitant wrong answer you would have checked.

It is not lying and it is not guessing badly.

Two plain limits explain almost all of it.

The first is that it learned from an enormous amount of text that stopped at a date. Anything that happened after that date is not in there.

The second matters more: it has never seen anything of yours. Not your policy document, not your price list, not last quarter's decisions, not the email thread where you changed your mind.

So when you ask about your own business, there is nothing in front of it. And here is the part that surprises people: it does not stop. It has no feeling of not knowing. Producing the most plausible-sounding continuation is the whole of what it does, and a plausible-sounding figure is easy to produce.

I want to be blunt about one thing, because it saves months. You do not fix this by wording the question better. You can write an instruction of real quality, with the role, the format and the conditions all spelled out properly, and still get an invented number, because the number was never in the room. That is a different problem from a vague answer, and it needs a different fix.

The fix is three steps and it is not clever.

Give it the document. That is genuinely the whole idea. Everything else is plumbing.

01 · Find. Before the model answers anything, the system searches your documents and pulls out the passages that relate to the question.

02 · Give. Those passages get put into the prompt, underneath the question, as material it must work from.

03 · Answer. The model answers from what is now in front of it, rather than from what it half-remembers about companies in general.

Closed book becomes open book. That is the entire mechanism, and once you have seen it you cannot unsee it in any tool that claims to know your data.

The industry calls this retrieval-augmented generation, usually shortened to RAG. The name is worse than the idea.

One detail is worth knowing because it explains why it works better than you expect. The search is not looking for matching words. It compares meaning. Somebody types how do I send an item back and the system finds your Returns Policy section even though not one of those words appears in it. That is why this beats the search box on your intranet, which has been failing at exactly this for fifteen years.

The free AI Agents pack. The reliability checklist in there covers this end to end, including how to prepare documents before you load them. Plus over two thousand ready-made n8n workflows, a workbook on the twenty building blocks that cover most of them, and the Agent-Plus Prompt Builder for writing agent instructions. Download the AI Agents pack →

Where it still goes wrong, in three places.

This is the part almost nobody tells you, and it is the difference between a demonstration and something you can rely on. When a system with sources still gives bad answers, it is nearly always the finding step, not the answering step. Three faults cover most of it.

The pieces are too big. You loaded a hundred-page handbook as one lump, or in enormous sections. What comes back is technically relevant and carries a great deal of unrelated text with it, and the model loses the thread inside the noise. Symptom: answers that wander off topic while sounding authoritative.

The pieces are too small. You cut it so finely that a passage came back with the sentence you needed and none of the context that gave it meaning. Twenty days per year is useless without employees with three to five years of service. Symptom: answers that are precisely correct and practically wrong.

Too few pieces come back. The system is only fetching the top result or two, and your question needed three sections to answer properly. Symptom, and this is the diagnostic one: everything it says is accurate, and it is somehow thin. Raise the number of passages it retrieves before you touch anything else.

The same question, two ways.

Without sources. You ask what the remote work allowance is after the update. Instant answer, complete sentence, a figure that is plausible for a company of your size and does not match your policy. Nothing in the reply signals uncertainty, because there was none.

With sources. The system searches the policy folder, finds the section headed Remote work, revised, puts it into the prompt, and the answer comes back with the current figure, the group it applies to, and the document it came from. You can check it in nine seconds.

Notice what actually improved. Not the model's intelligence. The material on the desk in front of it.

Three questions when the answers are still bad.

Copy this. It is the diagnosis I run before changing anything else.

Are the answers off topic? That is the finding step. Look at your chunk sizes first, and split documents by section rather than by length wherever you can. Are they accurate but thin? Too few passages are coming back. Raise the number retrieved. Are they right but slow? That is indexing and the size of the search model, not the writing model. Fix it there rather than downgrading the thing producing the answer.

And one preparation rule that sits above all three: strip anything sensitive before you upload it. Content from files you load can end up in an answer, so the file you hand over should contain what the system needs to do its job and not one thing more. Personal data, salaries, anything under a contract. Take it out before it goes in, not after.

What this will not do.

It will not make the model honest. It makes invention much rarer. It does not make it impossible, and for anything customer-facing you still want a human saying yes before it goes out.

It will not answer from a document you did not give it. Obvious written down, easy to forget when the answers start being good.

And it will happily quote a document that is out of date, with total confidence, exactly as it quoted the imaginary one. Sources move the problem from the model to your folder, which is a much better place for it, but only if somebody keeps that folder true.

One small thing to do before the next post.

Name the three documents that would answer most of the repeated questions you get asked.

Not build anything. Not upload anything. Just write down three file names.

For most people it is a policy or handbook, a price or product list, and one folder of past answers. When you can name them, you have the hard half of this done, because the shape of what you are building is now decided and the rest is assembly.

If you want the whole thing in one place.

If you would rather have the complete system than one piece at a time, that is what I wrote AI Agents Bible for. Five books in one, with Custom GPTs, Make.com and n8n each built step by step, thirty worked examples you can copy, and the parts most guides leave out, about cost, reliability and knowing what not to let run on its own.

Both this and the prompt engineering book also come as one 12-in-1 volume, if you would rather have the whole system in a single place.