Lesson 4 of 6

Why AI makes things up, and how to catch it

AI tools sometimes invent facts, sources and numbers. Here's why it happens, where it shows up, and simple habits that catch it.

3 min readBy Michael Alos

Every AI tool you'll use can produce answers that are confident, well written and wrong. People call this hallucination. It's not a bug that will be fixed next month. It comes from how these tools work, so the answer is good habits, not blind trust.

Why it happens

An AI writes by predicting what text fits. When it doesn't have the information it needs, it doesn't stop and say so by default. It produces something that looks like a good answer, because that's what fits.

The newer models make things up less often than earlier ones, and tools that search the web or read your documents do better still. But no tool is immune.

Where it shows up

  • Invented facts. A plausible statistic, a date, a product feature that doesn't exist.
  • Fake sources. Citations to articles that were never written, or links that go nowhere.
  • Wrong numbers. Totals, percentages and conversions that look right at a glance.
  • Made-up policy details. Ask it about your refund policy without giving it the policy, and it may invent a reasonable-sounding one.
  • Confident guesses about people and companies. Especially smaller businesses and people who aren't widely written about.

Five habits that reduce it

  1. Give it the source. Paste the policy, the price list or the report, and say "Answer only from the text above."
  2. Give it permission not to know. Add "If the answer isn't in the information I gave you, say so." It sounds simple, and it helps a lot.
  3. Ask it to quote. "Quote the sentence your answer is based on." Made-up answers have nothing to quote.
  4. Use web search for current facts, then open the links. A source you haven't clicked isn't a source.
  5. Keep a person in the loop for anything a customer will see, and anything involving money or legal commitments.

Check in proportion to the risk

You don't need to verify everything equally:

  • Low risk: brainstorming, internal notes, first drafts you'll rewrite anyway. A quick read is enough.
  • Medium risk: customer emails, social posts, proposals. Check names, numbers, dates and promises.
  • High risk: contracts, financial figures, medical, legal or safety information. Verify every fact against a real source, or don't use AI for it.

Try this (5 minutes)

  1. Paste a short document you know well, such as your returns policy or service terms.
  2. Tell the AI: "Answer only from the text above. If the answer isn't there, say you don't know."
  3. Ask three questions: two the document answers, and one it doesn't.

Did it answer the first two correctly? Did it admit the third wasn't covered, or did it make something up?

Common mistakes

  • Asking for sources and never opening them.
  • Letting AI answer customer questions about policies without giving it the actual policy.
  • Assuming a longer, more detailed answer is a more accurate one.

Check yourself

Why do AI tools make things up?

They write by predicting text that fits. When information is missing, they produce something plausible instead of stopping, unless told otherwise.

What's the simplest way to reduce made-up answers about your business?

Give the AI your actual information and tell it to answer only from that, and to say so when the answer isn't there.

Which kinds of work need the most careful checking?

Anything high risk: contracts, financial figures, and medical, legal or safety information.

Want your whole team to get this?

I run hands-on AI training for small business teams, built around the work your people actually do.

See team training →