Hallucination (AI)

Why a wrong answer sounds exactly like a right one

A language model does not look a fact up and then write it down. It writes, choosing what most plausibly comes next given everything in its context. When the context contains the answer, that process produces a correct sentence. When it does not, the process still produces a sentence, because producing one is the only thing the model does. Nothing in the output marks which of the two just happened.

So "the model sounded certain" carries no information at all, and hallucinations cluster in predictable places: wherever a specific detail was asked for that the model was never given. Ask it to summarise a document sitting in front of it and you will rarely see one. Ask for a statistic, a source URL, a competitor's pricing, or the result of an action it cannot observe, and you have built the exact conditions. The reliable tell is not tone, it is the absence of a source you can open — which is why an evidence chain is worth more than any confidence score.

What AI marketing tools actually make up

In marketing work the fabrications are not exotic. They come in a handful of shapes, and knowing the shapes is most of the defence:

• Numbers and studies — a percentage attached to a research firm that never published it. These read as authority, which is exactly why they get pasted into a landing page and then quoted back at you by a customer.
• Links and citations — addresses that 404, or that resolve but say something else entirely. A model can construct a plausible-looking URL for a page it has never seen.
• Quotes and social proof — testimonials, review scores and case-study outcomes with nobody behind them. Fabricated social proof is among the easiest things to disprove and the most expensive when someone does.
• Its own actions — the one most people miss. An agent that drafted a post and an agent that published one describe their work in identical words, and only one of them is telling the truth.

Two hallucinations from our own product

We operate a managed AI marketing team, so the honest version of this section is our own incident log rather than a survey of other people's. The first case was image links. Our Growth Officer delegated an image task to a specialist and wrote the customer-facing delivery message in the same turn — before the specialist had returned anything — assembling image addresses out of identifiers that happened to be sitting in its context. Of the 151 links it produced, 57 pointed at files that did not exist. Every one of them was correctly formed.

The cause was not carelessness on the model's part, it was our own instruction. The prompt taught the markdown syntax for an image without ever saying where the address comes from, which reads as a template to fill in. Given a blank to complete and no rule against completing it, the model completed it. We rewrote the instruction, but the instruction was never the real fix.

The second case was a research report. A new signup typed a company name, a one-word description, and a web address that did not resolve. The first pass said so honestly: the site could not be reached. A second pass ran a minute later against the same profile and produced a competitor teardown — named rivals, positioning, the lot — for a business that does not exist. Nothing had crashed and no rule had been broken. The second pass simply had less to work with, and a plausible-sounding name was enough to fill the gap.

Checks that hold up, and the one that does not

The instinctive fix is to write a firmer instruction. It does not hold, because the thing you are relying on to follow the rule is the same thing that is unreliable. The checks that survive contact with a model are the ones the model gets no vote in:

• Verify existence, not format. The sharpest thing we learned: what an agent invents is the identifier, not the shape around it. Any check the model could derive, it can also satisfy while making the answer up — so our media links are now checked against the actual file, never against a pattern.
• Gate whatever cannot be undone. A wrong draft costs nothing. A sent email cannot be recalled. Route irreversible work through an approval gate and keep a human in the loop on anything outbound.
• Let the agent read its own outcome. An agent that never learns whether an action succeeded will describe the version it expected. Feeding the real result back is what stops a failed post being reported to a customer as published.
• Require a source per claim. Every number, quote and link should trace to something you can open. An unsourced claim is not automatically false, but it is the one you have no way to check.