AI hallucination risks for business: the report that almost started a war

An analyst asked a chatbot about a ship's cargo. The chatbot got it wrong. The analyst then used AI again to turn that answer into a standard intelligence report, and the US military began preparing to board the vessel before anyone checked. That story, reported by CNN this week, is the clearest illustration yet of the real near-term AI hallucination risks for business: not rogue AI, but people acting on confident, wrong output that has been packaged to look authoritative.

What happened

According to CNN's exclusive report, citing four sources, a special operations command analyst queried a chatbot about intelligence reporting on a ship's manifest. The bot fused open-source information with classified signals intelligence and inaccurately identified what the ship was carrying. The analyst used AI again to package the findings into a standard intelligence report, which was disseminated.

Military planes were in the air and armed personnel were preparing to board before officials dug deeper and found the report had been generated with AI help. One source called it "entirely false" and said it "almost started a war". CNN also reports there is no single standard across the military and intelligence community for verifying what these AI tools produce. The story drew hundreds of comments on Hacker News within hours.

AI hallucination risks for business: the same chain

Replace "ship's manifest" with "supplier contract", "customer churn analysis" or "board pack", and the chain is familiar:

1. Someone asks an AI tool a question it cannot answer reliably from the data it has.
2. It answers fluently and confidently anyway.
3. The answer is reformatted — often by AI again — into a document that looks like finished work.
4. The document's format signals a level of verification that never happened.
5. A decision is made on it.

The most dangerous step is the fourth. A chatbot answer in a chat window invites scepticism; the same content in a company template, with headings and a confident summary, does not.

Why do AI chatbots hallucinate?

Language models generate the most plausible continuation of text. When the facts are missing, ambiguous or contradictory, plausible and true diverge — and the model has no built-in way to say "I do not know" unless it is designed and prompted to. Mixing sources of different reliability, as in the CNN case, makes it worse: the model blends them into one confident narrative and the provenance disappears.

How can businesses prevent AI hallucinations?

You cannot eliminate hallucination from language models, but you can stop it reaching decisions. The controls that matter:

1. Provenance on every AI-assisted document

Label documents and sections that AI helped produce, and record which tool, which sources and who reviewed it. A reader should never have to guess.

2. Answers with citations, or no answer

For factual questions over company data, use retrieval-based systems that cite the exact source passage, and configure them to say when the sources do not support an answer. A claim without a source should be treated as a guess.

3. Human review scaled to the stakes

Low-stakes drafts can go straight out. Anything that triggers spending, legal commitments, customer impact or safety decisions needs a named reviewer who checks claims against sources, not just reads for tone.

4. Confidence where the model can give it

For decisions with fixed options, models that return probabilities — such as the new Jev decision model from TypeSafe AI — let software act only above a threshold and route uncertain cases to a person.

5. Evaluation before rollout

Test AI features against real examples with known answers before deploying them, and keep testing after every model or prompt change. Consistency matters as much as accuracy — see our guide to AI consistency testing.

6. A policy people actually follow

Say which tools are approved for which data and decisions, and make the review step part of the workflow rather than a line in a PDF. Our note on AI agent governance covers the wider framework.

Should AI-generated reports be labelled?

Yes. The CNN case turned on a report that looked like normal analyst work. A simple rule — every AI-assisted document says so, with its sources and reviewer — would have prompted the question that was asked too late. Labelling is not a statement that AI is bad; it tells the reader how much checking has been done.

Where businesses are most exposed

• Financial analysis and board packs, where AI summarises figures from several spreadsheets and a wrong number looks exactly like a right one.
• Contract and policy review, where a missed clause or invented obligation carries legal cost.
• Customer-facing answers, where an assistant states a refund rule or price that does not exist.
• Code and configuration, where plausible but wrong changes pass a quick glance — the reason we argue for reviewing acceptance criteria, not just diffs.

The takeaway

The risk from AI in most organisations is not science fiction. It is a fluent wrong answer that nobody checked because it looked official. Design your processes so AI output carries its provenance, cites its sources and passes a human check proportional to its consequences. Our AI engineering services team builds AI features with citations, confidence thresholds and review built in, so that speed never comes at the cost of being wrong at scale.