That is AI reliability in miniature: reliable for work you check, unreliable for work you send out blind. A language model predicts the next word based on patterns. It does not know what is true, but it can write the answer down very nicely. That is why mistakes sound exactly as confident as the rest of the answer. That is the real risk: an error that looks like a fact.
Air Canada showed how expensive that can get. In 2024 their chatbot told a passenger he could apply for a bereavement fare retroactively. He could not, and the tribunal ruled that the airline simply had to pay: a company is liable for what its chatbot says. The defence that the chatbot was a separate entity did not hold. For you that means: whatever your AI sends to a customer is your promise.
How do you prevent hallucinations?
The intervention that works is organisational, not technical: put a human at the point where the work leaves the building. Alongside that, three things you can arrange today:
- give the model your own sources: an AI drawing on your rate card and your client file invents less than an AI talking from memory
- ask for uncertainty: have the model mark what it is unsure about with a tag such as [VERIFY]; that is one line in your instruction and it puts the check in the right place
- check facts, not language: names, amounts, quantities and dates are where it goes wrong, the phrasing is usually fine. Especially in combination with a clear tone of voice document
For tasks where a mistake costs money or trust, build in a hard stop: no quote leaves the building without a human eye, no amounts from a model that does not know your price list. That is the same arrangement you make with a new colleague, and nobody calls that distrust.
And that test at the top? Everything the model invented about your company is a gap you have not filled yet. Internally it works exactly the same way: the better your own information is organized, the less there is to invent. That is why we build the way we do: one process, four weeks, and control stays with the people who know the work.