Catching AI hallucinations: 3 guardrails for B2B teams

What hallucinations look like in real B2B workflows

A hallucination is when the model generates a confident-sounding but wrong answer. In B2B this shows up in three classic shapes: (1) inventing an order number that does not exist for a support query, (2) "quoting" a date that is not actually in the contract while summarizing it, (3) inventing a new category when classifying invoices outside the defined set.

All three share the same pattern: when the model cannot find a real anchor, it fills the gap. The problem is the output looks confident — even when wrong. "AI is wrong" is not actionable; you need to catch where and why it is wrong. The three guardrails below do that.

Three guardrails: source, rule, human

Source check: show the document and passage supporting the answer; verify that the passage supports the claim, not merely that a citation exists. Missing or conflicting evidence should route the case for review.

Rule check: verify the order exists, the user is authorised, the amount is valid and the category is allowed in application code. Valid formatting does not establish factual accuracy.

Human approval: before a payment, refund or binding message, an authorised person should see the exact output; changed output requires renewed approval. These layers have no universal success percentage. Test correct, wrong and missing-source examples: invalid output must stop, and the review queue must reach its owner.

Building an eval process before you ship

You cannot catch a hallucination rate by eyeballing outputs; you need a test set. Build a golden set of 50-200 real examples with known correct answers, and make sure it covers edge cases — missing data, ambiguous queries, out-of-scope requests. Run the model against it after every prompt or model change and before deploying that change, not just at launch.

Track two separate numbers: accuracy (did it get the right answer) and abstention rate (how often it declined to answer). A model that never abstains may answer questions it cannot answer reliably; one that abstains too often may be less useful. Safety cannot be assessed from this rate alone. The right threshold depends on how costly a wrong answer is compared to no answer.

Adversarial testing matters as much as normal-case testing — deliberately feed the model queries designed to trigger hallucination: questions about entities that don't exist, requests that mix two records together, prompts that ask it to extrapolate beyond the source document. If it can't fail gracefully in a controlled test, it won't fail gracefully in production either.

Treat this as a living pipeline, not a one-time gate. Every time you change the prompt, swap models, or add a new document source, re-run the eval set and compare the new accuracy and abstention numbers against your baseline before deploying. It's the same discipline as regression testing in traditional software — it just tests judgment instead of logic.

Which tasks carry the most hallucination risk

Do not rank risk only by whether text is open-ended or structured. A wrong accounting entry in a valid category can still cause harm. Assess impact, reversibility, data sensitivity and who relies on the output. A drafting workflow should not have the same permissions as one that contacts customers or moves money.