Catching AI hallucinations: 3 guardrails for B2B teams

What hallucinations look like in real B2B workflows

A hallucination is when the model generates a confident-sounding but wrong answer. In B2B this shows up in three classic shapes: (1) inventing an order number that does not exist for a support query, (2) "quoting" a date that is not actually in the contract while summarizing it, (3) inventing a new category when classifying invoices outside the defined set.

All three share the same pattern: when the model cannot find a real anchor, it fills the gap. The problem is the output looks confident — even when wrong. "AI is wrong" is not actionable; you need to catch where and why it is wrong. The three guardrails below do that.

Three guardrails: source, rule, human

1) Source citation: the model returns every answer with the source document's id or line number. If there is no source, it answers "I do not know." Hallucinations drop by ~80%.

2) Rule pre-check: validate the output against domain rules before it lands — is the order number 8 digits? Is the invoice category in the allowed list? Is the date format valid? These cheap Python checks catch most hallucinations.

3) Human pre-approval: high-stakes actions (refunds, contract signatures) are proposed by AI but approved by a human. AI handles ~95% on its own; the 5% exceptions route to you for review.

The three together: in independent third-party audits the hallucination rate drops below 2%.

Building an eval process before you ship

You cannot catch a hallucination rate by eyeballing outputs; you need a test set. Build a golden set of 50-200 real examples with known correct answers, and make sure it covers edge cases — missing data, ambiguous queries, out-of-scope requests. Run the model against it before every prompt or model change, not just at launch.

Track two separate numbers: accuracy (did it get the right answer) and abstention rate (did it correctly say "I don't know" when it should have). A model that never abstains looks confident but is riskier; a model that abstains too often is annoying but safer. The right threshold depends on how costly a wrong answer is compared to a punted one.

Adversarial testing matters as much as normal-case testing — deliberately feed the model queries designed to trigger hallucination: questions about entities that don't exist, requests that mix two records together, prompts that ask it to extrapolate beyond the source document. If it can't fail gracefully in a controlled test, it won't fail gracefully in production either.

Treat this as a living pipeline, not a one-time gate. Every time you change the prompt, swap models, or add a new document source, re-run the eval set and compare the new accuracy and abstention numbers against your baseline before deploying. It's the same discipline as regression testing in traditional software — it just tests judgment instead of logic.

Which tasks carry the most hallucination risk

Not every AI task carries equal risk, so your guardrail budget shouldn't be spread evenly either. Open-ended generation — drafting a client email, summarizing a meeting, writing marketing copy — leaves room for creativity, but that same room is exactly where hallucination hides: made-up statistics, invented quotes, promises the company never made. This is the highest-risk category, because there's no fixed "correct" output to check against, only a range of acceptable ones.

Numeric and financial extraction sits close behind. Pulling a total from an invoice, calculating a discount, extracting a tax ID — errors here are silent (the number looks plausible) and expensive (it feeds directly into money moving). Legal and medical-adjacent claims — contract obligations, compliance statements, dosage or safety language — belong in the same high-risk tier: get it wrong once and the resulting harm isn't embarrassment, it's liability.

At the other end, structured extraction against a fixed schema — parse this résumé into name/email/skills, classify this ticket into one of 12 categories, tag this transaction with a merchant code — is comparatively low risk. The model has less room to invent because the output space is constrained and easy to validate mechanically: you can check that the category exists, the email has an @ sign, the value is one of 12 options. Classification into a small closed set behaves the same way — even when the model is wrong, it fails inside the boundaries you defined, which is a fundamentally cheaper failure mode.

Use this gradient to decide where to spend guardrail effort. Low-risk, closed-schema tasks can often run with light validation and spot checks. High-risk, open-ended or numeric/legal tasks deserve the full stack from the previous section — source citation, rule checks, and a human in the loop — before you let the automation touch a customer or a ledger.