Large language models fabricate facts fluently. We describe a verification 'fence' that sits between a model and the user: every factual claim must cite a specific memory entry, and the fence checks that the citation exists, that any numbers in the claim match the cited source, that the claim's concepts are covered by the source, and that no overclaiming language is used. On a labeled set of 24 claims spanning five hallucination types, the fence achieved precision 1.00 and recall 0.929 (F1 0.963), with a single miss whose cause we identify and report.
The fence receives claims tagged as MEMORY (asserted facts) or INFERENCE (derived conclusions), each with a list of cited memory-entry IDs. For MEMORY claims it verifies: (a) every cited ID exists in the knowledge base; (b) every numeric token in the claim appears in at least one cited entry; (c) the claim's content words overlap the cited entry above a coverage threshold. For INFERENCE claims it additionally rejects overclaiming language (definitely, guaranteed, 100%, never, always) and requires at least one cited premise.
We built a knowledge base of eight true facts and a labeled test set of 24 claims: 10 true claims (exact, paraphrased, and inferential) and 14 hallucinations across four categories — wrong numbers, fabricated citations, overclaims, and unsupported concepts. A claim is 'positive' if it is a hallucination the fence should reject.
| Metric | Value | Meaning |
|---|---|---|
| Precision | 1.000 | of flagged claims, all were genuine hallucinations |
| Recall | 0.929 | of hallucinations, fraction caught |
| F1 | 0.963 | harmonic mean |
| Accuracy | 0.958 | overall correct decisions |
| Hallucination type | Caught |
|---|---|
| wrong number | 5 / 6 |
| fabricated citation | 3 / 3 |
| overclaim | 3 / 3 |
| unsupported concept | 2 / 2 |
| (true claims correctly passed) | 10 / 10 |
The fence's single failure: it passed 'BFT tolerates one half faulty validators' when memory says 'one third.' The numeric check inspects digit tokens; here the mismatch is between the words 'half' and 'third,' which the digit check does not see. The fix is a word-to-number normalization pass before the numeric check. We report this because it precisely bounds the method: the fence catches digit-level numeric fabrication and citation fabrication reliably, but word-level numeric claims can slip through. Precision remained perfect — the fence never blocked a true claim — which matters for usability, since a fence that cries wolf gets turned off.
Zero false positives across 24 cases is the practically important result: the fence adds safety without degrading correct answers. Recall of 0.93 with a single, well-understood failure mode makes this a deployable verification layer for any model whose outputs cite a knowledge base, not only the system it was built for. The method is model-agnostic: it verifies outputs, not the model that produced them.
The fence adds safety without degrading correct answers. Zero false positives across 24 cases is the practically important result. Recall of 0.93 with a single, well-understood failure mode makes this a deployable verification layer. Preliminary results, not peer reviewed.