What it measures
Half the set is supported by ordinary facts. Half is intentionally missing evidence — fictional cities, lost books, imaginary treaties. A useful system answers the first and abstains on the second. A fluent system that invents an author for a book that does not exist is scored as a hallucination, not as creativity.
Metrics: answer accuracy, abstention accuracy, hallucination count, false abstention count, results by category. Export JSON or a Markdown report from the demo.
How to run it
The offline baseline is a transparent rule: answer when the item is labeled supported, abstain otherwise. It is not an LLM. It exists so the pipeline, labels, and exports can be checked without a model.
The optional local browser LLM uses WebLLM to run quantized Llama-3.2-1B-Instruct through WebGPU. First load downloads weights into the browser cache. No server upload, no API key. Chrome or Edge with hardware acceleration.
Limits — say them with the result
- Thirty items. Not a leaderboard, not a paper-scale eval.
- The offline baseline must not be quoted as model performance.
- The 1B local model is small; quality depends on hardware, version, and prompt.
- Future work: more models, human labels, confidence intervals across repeats.
I built this as an independent exploration of LLM reliability, in parallel with AegisLand’s abstention work on cameras. The shared claim is narrow: refusal is a first-class output, and it should be measured.