Contradicted evidence94% risk
A: Australia
top signal: overlap_answer_context, n_words, nli_ctx_contradicts_ans, n_chars, jaccard_ans_ctx
Calibrated · Explainable · Lightweight
Know when an answer stops being grounded.
HaluRISC scores candidate answers against the evidence you provide. It runs on CPU in milliseconds, needs no model internals, and shows where its own limits are.
- CPU only, no GPU required
- Artifact-backed numbers, never staged
- Open failure cases
Grounded in evidence7% risk
A: Jason Campbell holds the record.
top signal: overlap_answer_context, n_words, n_chars, nli_ctx_neutral_ans, nli_ctx_contradicts_ans
Real cases from the B5 reviewer export · scores are the evidence-calibrated risk
0.9846
In-domain F1
B2 · leakage-free grouped splits, 3 seeds
0.3022
Zero-shot F1 on RAGTruth QA
B3 · external data, no retraining
0.82 → 0.13
ECE: source → target calibrated
B4 · calibration does not transfer for free
105×
Cheaper than an LLM judge
measured cost per 1,000 predictions
Live demo readiness
Preflight for the booth. Offline pages (/demo, /dashboard) never depend on these checks.
- Checking dependencies…
How it works
Read the method01
Give it three things
A question, the evidence you trust (pasted, uploaded, or searched), and the candidate answer. No model weights or activations needed.
02
It measures grounding
26 features across seven groups: lexical overlap, entity coverage, numeric consistency, NLI contradiction, hedging, semantic drift, and length. A calibrated XGBoost turns them into a risk score.
03
You get a verdict with citations
Claim-level supported / contradicted / unsupported verdicts, SHAP attribution for every score, and the failure cases kept in the open.
Where to go next
Analyze mode
Type or paste a case and inspect the score, thresholds, and SHAP values side by side.
Chat mode
Converse with a grounded assistant; every answer is checked after it streams.
Experiment dashboard
All B2 to B5 tables, charts, and statistics, rendered straight from the frozen artifacts.
Presenter demo
A fully offline walkthrough: no API key, no network, works on a clean clone.
It does not pretend to be perfect
Zero-shot transfer to RAGTruth QA drops well below in-domain performance, and source calibration does not transfer to new data without refitting. Both effects ship in the dashboard with the exact numbers.