Version B evidence
Experiment dashboard
Every value is read from artifacts/results at request time, never hardcoded.
How to read these metrics
- F1
- Balance of precision and recall. The headline detection metric on the test split.
- AUROC
- Probability that a random positive case scores above a random negative one.
- PR-AUC
- Precision-recall area. More informative than AUROC when classes are imbalanced.
- MCC
- Matthews correlation coefficient. A balanced single-number quality score.
- ECE
- Expected calibration error. Gap between predicted probabilities and observed frequencies.
- Brier
- Mean squared error of the predicted probabilities. Lower is better.
- Platt / isotonic
- Post-hoc calibration methods. Platt is the deployable default here.
- SHAP
- Per-feature attribution explaining how the model reached a score.
- McNemar
- Paired significance test between two classifiers on the same test cases.
- Bootstrap CI
- Confidence interval from resampling with replacement.
Perturbation-induced failures (B5.6)
Score moved by more than 0.3 or SHAP top-1 flipped under a controlled text edit.
- entity29
- support_removal19
- numeric8
- date4
- irrelevant_insert1
61 flagged evaluations from the B5 perturbation protocol — candidates for the manual review sheet, not automatic evidence of model failure.
Manual review sheet (B5.5)
Sampled cases for two reviewers — fill reviewer_1 / reviewer_2 / agreement in b5_review_cases.csv.
| ID | Label | Raw | Calibrated | Top-5 SHAP | Question (truncated) |
|---|---|---|---|---|---|
| q_8518_correct | 0 | 0.8016 | 0.9354 | overlap_answer_context, n_words, nli_ctx_contradicts_ans, n_chars, jaccard_ans_ctx | "Touch Too Much" is the fourth track on an album by a band that hails from which country? |
| q_3061_correct | 0 | 0.9135 | 0.9751 | overlap_answer_context, n_words, n_chars, nli_ctx_contradicts_ans, nli_ctx_entails_ans | What sort of art did Herbert Abrams complete using the Chief of Staff of the United States… |
| q_7742_correct | 0 | 0.8612 | 0.9609 | overlap_answer_context, n_words, n_chars, nli_ctx_neutral_ans, nli_ans_neutral_ctx | What is is a United States federal statute enacted by the 111th United States Congress and… |
| q_1402_correct | 0 | 0.7594 | 0.9087 | overlap_answer_context, n_words, nli_ctx_contradicts_ans, n_chars, nli_ans_neutral_ctx | Which chalet-style building was demolished when construction began on the seventh Disney V… |
| q_8650_correct | 0 | 0.9790 | 0.9860 | overlap_answer_context, n_words, n_chars, cosine_q_ans, jaccard_ans_q | Which is the only national park located in Galicia, Spain: Timanfaya National Park or Atla… |
| q_4156_correct | 0 | 0.5273 | 0.5578 | overlap_answer_context, n_words, n_chars, overlap_answer_question, nli_ctx_contradicts_ans | The 2010 Insight Bowl was the 22nd edition, of the college football bowl game, played at S… |
| q_5922_correct | 0 | 0.6374 | 0.7706 | overlap_answer_context, n_words, n_chars, overlap_answer_question, nli_ctx_neutral_ans | Josefina Villamil Tinajero is a Mexican-American educator and author, she has worked as a … |
| q_4329_correct | 0 | 0.7779 | 0.9214 | overlap_answer_context, n_words, n_chars, nli_ctx_contradicts_ans, entity_overlap_ratio | What association football match between A.C. Milan of Italy and Liverpool F.C. of England,… |
| q_7876_correct | 0 | 0.7563 | 0.9064 | overlap_answer_context, n_words, n_chars, nli_ctx_contradicts_ans, nli_ctx_neutral_ans | Plutonium from the Hanford Site comes film which U.S. state? |
| q_4539_hallucinated | 1 | 0.2038 | 0.0663 | overlap_answer_context, n_words, n_chars, nli_ctx_neutral_ans, nli_ctx_contradicts_ans | The Cincinnati Reds record holder for most stolen bases by a rookie in a season is a nativ… |
| q_2910_hallucinated | 1 | 0.2300 | 0.0822 | overlap_answer_context, n_chars, n_words, cosine_q_ans, nli_ctx_contradicts_ans | Say Yes is the 12th episode of the horror series shown on which cable network? |
| q_1921_hallucinated | 1 | 0.0713 | 0.0214 | overlap_answer_context, n_chars, n_words, overlap_answer_question, jaccard_ans_q | in 1927, Kansas Jayhawks football team played their home games in a city that had how many… |
| q_5384_hallucinated | 1 | 0.0036 | 0.0118 | overlap_answer_context, n_chars, n_words, cosine_ctx_ans, nli_ctx_neutral_ans | Rock im Park 2001 is a live album and DVD by the American nu metal band Limp Bizkit, relea… |
| q_1729_hallucinated | 1 | 0.0790 | 0.0228 | overlap_answer_context, n_chars, n_words, jaccard_ans_ctx, nli_ctx_contradicts_ans | What was the 2010 population on the city where Lave Tree State Monument is located? |
| q_1739_hallucinated | 1 | 0.1692 | 0.0495 | overlap_answer_context, n_chars, n_words, overlap_answer_question, jaccard_ans_q | In which single by rap duo Mobb Deep was Curtis James Jackson III featured on? |
| q_2184_hallucinated | 1 | 0.0323 | 0.0152 | overlap_answer_context, n_words, nli_ctx_neutral_ans, nli_ctx_contradicts_ans, avg_word_len | What is the title of the book written by the host of "Family Feud"? |
| q_2251_hallucinated | 1 | 0.1832 | 0.0557 | overlap_answer_context, n_chars, n_words, cosine_ctx_ans, overlap_answer_question | ConnectU was founded by Cameron Winklevoss, Divya Narendra, and this professional rower? |
| q_2436_hallucinated | 1 | 0.2036 | 0.0661 | overlap_answer_context, n_words, nli_ctx_contradicts_ans, n_chars, cosine_q_ans | The inner urban neighbourhood Eureka Tower is in had a population of what in 2011? |
| q_5173_hallucinated | 1 | 0.0114 | 0.0126 | overlap_answer_context, n_words, n_chars, nli_ctx_neutral_ans, nli_ctx_contradicts_ans | What was the company orginally named first? |
| q_6989_hallucinated | 1 | 0.5180 | 0.5374 | overlap_answer_context, n_chars, n_words, nli_ctx_contradicts_ans, nli_ans_neutral_ctx | Which air transport operations hub was commanded by the most recent serial killer profiled… |
B3 zero-shot error cases (b3_error_cases.json)
FaithBench text is redacted (CC BY-NC-SA) — join locally via sample_id.
| Dataset | Task | Generator | Label | Score | Answer (truncated) |
|---|---|---|---|---|---|
| ragtruth | qa | gpt-4-0613 | 0 | 0.9996 | The proper way to dispose of a worn USA flag is either by separating the blue fi… |
| ragtruth | qa | llama-2-7b-chat | 0 | 0.9991 | Based on the provided passages, the recommended time to bake bratwurst in the ov… |
| ragtruth | qa | mistral-7B-instruct | 0 | 0.9996 | Based on the given passages, the proper way to dispose of a worn USA flag is eit… |
| ragtruth | qa | gpt-4-0613 | 0 | 0.9988 | Your Snapchat account may be temporarily locked if you have used third-party app… |
| ragtruth | qa | llama-2-7b-chat | 0 | 0.9975 | Based on the provided passages, patterned rollers can work effectively for vario… |
| ragtruth | qa | gpt-4-0613 | 0 | 0.9953 | A carbon footprint measures the environmental impact of an individual's lifestyl… |
| ragtruth | qa | llama-2-13b-chat | 0 | 0.9983 | Based on the given passages, here is a brief history of the word Gemini: Accord… |
| ragtruth | qa | gpt-3.5-turbo-0613 | 0 | 0.9981 | Based on the given passages, signs that your male coworker may be flirting with … |
| ragtruth | qa | gpt-3.5-turbo-0613 | 0 | 0.9990 | To bake a hard-boiled egg, place the eggs into a saucepan and pour in enough col… |
| ragtruth | qa | gpt-3.5-turbo-0613 | 0 | 0.9967 | The nervous system and endocrine system have both similarities and differences. … |
| ragtruth | qa | llama-2-7b-chat | 1 | 0.9995 | Based on the provided passages, the average temperature in Tallahassee, Florida … |
| ragtruth | qa | llama-2-7b-chat | 1 | 0.9987 | Based on the provided passages, the proper way to fertilize your grass is as fol… |
| ragtruth | qa | llama-2-70b-chat | 1 | 0.9986 | Based on the given passages, the average weather in Sausalito appears to be mild… |
| ragtruth | qa | llama-2-13b-chat | 1 | 0.9984 | Based on the given passages, forest fires are beneficial to conifers like jack p… |
| ragtruth | qa | llama-2-70b-chat | 1 | 0.9987 | To quickly get rid of mice, take immediate action and seal all entry points, inc… |
Version A error analysis (legacy)
Auto-tagged categories — review cases before paper use.
12 FP / 36 FN of 3000 test samples (sampled 10 FP / 10 FN).
| Category | FP | FN |
|---|---|---|
| label_ambiguity | 7 | 4 |
| short_answer_ambiguity | 3 | 6 |
Borderline / SHAP waterfall figures (legacy)
