Detect LLM Hallucinations with HCPD: Zero-Source, 88% AUROC - Aoyii

HCPD is the ICML 2026 method that detects LLM hallucinations from the question-answer text alone, no logits, no hidden states, no knowledge-base lookup, and it hits 88.19% AUROC on Llama-3.1-8B (89.62% on Qwen3-8B), roughly 10 points above the next-best baseline. Training and evaluation code are fully open source on GitHub, built on the open-r1 stack with GRPO + LoRA + vLLM. This guide covers the mechanism, the repo layout, and the exact commands to reproduce the results.

Why you should care

Hallucination is the number-one blocker for putting LLM output into production, in education, healthcare, and any decision-support surface. Most detectors cheat in one of two ways: they read output probabilities or internal representations, or they fact-check against a knowledge base. Neither works when you call a black-box API and only ever see the final text, and reference-based checks are slow and brittle when coverage is thin. HCPD is built for that zero-source setting: given only the (question, answer) pair, it decides whether the answer can be trusted. That covers every model you do not control, including locally deployed ones that drift silently from the official version.

How HCPD works

The core is the Human-like Criteria Probing (HCP) mechanism, which turns detection into a dynamic, explainable evaluation instead of a fixed classifier:

Adaptive criteria - the agent starts from five generic dimensions (factual correctness, logical consistency, semantic accuracy, temporal consistency, social compliance), then generates finer-grained criteria with an importance weight per criterion for each specific (q, a) pair. A history question, for instance, weights people, places, and dates heavily.
Weighted scoring - each criterion gets a score, and the weighted sum becomes the final truthfulness score. The criteria, weights, and per-dimension analysis are all emitted, so the verdict is auditable.
GRPO reward alignment - prompting alone is not stable enough, so the scoring agent is trained with GRPO. Weak supervision labels come from semantic consistency (BLEURT) between candidate answers and the reference, mapped to a 1-10 fine-grained scale, with no expensive human annotation.
Multi-sampling aggregation - at inference, several full evaluations run in parallel and are aggregated, cutting the variance of single-sample generation.

GRPO here is the same RL family used in self-play coding models that improve without human labels - a reward signal replaces manual supervision.

What you get in the repo

The TRISKEL10N/HCPD repo is an open-r1 fork. The scoring agent is Qwen2.5-7B-Instruct; evaluated targets are Llama-3.1-8B and Qwen3-8B; benchmarks are TriviaQA, SciQ, NQ Open, and CoQA. Requirements are 2x 80GB NVIDIA GPUs, CUDA 12.4, Python 3.11, and PyTorch 2.6. Training uses LoRA (r=8, alpha=32) with GRPO, colocated vLLM for rollouts, and a BLEURT-based reward.

Setup and quick validation

# 1. Install the environment
bash setup.sh
conda activate HCPD

# 2. Build the hallucination datasets (auto-download to ./.cache)
bash generate_datasets.sh

# 3. Verify with the released pretrained checkpoints (Google Drive)
bash quick_validation.sh

quick_validation.sh runs haldet_eval_vllm.py across all four datasets and both target models using the released LoRA checkpoints. On lighter GPUs, run one dataset first before the full sweep.

Train and evaluate a target model

The run scripts show the full pipeline. Here is the TriviaQA script, trimmed with comments:

# scoring agent to be aligned
MODEL_PATH="Qwen/Qwen2.5-7B-Instruct"
export VLLM_WORKER_MULTIPROC_METHOD="spawn"

# Train: GRPO + LoRA, vLLM colocated for rollout generation
accelerate launch --config_file recipes/accelerate_configs/zero3.yaml \
  src/open_r1/haldet_grpo.py \
  --config recipes/Qwen2.5-7B-Instruct/grpo/config_demo.yaml \
  --seed 42 --vllm_mode colocate --vllm_gpu_memory_utilization 0.5 \
  --metric_type bleurt --dataset_name TriviaQA_llama3.1-8B \
  --learning_rate 0.0002 --beta 0.05 --model_name_or_path $MODEL_PATH

# Eval: loop over the 10 saved checkpoints
for idx in {1..10}; do
  python src/open_r1/haldet_eval_vllm.py \
    --base_model $MODEL_PATH --dataset_name TriviaQA_llama3.1-8B \
    --lora_path "data_bleurt/seed_42/.../checkpoint-$((1245*idx))" \
    --seed 42 --metric_type bleurt
done

Key knobs in recipes/Qwen2.5-7B-Instruct/grpo/config_demo.yaml: num_generations=10 rollouts per prompt, max_prompt_length=512, reward_funcs=score_align, and LoRA on all attention projections.

Practical tips

Works on any black-box model. HCPD only needs the question and the answer, so it runs against GPT, Claude, Gemini, or any API without special access. For fast local serving, see the OpenVINO local inference acceleration guide.
Audit the verdicts. The emitted criteria, weights, and per-dimension scores are useful for compliance and for debugging why a RAG answer got flagged.
Tune the aggregation. Multi-sampling trades latency for stability - fewer samples on low-latency paths, more on high-stakes ones.
Retrain on your domain. The weak-supervision scheme lets you rebuild the reward on your own Q&A data and realign the scoring agent cheaply.

FAQ

Q: Do I need access to the target model logits or internals?
A: No. HCPD is a zero-source detector - it only reads the question and answer text, which is why it works on black-box APIs.

Q: How accurate is it?
A: 88.19% AUROC on Llama-3.1-8B and 89.62% on Qwen3-8B across four QA benchmarks, about 10 points above the next-best baseline.

Q: What GPU do I need to reproduce the training?
A: The paper setup uses 2x 80GB NVIDIA GPUs with CUDA 12.4. For a smoke test, run quick_validation.sh with the released checkpoints first.

Resources

Paper: arXiv 2606.12900
Code: github.com/TRISKEL10N/HCPD
Benchmarks: TriviaQA, SciQ, NQ Open, CoQA