Hugging Face Paper Proposes Finer-Grained LLM Safety Refusals

Real deployments rarely fit topic-level pictures, the blog post explains. The same base model may be adapted for a general assistant, an educational product, an enterprise system or a public-sector service, and each setting needs different boundaries within the same topic. A civics tutor and a public-sector assistant can share a model yet require opposite behavior on politics: both should answer factual questions about an election, but only one may need to refuse a request to write targeted political manipulation. A topic-level guard cannot express that split.

Examples show where existing tools fall short. LlamaGuard-3, the post points out, covers elections only as "factually incorrect information about electoral systems and processes," which excludes persuasion and manipulation and, at the same time, excludes the factual prompts a deployment must keep answering.

The paper, "Safety for Whom? Boundary-Aware Self-Distillation for Controlled LLM Safety Refusal," formalizes the setting as a topic universe, all political prompts in the experiments, containing a target-harmful subset. The intended policy is not to refuse all of politics, but to refuse the harmful subset while continuing to answer the benign complement. The ideal behavior is a sharp step: refuse inside the subset, answer everywhere else in the topic. A trained model, however, learns a refusal probability that approximates the target and can push refusal outward into the benign complement near the boundary.

The researchers use political persuasion as a testbed since manipulative persuasion can cause real harm while factual political information remains legitimate, which is exactly the case where topic-level refusal is too blunt. To build training data, they use self-generation: take the target model, steer it toward a refusal on each harmful prompt, and keep traces a guard model verifies as genuine refusals. Examining that standard pipeline at boundary level exposes three weaknesses, the post says.

The first is a coverage gap. A single steering attempt does not always yield an accepted refusal, and those prompts are silently dropped from training. In an audited pool, single-shot generation dropped 19.88% of prompts — 8,009 prompts — and these may be the hardest examples. To avoid discarding them, the researchers apply an escalating retry strategy that resamples the same prompt with progressively stronger steering, bringing residual failures down to 0.20%, or 79 prompts. Coverage repair leaves 40,293 harmful training prompts where a naive pipeline would have thrown thousands away.

The second weakness is downside reactions. Safety tuning tends to produce false refusals on benign prompts that look superficially dangerous. To compensate, the researchers add 11,955 verified surface-dangerous benign prompts across 18 semantic types, so the model sees safe prompts with dangerous-looking wording during training rather than only at evaluation.

The third is that ordinary harmful and benign splits do not measure the shape of the boundary at all. A model can improve its harmful-refusal rate simply by expanding refusal into nearby permissible prompts, and a topic-level metric would call that an improvement. The paper therefore uses held-out harmful-benign pairs, 1,539 per side, to measure both sides of the boundary directly.

Training on political refusal data works in the obvious sense. On Qwen3-8B, the escalated-coverage model raises in-distribution political refusal from 9.47% to 84.75%. It also transfers: the mean unsafe-response rate across three broader harmfulness benchmarks — HarmBench, StrongREJECT, and WildJailbreak — scored by LlamaGuard-3, falls from 26.26% to 0.14% in the strongest configuration.

Reported alone, those numbers look like a clean win, the post cautions. At the same checkpoint, over-refusal on XSTest rises from 2.00% to 74.00%.