Deleting half a skill and proving nothing broke

Shortening two caliper skills, grill-skill and evaluate-skill, by about 45% using a non-inferiority eval bar: prove the shorter version is no worse before shipping the cut.

· Updated September 12, 2026

I was scrolling X when a post from Matt Pocock caught me. [2] His point: agent-authored skills are littered with no-ops. Lines like “be thorough” or “make the commit message very detailed.” Models are already trained hard to do those things, so the line changes nothing and costs tokens on every load. His test is simple. Remove the line. If the output does not change, it was a no-op. [3]

Editing a skill almost always means adding to it. I wanted to go the other way. The hard part was never the deleting. It was proving that smaller did not mean worse.

Skills are always-loaded context once they fire. Every line in a SKILL.md is paid for on every invocation, whether or not it ends up mattering. Pocock’s writing-for-agents skill says to prune to the shortest document that still behaves predictably, and to hunt no-ops “sentence by sentence.” [1] Perplexity’s skill guide offers the same test in different words: for every sentence, ask “would the agent get this wrong without this instruction?” If not, it cannot afford to be there. [4] The second clause is the hard one. It is easy to delete lines. It is hard to know whether you deleted a load-bearing one.

I took two skills, grill-skill [5] and evaluate-skill [6], and cut each by roughly 45 percent. The claim I wanted to defend was not “these are better now.” It was narrower:

The shortened skill is no worse than the original, and still clearly beats the raw agent with no skill at all.

That is a non-inferiority test, and it changes how you evaluate.

Ablation is all you need

The reflex when you touch a skill is to add: a sharper instruction, another edge case, a fix for the slop the model wrote last time. That is a legitimate move. It is also the only move most people have, which is how a skill ends up at 160 lines.

A skill should get shorter over its life, not longer. Every model release absorbs a bit more of what your skill was compensating for, so the same text is worth less each month you leave it alone. And context is not free real estate. Fill it with instructions the model no longer needs and you are paying tokens to make it dumber. SkillsBench found the same thing at benchmark scale: focused skills with at most three modules beat exhaustive bundles. [7]

So the exercise was ablation. I held each skill’s eval fixed and varied only the skill text, comparing pass@k across three variants:

control   = the current full skill
candidate = a shortened variant
baseline  = the raw agent, no skill at all

The baseline matters more than it looks. Without it, a shortened skill that quietly stopped working could still “pass” against a weak control, and you would never know the skill had stopped contributing anything. The baseline is the floor. If the candidate sinks toward it, the cut removed something real.

The decision rule:

  • Non-inferior at k=3. The candidate shows no task-level regression against the control.
  • Still earns its keep. The candidate still clearly beats the no-skill baseline.
  • Equalling or beating the control is a bonus, not a requirement. The claim is “no worse,” not “better.”

Debugging ran at k=1 to shake out spec and prompt bugs cheaply. Only the accept-or-reject decision ran at k=3. Both skills used the claude-code backend so the numbers are comparable.

Harden the eval before you trust it

Partway in I realized the existing evals were bad. They did not exercise the prose I intended to cut. They tested YAML serialization and structure, the mechanical output, and never touched the interview discipline or the judgment guidance that made up most of the text. An eval that cannot feel the damage makes every cut look safe.

So I rewrote the evals first. It feels backwards, since you are strengthening the thing that might block your change, but it is the only way the “no worse” claim means anything. A green light from an eval that can’t see the wound is not evidence.

For grill-skill, the old eval pre-supplied the whole task structure, so it only ever tested serialization. The new one has three tasks:

  • Interview discipline (first turn). An open prompt, no structure supplied. The judge checks that the agent interviews the user and stops, rather than inventing answers and charging ahead. A deterministic assert confirms no spec was written during the turn.
  • Gap-fill detect-and-ask. An existing eval is present. The judge checks that the agent reports the existing tasks and asks what is missing before touching anything, with an assert that the existing spec is left unmodified.
  • Serialize a valid spec. A fully-directive prompt, graded by a deterministic structure assert.

The first task taught me something worth writing down. The prompt has to signal that the user is present and will answer. Without that signal, an agent that just does the task instead of interviewing is being rational, not disobedient. The eval was measuring the wrong thing until the prompt made the human’s presence explicit.

grill-skill: 143 → 55 lines

The grill-skill cut was the clean one.

BeforeAfterReduction
SKILL.md143 lines / ~1240 tok55 lines / ~703 tok~43%

What came out:

  • The inline caliper command blocks in the interview phases that duplicated REFERENCE.md. Those became a pointer: one source of truth instead of two copies that drift.
  • The “quoted script” instructions, long blocks of exact wording to say to the user, collapsed into short directives.
  • The intro paragraph that restated the description.

What stayed, verbatim in spirit: the load-bearing interview discipline. Interview, confirm, one question at a time, do not run ahead. That is the behavior the skill exists to enforce, so it does not get compressed.

The result at k=3:

TaskBaseline (no skill)Control (full)Candidate (short)
Interview discipline1/13/33/3
Gap-fill: detect & ask first0/12/33/3
Serialize a valid spec0/10/30/3

On the informative tasks the candidate matches or beats the control, and it beats the baseline where the skill earns its keep. The gap-fill discipline is 3/3 with the skill and 0/1 without it. That row is the whole argument. The skill’s job is to make the agent ask before acting, and the raw agent does not do that on its own.

The third task is 0/3 everywhere. Control, candidate, and baseline all fail it. That is not a regression. It is a pre-existing bug where the agent writes a spec missing the top-level skill: block on a fully-directive prompt. It is non-informative for the ablation because it is equal across every variant, and it is a real bug worth a separate fix. An eval that fails uniformly tells you nothing about your cut. It can still tell you something about the skill.

Verdict: confidently reduced.

evaluate-skill: one aggressive cut, then bisect

evaluate-skill did not go as smoothly, which is the more useful story.

I started with a maximally aggressive candidate: disclose everything to REFERENCE.md and leave SKILL.md as a thin shell.

v0-full   159 lines / ~1713 tok
v1-short   42 lines / ~734 tok   (~57% smaller)   rejected
v2-milder  62 lines / ~922 tok   (~46% smaller)   shipped

v1 moved four judgment sections out of SKILL.md into REFERENCE.md: the artifact-versus-transcript checks, the task-quality checklist, common eval patterns, and how to write expect: rubrics. About 70 lines of always-loaded text, gone.

It regressed. On “create a valid spec,” v1 dropped from 3/3 to 1/3 against the control. The aggressive cut had removed too much structural cue. The agent no longer had enough inline scaffolding to reliably produce a valid spec skeleton. This is the failure the whole method exists to catch, and it is why you do not ship on a read-through.

So I bisected. v2 restores a compact spec skeleton inline (skill.path, backend, the judge shape) while keeping the judgment prose in REFERENCE.md. The structural cue the agent needs to emit valid YAML lives in SKILL.md. The reference material it only sometimes needs lives one hop away.

v1 removed:  structural skeleton + judgment prose   -> regressed
v2 removed:  judgment prose only                    -> parity

v2 matched the control across the spec-creation tasks, within one-sample noise, and beat the baseline on valid spec creation. I also added a disclosure check, a short review task that plants a vague expect: and a script-checkable fact, to confirm the agent still reaches the judgment guidance now that it lives in REFERENCE.md. Progressive disclosure only works if the agent actually opens the drawer.

An honest caveat: the final clean k=3 confirmation for v2 was cut short by a spending cap on the backend account. v2 produced correct specs on every attempt it was allowed to run, and the failures were rate-limit artifacts rather than skill regressions. A clean re-run after the cap resets would move this from “moderate confidence” to “high.” I would rather write that down than round it up.

De-noising the asserts

My first deterministic asserts were too brittle, and the brittleness made the control flaky, which muddied every comparison.

They did exact-match on paths, and they crashed when the agent wrote judge: codex as a string instead of judge: {backend: codex}. A crashing assert is not a failing task. It is noise that shows up in both columns and hides the signal you care about.

The fixes:

  • Accept string-or-dict backends (judge: codex and judge: {backend: codex} both valid).
  • Tolerate ./SKILL.md versus SKILL.md.
  • Treat invalid YAML as a clean fail, not a crash.
  • Soften the review rubric to accept any wording that makes both required points.

The final verdicts come from this de-noised run. When the control is flaky, the numbers tell you about the harness rather than the candidate you set out to measure.

The size summary

SkillBeforeAfterReduction
grill-skill143 lines / ~1240 tok55 lines / ~703 tok~43%
evaluate-skill159 lines / ~1713 tok62 lines / ~922 tok~46%

Token counts are approximate (characters over four, since there is no exact tokenizer in the environment), but the relative reductions hold under any tokenizer. Line counts are exact.

The description is the most expensive line in the skill

I rewrote both descriptions too, and this is the part I would push hardest on if you only change one thing.

The description is not just another section of the skill. The body loads when the skill fires. The description is in context on every single turn, because it is what the router reads to decide whether the skill applies at all. So it is the most expensive line in the file, and it charges you on all the turns where the skill never runs, which is most of them.

So the description has to earn that cost. I rewrote both to lead with the job word (grill, evaluate) and dropped “caliper,” the tool the skill happens to be built on. A router deciding relevance needs to know what the skill does, not what library sits underneath it. The tool name is fine in the body. In the always-loaded description it is dead weight you pay for on every turn. Perplexity’s guide calls the description “the hardest line in the Skill” and tells engineers to write what users say when they want the thing, not what the skill does. [4]

The honest gap is that I never measured whether the new descriptions trigger better. That needs its own test: fire each skill against a batch of prompts and check it activates on the ones it should while staying quiet on the ones it shouldn’t. Until that exists, these rewrites are reasoned, not proven.

The loop

This is the same shape as improving a skill, run in reverse:

Shortest plausible cut
  -> harden the eval so it can feel the cut
  -> ablate: candidate vs control vs baseline
  -> regressed? bisect to a milder cut
  -> ship the smallest cut that stays non-inferior

The discipline that makes it work is refusing to trust a read-through. Every line you want to delete looks deletable, which is exactly why you want to delete it. The eval is the only thing standing between “looks redundant” and “was redundant.”

Which is the whole argument, really. A smaller skill saves tokens on every invocation and that is worth real work. A skill that quietly stopped working costs more than it ever saved, and you find out on a real task, months later, when you have forgotten you touched it. Measure the cut against a baseline or don’t make it.

Postscript, September 2026

The trigger test now exists. Caliper tasks can carry an activates: assertion that checks which skills the agent chose to load, scored separately from whether the task passed, and a --ablate flag that reruns a spec with a named skill removed to produce the baseline column automatically. [5] The description rewrites above are still unproven until I run them through it.

Pocock’s skill was renamed in July from writing-great-skills to writing-for-agents. The no-op hunting section survived the rename. [1]


References

[1] Matt Pocock, "writing-for-agents," from mattpocock/skills. Published June 17, 2026 as writing-great-skills, renamed July 23, 2026. github.com

[2] Matt Pocock (@mattpocockuk), "Take a look at your favourite skill... Agent-authored skills are LITTERED with no-ops," X, June 2026. x.com

[3] Matt Pocock (@mattpocockuk), follow-up clarifying that a phrase is a no-op in the context of a given skill, when removing it produces no change, X, June 2026. x.com

[4] Perplexity Engineering, "Designing, refining, and maintaining agent skills at Perplexity," May 1, 2026. perplexity.ai

[5] Caliper, "grill-skill," in a local-first eval harness for Claude Code, Codex, Pi, and Hermes skills. github.com

[6] Caliper, "evaluate-skill," same repository. github.com

[7] X. Li et al., "SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks," arXiv:2602.12670, February 2026. arxiv.org

A sunlit Brooklyn brownstone street in autumn: parked cars, iron railings, stoops, and a canopy of turning leaves.