Why AI Agent Skills Work Until They Suddenly Don't

A new preprint from researchers at Princeton, Stanford, UC San Diego, USC, and Johns Hopkins, Demystifying Agent Skills: Why They Work-Until They Don't, puts hard numbers on something a lot of teams building AI agents have sensed but couldn't quite prove: giving an agent a reusable "skill," a written package of procedural know-how, reliably helps until it suddenly doesn't, and the researchers can now say precisely why.

What a skill does, measured across 8,135 trials

The team ran controlled tests across 8,135 trial records and multiple agent setups, comparing three conditions: no prior experience at all, raw "workflow memory" (unedited traces of past attempts), and a distilled skill document. Skills won, but the more interesting finding is why. Skills weren't outperforming raw memory because they contained more facts. Only 4.5% of the benefit came from injecting new knowledge the agent didn't already have. The other 65.7% came from what the researchers call procedural anchoring, skills work mainly by stabilizing the order of operations: what to check first, which tool to reach for, what verification step not to skip. Skills beat workflow memory by 6.06 percentage points on matched tasks, largely because raw memory drags along failed attempts and noise that a clean skill document strips out.

Where the same mechanism turns into a liability

The failure side of the study is just as concrete. As a skill library grows from 5 candidate skills to 100, the agent's precision at actually using the correct one in execution collapsed from 29.6% to 3.3%. That's not a retrieval failure in the traditional sense: the agent could often still find the right skill in an offline test, but during live execution it frequently pulled in extra, irrelevant skills alongside it, or applied a plausible-looking one too rigidly to a situation it didn't quite fit. The researchers documented a specific failure mode, skill guidance misapplied or ignored, that shows up in 10% of skill-based runs versus under 1% for raw memory. In short, having the right skill available and actually using it correctly turned out to be two separate problems.

Why this matters for anyone building on agent skills

The practical takeaway for teams deploying AI agents, including anyone building content or marketing automation on top of tools like Claude's skill system, is that bigger skill libraries aren't automatically better. Retrieval degrades faster than most people would expect, and success depends less on cramming in more procedural documents than on keeping libraries small, distinct, and well-matched to the tasks agents actually face. The researchers frame this as a lifecycle problem: a skill has to be built well, found correctly, and applied with judgment, and failing at any one of those stages undoes the benefit of the other two.

If your team is building or maintaining a growing library of AI skills and workflows, this research is a useful gut check before adding "just one more skill" to the pile. Our growth strategy work applies exactly this kind of discipline to AI tooling decisions, and our AI marketing services team can help you build agent workflows that hold up as they scale.