A skill is easy to write. It’s a markdown file, some instructions, maybe a script or two, and a folder of reference material that teaches an agent how to do one specific job. You can write one in an afternoon. That’s not where an AI agent platform falls apart.

They fall apart three weeks after deployment, once the agent is carrying fifteen skills at once and the routing starts breaking in ways nobody can quite reproduce. A skill that triggers too broadly pollutes the context window on turns that have nothing to do with it. A skill with a bloated body drags down the conversation around it, even on the turns where it never fires.

Researchers from Anthropic, Google, OpenAI, UC Santa Barbara, and MIT put numbers on this earlier this year. Their SkillsBench benchmark ran skills against 84 real tasks. On 16 of them, adding a skill made the agent perform worse, in one case a 39-point drop in success rate. You won’t see that in a demo. A demo only ever loads the one skill you’re showing off.

Zoom out, and it matches the broader picture. Gartner expects more than 40% of agentic AI projects to be canceled by the end of 2027, and the reasons it gives are escalating costs, unclear ROI, and inadequate risk controls. Most of what gets killed was never a bad model. It was a system nobody could keep reliable once it left the proof-of-concept stage.

Enterprise AI platform – the folly of managing skills on its own

Once a skill passes testing, the instinct is to treat it as done. But a skill on its own can’t tell you whether it’s still doing its job once the model changes, the harness changes, or five new skills get added to the same library. All three happen on a schedule you don’t control.

What you actually want to manage is the skill and its eval together. An eval is a test committed in the same folder as the skill, versioned as part of the same artifact rather than something you get to later. It decides whether the skill is allowed to ship at all.

The mechanism is a baseline: run the task with the skill, then run it again without it, and look at the difference. You can’t tell whether a skill is helping from its output alone. A skill that can’t beat its own baseline isn’t ready, no matter how good it looks in isolation. Then you add the checks around that core test — whether it triggers on the prompts it should and stays quiet on the ones it shouldn’t, whether it calls the right tools in the right order, and whether adding it breaks something that already worked.

Keep doing this, and the library starts to compound instead of drift. A year in, you’re not holding fifty skills of unknown quality. You have a set of tested assets that get more valuable every quarter, because what’s stored isn’t just the procedure. It’s proof the procedure still works.

The SkillsBench numbers make the case: curated, evaluated skills lifted task performance by an average of 16.2 percentage points. Skills an agent wrote and checked for itself, with no outside eval, added basically nothing — a 1.3-point loss on average. Same category of thing, “a skill,” wildly different outcome. The only variable was whether anyone had actually tested it.

Philipp Schmid, who works on agent tooling at Google DeepMind, asked the obvious question in a piece on this last year: “You wouldn’t ship code without tests, but why ship skills without evals?” Nobody has a good answer to that. (More from Schmid here.)

It gets harder over time, not easier

Most teams haven’t run into this part yet. An agent is a model plus a harness — Claude Code, Codex, Antigravity, Google’s ADK, whatever you’ve standardized on. The model does the reasoning. The harness holds the conversation, calls the tools, and decides which skills to load. Both pieces are moving fast, and neither is going to sit still for you.

The skill itself is portable. The same folder runs on any harness that speaks the format. The eval score isn’t portable, and it isn’t stable either. Every harness routes skills differently, so the same skill gets a different trigger score depending on where it runs. Change the underlying model and the numbers move again. A description tuned for one model can under-trigger on the next one. An eval that passed last quarter can start failing this quarter, and it’s not because the skill got worse. The ground under it moved.

Most teams respond to this by averaging it all into one score for the org. Wrong move. The spread is the signal. Knowing a skill is reliable on Claude and ADK but weak on Codex, and that it shifted after the last model update, is exactly what a team running across vendors needs to know. A single averaged number erases all of it.

AI model evaluation – what changes once skill is integrated with eval?

• New skills have to earn their place. A skill has to beat its own baseline, and it can’t break an eval that already passed elsewhere in the library — so a bad update doesn’t quietly take three good skills down with it.
• Model and harness swaps stop being fire drills. Every upgrade is a rerun, not a rewrite. You find out what broke in minutes instead of from a support ticket three weeks later.
• You get a diagnosis instead of a vibe. Per-harness, per-model results tell you which combination is actually degrading and why — which is the entire point of not averaging.

Here’s an example:

A compliance-monitoring agent we built didn’t need to become four separate agents to cover four regulated industries. One universal agent, four skill folders — financial services, healthcare, manufacturing, retail — each with its own evals, each versioned and owned on its own. A new industry means a new skill and a new baseline. Not a new build.

None of this argues for slowing down on skills. It argues for treating them like code: dependencies, with tests, owned by whoever owns the underlying knowledge, re-evaluated whenever the model or harness underneath them changes. The model and the harness are going to keep turning over on somebody else’s release schedule, not yours. Skills and evals, versioned and rerun together, are the one layer your organization actually owns.

Conclusion

If your agent’s routing has started breaking in ways testing never caught, that’s usually the eval layer missing. Are you evaluating AI for your enterprise? Get in touch with Onix’s AI experts to schedule an AI readiness assessment and start building a skill library that compounds instead of drifting.