Inside Strikersoft's AI Skill Library: How We Proved It Actually Works
Ask a team whether their new AI coding habit is helping, and most will say yes.
Ask how they know, and the answer is usually a feeling, not a number.
Writing it down isn't the same as knowing it works
Like a lot of teams, we've been building a shared library of "skills": written-down pieces of engineering knowledge, coding conventions, testing patterns, how we expect a controller or a repository to look, that our AI agents load depending on what they're working on. The obvious risk with a library like this is that it just grows forever, full of rules nobody has checked are still true. Ihor Segodin, our Java Team Lead and the person who started the library, wanted proof of that, not just a feeling.
Building a way to test it, not just trust it
That's what led Andrii Vasyliev, one of our Java engineers, to build an evaluation harness. It runs a coding task twice, once with a given skill loaded and once without, and scores what comes back against pass/fail criteria written specifically for that skill. Each skill in the library now ships with its own eval spec alongside it: trigger and fixture files that define when the skill should fire, and just as important, when it shouldn't. A positive scenario checks that a real match gets caught; a negative one checks the skill doesn't overreach into logic it has no business touching. Running the harness produces a pass rate, the conditions it was tested under, and a timestamp, all logged alongside the skill itself.
In one run, on the same task set, the agent passed 100% of the checks with the skill loaded and 78% without it. That gap is the point, not the demo.
Michael Purnyn underlined that ISO-compliant AI governance is a routine, not a checkbox. By repeatedly testing models and tracking the results, organizations can quickly see when a new model underperforms and fix the issue before it impacts real users. The same logging catches slower drift too: track the numbers over time and a dip shows up within a week, whether the cause is a new model version or the context around the skill quietly shifting, instead of surfacing a month later as a production incident nobody traces back to the change.
What this isn't, yet
We're not overstating where this is. The harness is something Andrii built himself over the past few weeks, not a mature internal product, and the numbers so far come from a handful of tasks, not a broad benchmark. And a skill that helps today isn't permanent: swap the underlying model, and a rule that used to sharpen the output can start getting in the way, or simply stop mattering. Asked directly how effective the whole skills system has been, Ihor was straight about it: he's not ready to say yet.
Skill evolution, not a finish line
That's exactly why we keep testing rather than treating "we wrote it down" as the finish line. Internally we've started calling it skill evolution: don't just write down what worked once, keep proving it still does.
Related reading: How do you know AI is actually helping? Measure it
