← Return to studyEsc

FROM THE BOOKSHELF / A PAGE TO KEEP

AI 洞察
51 分钟

Skill.md Is the New npm — And We're Watching the Same Mistakes Again

胡新宇

Published on 2026-08-22

A single SKILL.md file hit 18K GitHub stars. In three weeks, JetBrains shipped a Skills Manager, Alibaba open-sourced skill-up (CI for Skills), and Tencent's SkillHub hit 100K Skills. The agent framework wars are over.

Skill.md Is the New npm — And We're Watching the Same Mistakes Again

On August 11, 2026, a single GitHub repo — ayghri/i-have-adhd — crossed 18,000 stars for shipping almost nothing but a SKILL.md file. The repo trains no model, ships no agent framework, registers no MCP server. One Markdown file, ten rules: answer first, steps second, action at the end. That was enough.

Three weeks later, JetBrains shipped a native Skills Manager inside IntelliJ IDEA. Alibaba open-sourced skill-up, a CI-style regression-test framework for Agent Skills. Tencent's SkillHub crossed 100,000 indexed Skills and 10 million monthly downloads. freeCodeCamp published a 2,800-line "AI Evaluation Engineering" handbook. The same week.

The inversion: from agent framework wars to the Skill economy

The agent framework wars of 2024-2025 were the equivalent of arguing whether to write your own HTTP server. The real product — the npm package equivalent — is the Skill. And the infrastructure around it is being built right now, by every major player, at the same speed npm reached in 2011.

The Inversion: From Agent Code to Skill Files

Twelve months ago the hot debate in AI engineering was about agent frameworks. LangChain or CrewAI, AutoGen or a hand-rolled orchestrator. Which wiring library wins. The August 2026 discourse sounds nothing like that.

The question now is: which Skill did you install?

Four product launches in the same three-week window prove the shift:

  • JetBrains shipped Skills Manager natively in IntelliJ IDEA on Aug 12, with a Skills Repository that the company pre-verifies for security vulnerabilities. The default repo lives at github.com/JetBrains/skills and individual Skills can be installed globally, per project, or per agent. (Source: JetBrains Blog, 2026-08-12)
  • Alibaba open-sourced skill-up on Jul 23 — a CLI eval framework purpose-built for Skills, with evals/eval.yaml plus per-case YAML, JUnit XML output, HTML reports, and a CI exit code. The repo is at github.com/alibaba/skill-up. (Source: Alibaba Engineering, 2026-07-23)
  • Tencent's SkillHub crossed 100,000 indexed Skills and 10M+ monthly downloads by Aug 10, and built TRACE — a five-dimension parallel eval system (Trust, Reliability, Adaptability, Convention, Effectiveness) that cut eval time per Skill from ~10 minutes to ~1 minute by fanning out across five sub-agents that share one Skill context. (Source: Tencent Engineers, 2026-08-10)
  • freeCodeCamp published "AI Evaluation Engineering: Build a Production-Grade LLM Evaluation Platform from Scratch" on Aug 10 — a 2,800-line handbook treating evals as the new unit of trust for AI systems. (Source: freeCodeCamp, 2026-08-10)

Three layers of infrastructure — registries, eval frameworks, IDE managers — are forming at the same speed. That's the npm 2011 fingerprint, and almost nobody is talking about it that way.

The npm Parallel

The story maps almost one-to-one to early npm. Anyone who shipped JavaScript in 2010-2012 has the muscle memory to recognize what's about to happen.

npm 2010-2012Agent Skills 2026
"Just include the JS file""Just include this Skill"
npm install lodashclaude plugin marketplace add ayghri/i-have-adhd
Registry race: domain squatting, dup packagesSkillHub 100K Skills, 80/20 rule, similar grab-and-publish
npm test becomes the trust signalskill-up run / TRACE eval becomes the trust signal
package.json + lockfile pins versionsSkill version pinning still loose — Skills ship to 5+ agents
Security: event-stream hack (2018)Security: JetBrains pre-verifies Skills before listing

The most important parallel is the trust problem. npm took roughly six years to standardize security (event-stream in 2018 was the wake-up call that pushed npm audit into every install). Skills are six months old and JetBrains is already ahead of where npm was in 2014 — Skills Repository entries are pre-verified for security vulnerabilities before they reach the IDE, and the documentation explicitly recommends that teams maintain their own internally-reviewed Skill Repositories rather than pulling from the open internet.

But the install-and-trust loop is identical. Most developers will skip the eval. Most teams will not pin versions. Some Skill will ship a malicious payload to thousands of agents, and that incident will be the event-stream of the Skills era. This is not a prediction. It's an arithmetic of incentives.

The Eval Problem — Why This Time Might Be Different

The freeCodeCamp handbook opens with a story every AI engineer should memorize:

A team builds a RAG application for legal research. They test it with 40 hand-picked questions. The answers look good, so they ship it. Three weeks into production, a paralegal flags an answer that cites a statute incorrectly. The faithfulness score is 0.91. Healthy. They check answer relevancy. Also healthy. What they didn't check: context recall. The retriever had been silently failing on multi-hop legal questions. The model, being a good language model, had been constructing plausible-sounding answers from the partial context it received. The system passed every eval the team ran. It failed on the eval they didn't know they needed.

That sentence — the eval they didn't know they needed — is the entire AI engineering discipline in 2026. Every dashboard you ship without a context recall metric is a faithfulness 0.91 waiting to happen.

TRACE: parallelizing five eval dimensions cuts Skill review from 10 minutes to 1 minute

The infrastructure is finally catching up:

  • Alibaba's skill-up solves the "Skill silently regresses after a one-line description edit" problem with a single command: skill-up run ./evals/eval.yaml. The framework produces pass/fail per assertion, total pass rate, token consumption, and an exit code you can wire into CI as a merge gate. (Source: Alibaba Engineering, 2026-07-23)
  • Tencent's TRACE parallelizes five independent eval dimensions across five sub-agents, sharing a single Skill context, then aggregates results. Same Skill, ten times the throughput, and the per-dimension eval logic stays independently maintainable so adding a sixth dimension doesn't rewrite the pipeline.
  • Simon Willison's smevals — his third iteration on eval tooling — formalizes the vocabulary: eval / task / config / run / grader / check. Runners execute tasks, graders score them, checks are the atomic assertions. Output is a static HTML report you can host anywhere. (Source: Simon Willison's Weblog, 2026-07-31)
  • GitHub Copilot's team published an evaluation methodology for the agentic harness across models and tasks in June, treating harness choice as a first-class performance variable rather than an implementation detail. (Source: GitHub Blog, 2026-06-25)

The actionable primitive for the rest of 2026 is not "pick the best framework". It's "write a Skill, then write evals for it". That two-line habit is the difference between a team whose agents quietly degrade and a team whose agents improve with every commit.

What to Do This Week

The Skill economy is moving fast. Here is what an engineer reading this on August 22 should do before Friday:

  1. Inventory your Skills. Run claude plugin list, check your /skills directory, or open the JetBrains Skills Manager. Most engineers will discover they have eight Skills installed and zero evals for any of them. That is not a stable equilibrium.

  2. Pick one Skill and write an eval. Copy Alibaba's evals/eval.yaml pattern. Even for a personal Skill — a custom code-review Skill, a release-plan Skill, a Skill that wraps your team's deploy script. Three cases, five checks each. Run it on every change. The cost is an afternoon; the value compounds forever.

  3. Pin Skill versions. Until lockfile semantics stabilize across the major agents, treat every Skill.md change like a dependency upgrade. Read the diff. Run the eval. Pin the commit hash if you have to. The Skills that ship to five different agents today will drift unless you pin them. If you are running the same Skill on Claude Code and Codex concurrently, expect divergence the moment either agent updates its harness — this is the Skills-era equivalent of node-gyp breaking on a minor Python release.

  4. Watch the registry layer, not the framework layer. JetBrains Skills Repository, Anthropic's Skills distribution, ClawHub, Tencent SkillHub — the winner is the one that solves discovery plus verification. Framework wars are a solved problem; registry wars are not. The platform that ships both a curated catalog and a verified-by-default trust signal will absorb the others the same way npm absorbed cnpm and bower.

The Sharp Part

The agent framework wars are over. The Skill economy just started.

Most teams will repeat npm's 2010-2015 mistakes: install everything, evaluate nothing, get burned by a Skill that quietly degrades six weeks after a one-line edit. The teams that win will be the ones who treat Skill + eval as the atomic deployable — not the ones who picked the cleverest framework back when that was still the question.

The new question is not which agent. It is which Skill, and whether anyone wrote the eval that proves it works.

#AI Agent#AI工程#Claude#Agent#技能