Your agent mostly sees one line of each skill. Write that line well.

An AI coding agent can have dozens of skills installed. Each skill is a page of instructions, but the model does not see the pages up front. It sees a list with one line per skill: a name and a short description, cut off at about 170 characters. It reads a skill's full page only when it calls a tool to open it.

Moving an instruction from the system prompt (the standing instructions sent with every request) into a skill keeps that prompt short. But then the model, seeing only one line, decides whether to read more. So which part does the work: the line, or the page?

The short answer

• For diagrams and math, the one-line description did most of the work. The model usually did not open the skill.
• Most graphs and chemical drawings came after the model opened the skill, so its full page mattered too.
• Our first descriptions missed up to 1 in 5 drawings (20.5 in 100).
• Rewriting them and the pages won most back, not all.
• If the description misses a request, the page goes unread. So name the requests it should catch, in their words.
• Test each change on questions that should not trigger it. New wording can also push out other output.

scroll sideways for the rest

Does one line matter at all?

First we checked that one line in the system prompt matters. The day before the skill tests, we removed the line that asks for math in LaTeX, the code most math typesetting uses. We asked 12 math questions and 6 control questions that need no math, three times each, on both models. We did this with tools on (agent mode) and off (chat mode).

Without the line, the smaller model wrote 16 of 36 math answers in plain symbols such as ×, ² and ≈ with tools on, and 13 of 36 with tools off. The large model did so on 8 and 5 of 36. With the line, all four fell to 0 of 36. That is a drop of 44.4 and 22.2 points with tools on, and 36.1 and 13.9 points with tools off (ranges in the appendix). Chemistry markup went from 0 of 6 answers without the line to 6 of 6 with it, in every setting.

This test was small and ran one version after the other. Its ranges also count repeats of one question as separate questions, so they look more certain than they are. Still, one line changed over a third of the smaller model's math answers. Could that line live in a skill list instead?

The test

We asked whether four skills, each listed with its description, could replace the inline instructions for four kinds of output. The bar was to lose no more than 5 points on either model.

The four kinds are typeset math (LaTeX), diagrams, graphs of functions and numeric data, and drawings of molecules. We compared two versions of the agent's setup:

• The inline version (A) kept the instructions as shipped. It had one line for diagrams, one for graphs, and one for math that also covers molecules. Its skill list named only the graphing skill.
• The skill version (S) removed those lines and listed four skills instead: math, diagrams, graphs and molecules. Each description was cut to 170 characters, as the agent's list cuts every entry.

Each kind had 12 test questions that call for it, plus control questions that need none of it.

data (csv)

The diagram and graph questions describe something that reads faster as a picture. Examples are a call sequence, a pipeline, spend broken down by provider, or numbers along a numeric axis. Nine of the twelve graph questions never say graph, plot, chart, draw or visualise.

We set the pass rule before collecting any data. A skill passes only if two things hold on both models. Its worst plausible loss must stay under 5 points. And its output must show up in no more than 5% of control answers.

How the test ran

Each trial is one question asked once. It replays a real first request from a coding agent, with its full system prompt and tool list. It also carries a real user's list of about 100 skills and the product's model settings. Only the last user message changes, to the test question. The two versions differ only in the instruction lines and the skill list.

If every tool call in a reply opens one of the four skills, a stand-in returns that skill's page. The trial then goes on for up to two more replies. Any other tool call ends the trial. A trial that ends on a tool call, with no figure, leaves no answer to score. So we count two ways: over answered trials only, and over all trials, with unanswered ones counted as misses. The pass rule uses the worse of the two.

We ran each question 10 to 13 times, then topped both versions up once, to 156 to 1,008 trials per version. Control questions ran 18 to 42 trials per version. We wrote down the question, measures, pass rule and sizing rule before making any paid call. After the first run we changed one thing: we added the all-trials count and judged on the worse view. The second run used only fresh trials.

Each result comes with two 90% ranges, from two methods. A range is the span the true difference plausibly falls in. The appendix covers both methods, the sizing rule and the order of the runs.

Results with the first descriptions

data (csv)

No kind of output passed.

The large model's diagram loss came from two questions with numbers by category: spend by provider, and test counts per package. The inline version charted them on 34 of 37 trials. The skill version charted them on 0 of 37, and gave a markdown table every time. Its description said to draw when "a category comparison reads faster as a picture". The model did not see a table of spend by provider as one.

The graph losses came from one kind of question on both models: numbers along a numeric axis. An effort-level curve went from 15 of 15 to 1 of 14 on the large model. On the smaller model it went from 31 of 31 to 0 of 31.

The smaller model gained on diagrams. Its skill version drew sequence diagrams, which show messages passing between parties step by step, where its inline version mostly drew nothing. For how a secure web connection opens (a TLS handshake), diagrams went from 2 of 84 to 83 of 84. For a sign-in through another site (an OAuth login flow), they went from 27 of 84 to 79 of 84.

The large model's molecule loss came from the skill page. The page advised taking a molecule's 3D layout from a public database. That sent the model to fetch on 10 trials, which ended them before any drawing. Over answered trials, which leave those 10 out, there was no loss.

Results after the rewrite

We rewrote three descriptions to name the kinds of request the models had missed:

• The diagram description gained "numbers by category or share (bar, pie; even beside a table)" and "dependent steps".
• The graph description became "any function, equation (even simple) or numbers on a numeric x (time, size), table or not". It also told the model it needs no tool first.
• The math description added "worked calculation" and named the plain symbols to avoid.
• The molecule description did not change.

The same change also revised the skill pages, with advice aimed at the same misses. For example, the graph page now says never to run a tool just to compute values the graph can compute. The appendix lists every page change. Then we re-ran the whole test on fresh trials. So every change between the two runs mixes the new descriptions with the new pages.

data (csv)

The two runs are separate samples a few hours apart. The inline version's results moved between them too, so only the differences compare across runs. Table 4 gives the second run in full.

data (csv)

Under the pass rule as written, math and molecules pass on both models. Math's pass over all trials rests on a flaw in one range method, explained in the appendix. Diagrams and graphs still fail, in three places.

The large model fails on diagrams for two reasons. First, it drew a diagram on 3 of 42 control trials (7.1%), and the true rate could be as high as 19%. All three came on one question asking what an idempotency key is, a tag that stops a request from being carried out twice. Each was a sequence diagram drawn without opening the skill: the new description fired by itself. Second, its cautious bound over all trials is −6.5, just past the −5 limit. On the two category questions it now charts 20 of 22 trials, against 26 of 26 for the inline version and 0 of 37 in the first run.

Graphs fail on both models. The smaller model still misses the same two questions. On the effort-level curve, the inline version drew 20 of 20 answered trials and the skill version 0 of 21. On response-time percentiles sampled every ten minutes, the split was 8 of 21 against 0 of 21. The skill version answered all 42 with a table and never opened the skill.

The large model passes over answered trials but fails over all trials, where its range is too wide (low end −10.8 or −11.5, by method). About a third of its graph trials ended on a tool call in both versions (54 and 57 of 156). Most were a shell command to work out the numbers first, and our sizing rule did not allow for that. It now draws the effort curve on 11 of 11 answered trials, up from 1 of 14. But all eleven are diagram-skill charts, not graphs from the graphing skill, and none came after opening it.

Where the behaviour comes from

Three clues point to the description. The one exception is the large model's graph gain.

The first clue is how often the skill version drew without opening anything (Table 5).

data (csv)

For diagrams and math, most hits came from trials where the model never saw the page.

A follow-up math test changed only one thing: the math line in the prompt against the math skill's entry in the list. It mixed the versions in random order, with 372 math trials and 78 control trials per version and model. The skill version passed on both models. Its cautious bound was −0.8 points over answered trials, and −3.7 (large) and −3.4 (smaller) over all trials. No version put math on a control question (0 of 78 each, so the true rate is at most 4.7%).

The large model never opened the skill, and typeset all 340 answers it gave. The smaller one opened it on 13 of 372 trials (3.5%). No version answered a math question in plain symbols (0 of 372). The list entry did the job of the line it replaced.

Graphs and molecules work differently: the models usually opened the page before drawing. For the smaller model, 156 of its 163 graphs came after opening the graphing skill. For the large model, 69 of its 99 did. The page changed between the runs too, so the large model's graph gain is not the description's alone.

The second clue is where the failures sit. The trials that failed were ones where the model had only the description to go on. The smaller model's 42 missed graphs in the second run opened nothing. The large model's three unwanted diagrams came without opening the skill. In the first run, the large model answered both category questions with tables. The description was its only cue to draw a chart, and it did not match.

The third clue is what changed. The rewrite targeted the exact requests that had failed: categories, shares and tables for diagrams, and numbers on a numeric axis for graphs. Those results moved, apart from the smaller model's graphs. On the large model, the two category questions went from 0 of 37 to 20 of 22, eighteen of them without opening a skill. The effort curve went from 1 of 14 to 11 of 11. All eleven were diagram-skill charts, and nine came without opening any skill. The pages changed too, but a page can only act when the model opens it. For diagrams and math, that happened on a minority of trials.

The molecule skill shows how the work splits. A structure drawing needs exact syntax, so the models opened the page on 77% to 92% of molecule trials. What the page said then changed what they did. The first page's database advice cost the large model 10 trials, and narrowing that advice removed the loss. The description decides whether the model produces the output. The page shapes how, and only once it is opened.

Lines have side effects

An instruction that triggers one thing can suppress another. The molecule instruction is the clearest case we measured.

This test ran in chat mode, with no tools and no skill to open. We compared the shipped instructions (M) with the same instructions minus the molecule line (A). That line is the molecule skill's description, written into the system prompt. We asked 12 molecule questions, 20 times each: 240 trials per version and model, in mixed order.

data (csv)

The line cut both LaTeX and the written formula. Plain-text formulas did not rise to fill the gap: −1.7 points on the large model and +0.4 on the smaller one. Every reply with the line included a drawing of a molecule the question asked about. So the chemistry was still there, as a drawing instead of a written formula. On some questions the swap was total. The large model wrote nicotine's formula in text on 18 of 20 trials without the line, and on 0 of 20 with it. Whether a drawing may stand in for a stated formula is a product choice that this test cannot make.

It happened elsewhere too. The rewritten diagram description drew a diagram on a question that only wanted a definition. In an earlier chat-mode run, we used each skill's description as the chat instruction. There, the first math description ("any formula, unit or equation" in LaTeX) put formulas in 5 of 18 large-model control answers, all on the six about prices. The shipped line put none. That run kept no per-trial records, so this comes from its written analysis and cannot be re-checked.

What this means for you

If you move instructions into skills, treat the description as the instruction. On most trials it was the only part the model saw. If a description does not match the request, the model never reads the page.

• Name the requests that should trigger it, in the words those requests use. "Category comparison" did not match spend by provider. "Numbers by category or share", with "even beside a table", did. The models matched the request against the description's words. They did not stretch a general phrase to cover a specific case.
• Put the trigger first. The list cuts every entry at 170 characters.
• Test each change on questions that should not trigger it. Widening the diagram trigger won back the large model's loss. It also fired on a question that wanted a sentence. Our 18 to 42 control trials per version were too few to pin that rate down.
• Check what a new trigger pushes out. A line that asks for a drawn molecule can stop the model writing the formula. If you need both, ask for both.
• Where the output needs exact syntax, expect the model to open the page. Then everything on it, side advice included, changes what the model does.

What we could not settle

1. The smaller model's graphs. After the rewrite, it still answered numeric-axis questions with tables. The next test is a description that names such data more directly.
2. The large model's unwanted diagrams. All three came from one question, and the true rate could be as high as 19%. Pinning it down needs a few hundred control trials.
3. Description against page. The rewrite changed both. A run that changes only the description, with the page held fixed, would separate them. Only the follow-up math test was built that way.
4. Sizing when many trials go unanswered. About a third of the large model's graph trials ended on a tool call. The sizing rule ignored that, so the all-trials range could not clear −5 at the size we ran.
5. A molecule description that keeps the formula. In a follow-up, a description that also asks for "the formula or equation in the text" kept every drawing. It also lifted written chemistry above the no-line level on both models. It passed the control check only under a rule we re-registered, by so small a margin that the other range method would have failed it.
6. Other models. Both models come from one family. We do not know whether models from other vendors weigh list entries the same way.

Limitations

Each of these is spelled out in full in the appendix.

• Every trial replays one real request, with one user's list of about 100 skills.
• Any tool call other than opening a skill ends the trial, so some figures go unseen.
• Both range methods have known flaws.
• The rendering checks accept some wrong output and reject some fine output.
• Some questions were written for testing rather than taken from real chats.
• In the second run, the inline version listed the graphing skill with its older description.
• The first skill run tested one version after the other, so time of day may explain part of its differences.
• Some runs kept no per-trial records, so their figures cannot be re-checked.
• The math-line test was small and graded by simple rules.
• Control questions were few, so low rates of unwanted output are not pinned down.

Method appendix

Sizing

Stage 1 ran 10 repeats of each diagram and graph question, which is 120 trials per kind, version and model. It ran 13 repeats of each math question and 10 of each molecule question. From the inline version's rate alone, never from the difference, we set the final size per version, between 150 and 1,000 trials. Both versions were then topped up to that size once, with more repeats of the same questions. Control questions were not topped up.

From stage 1, arm A's rate $\hat p$ for each kind and model sets the final trials per arm:

$$n = \min\left(1000,\ \max\left(150,\ \left\lceil \frac{2\hat p(1-\hat p)\,(1.645 + 0.842)^2}{0.05^2} \right\rceil\right)\right)$$

Here $\hat p$ is a hit rate, and the denominator is the 5-point margin as a proportion, squared. The rule gives 80% power for non-inferiority at that margin when the true difference is zero. It uses the answered-trials rate, which is why kinds with many unanswered trials ended up too small for the all-trials view.

What the rewrite changed on the pages

The diagram and graph pages now say to chart such numbers "even when a table carries the exact values". The graph page says never to run a tool just to compute values the graph can compute. The math page covers "every line of a worked calculation". The molecule page says drawing "needs no tool call". Its lookup advice was narrowed to "only when the user asks for a verified" structure. All four pages had also been trimmed before the rewrite.

Run order

The first skill run ran each arm as a block, one after the other, so time is confounded with arm. The second run started both arms as parallel processes in the same second, and the skill arm finished up to 340 seconds behind. Only the two follow-up runs interleaved arms within one plan: the follow-up math test and the molecule side-effect test. Each ran one block per repeat and question, in a seeded random order.

Intervals

Every difference gets two 90% two-sided intervals, so each lower end is a one-sided 95% bound. The pass rule's "worst plausible loss" is that lower end.

The Newcombe hybrid score interval, the per-trial range in the tables and Figure 1, treats trials as independent. It combines the two arms' Wilson intervals at $z = 1.645$. The prompt-cluster bootstrap, the per-question range, resamples whole questions instead. It draws the kind's questions with replacement and sums each drawn question's hits and trials in both arms. It then computes the pooled difference, and repeats 5,000 times with a fixed seed. The 5th and 95th percentiles bound it.

Repeats of one question are correlated, so Newcombe runs narrow, and the bootstrap decides when the two disagree on the gate. Where they agree, we quote the lower bound of the two. The control gate reads the skill arm's false-positive rate as a point estimate. Upper bounds quoted on control rates are Wilson 95% upper bounds.

The math-line run

Here are the math-line run's differences, line on minus line off, with 95% Newcombe intervals. In agent mode they were −44.4 points (−60.4 to −26.7) for the smaller model and −22.2 (−38.1 to −8.0) for the large model. In chat mode they were −36.1 (−52.4 to −19.4) and −13.9 (−28.7 to −1.5). These intervals treat repeats of one question as independent.

Limitations in full

• Every trial replays one captured request, with one user's skill list of about 100 entries. The four skills compete with that list, not a clean install's. The captured system text names the large model, so the smaller model runs under a prompt that names the other one.
• The stand-in answers only skill openings. Any other tool call ends the trial, so a figure the model would have drawn after a shell command or a fetch goes unseen. That is why we report two views and gate on the worse one.
• Newcombe treats correlated repeats as independent, so it runs narrow. The cluster bootstrap has only 12 questions per kind to resample, and it can collapse to zero width (see below).
• Rendering checks use the product's own renderers and validators, which accept some wrong input and reject some reasonable input. In the first run, the graph validator rejected one option spelling that a later version accepts. Counting arm A's misses of that kind as hits, the first run's graph losses widen to about −21 points on both models.
• Nine graph questions came from a data-heavy suite written for graphing, which is why so many graph trials ran a tool first. Six molecule questions were written for this run rather than taken from logged chats.
• In the second run, arm A listed the graphing skill with its older description, frozen for the first run.
• The first skill run ran each arm as a block, so time is confounded with arm (see Run order).
• The first skill run's and the chat-mode run's per-trial rows were not kept, so their figures come from the written analyses and cannot be re-checked. The second run, the follow-up math test and the molecule side-effect test kept their rows. Re-analysing them reproduces every number we quote from them.
• The math-line run used three repeats per question (36 math trials per cell), ran one arm after the other, and graded with heuristics. We read every currency, false-positive and unrendered-math flag by hand.
• Control questions were few: 18 to 42 per arm in the skill runs. So a 0% false-positive rate still has a Wilson 95% upper bound of 8% to 18%.

The degenerate cluster bootstrap

The cluster bootstrap fails silently when every question's difference between arms is the same number. Every resample then returns that number, and the interval has zero width. It measured no spread between questions, so it reports no uncertainty.

That happened to LaTeX in the second run, over all trials. On both models, the only misses were the 13 trials of one compound-interest question. Each ended on a tool call before any answer, in both arms. The other 11 questions hit 13 of 13 in both. Every per-question difference was exactly 0, so the cluster interval was 0.0 to 0.0. Newcombe on the same counts (143 of 156 in each arm) gave −5.3 to +5.3.

data (csv)

The registered rule let the bootstrap decide, so LaTeX passed and was set to be adopted. We then added a rule: when the bootstrap is degenerate, Newcombe decides. Under it the all-trials view fails on both models, and the decision flips to not adopt. That failure reflects sample size, since the arms were identical, and 156 trials cannot put the lower bound above −5. It is why we ran the follow-up math test, interleaved, with 372 trials per arm, which passed under the stricter rule. The first run had the same pattern on the large model and passed it the same way. A degenerate bootstrap is easy to miss because its interval looks like the best possible result.

Runs

data (csv)

@techreport{ezolabs2026skills,
  title       = {Your agent mostly sees one line of each skill. Write that line well.},
  author      = {{EZO Labs Research}},
  institution = {EZO Labs Inc},
  type        = {Research report},
  number      = {EZO-R-2026-05},
  year        = {2026},
  month       = oct,
  url         = {https://ezolabs.io/research/skill-descriptions}
}