Mintlify Found Agents Ignored Its CLI Guide Until AGENTS.md Named It
Mintlify's Nov. 10 benchmark found coding agents skipped mint instructions until one AGENTS.md line pointed to it. Then Claude Sonnet passed every run.

Coding agents ignored Mintlify’s new mint instructions command until a line in a repository’s root AGENTS.md told them to run it, the company said in a Nov. 10 engineering post. With the command listed only in mint --help, Claude Sonnet passed 16% of Mintlify’s test runs, compared with 20% without the command at all. After the AGENTS.md pointer and a rewrite of the guidance, Sonnet passed 100%.
Agents Read AGENTS.md Before Help Text
With one pointer in AGENTS.md, agents ran mint instructions before their first edit in 30 of 30 tests, Mintlify said. In the first version, the company assumed agents facing an unfamiliar task would check the CLI’s help output. Instead they read the task, looked around the repository and started editing.
The wording of the pointer changed the score. A generic line, “run mint instructions before Mintlify tasks,” produced a 66% pass rate, because agents treated a task like “set up an automation” as a GitHub Actions job and never checked. Naming the task areas (automations and CI, access and publishing, AI features) raised it to 83%. Rewriting the guidance around what agents still got wrong took it to 100%.
A Pointer to the CLI Beat a Pointer to the Docs on Every Model
Mintlify ran the same AGENTS.md wording against two targets, its Markdown docs and the CLI command. Pass rates, per the post:
| Model | No guidance | Pointer to docs | Pointer to mint instructions |
|---|---|---|---|
| Claude Sonnet | 20% | 57% | 100% |
| Claude Opus | 28% | 72% | 94% |
| Claude Haiku | 6% | 28% | 89% |
| Codex | 36% | 81% | 89% |
Mintlify called the 8-point gap on Codex small enough to be noise. Haiku rose from 39% to 89% after the company moved the facts agents most often missed into an overview topic marked “Start here.” With the command, agents made 0.02 false claims about Mintlify per run, down from 0.69 without it.
The Command Serves Docs Pages and Updates Without a Release
mint instructions lists 10 topics and 10 subtopics, and mint instructions <topic> prints step-by-step guidance for one task, such as mint instructions publish domain, the post said. Each topic is a page in Mintlify’s docs, so an edit to the page reaches every agent on its next run with no CLI release or reinstall. The command is linked from every docs page’s Markdown, llms.txt, skill.md and mint --help, and the starter repository’s AGENTS.md now carries the pointer. Mintlify first listed the command in its Oct. 30 changelog.
Key Takeaways
- Listed only in
mint --help, the command did nothing: Sonnet passed 16% of runs with it and 20% without. - One AGENTS.md line got agents to run it first in 30 of 30 tests.
- Naming specific tasks in that line mattered: 66% generic, 83% specific, 100% after a guidance rewrite.
- Task-scoped CLI guidance beat a pointer to the full docs on all four models tested.
The Take
We think the finding that matters is where agents look. An agent reads AGENTS.md before it starts, and it acts on what it finds there. It does not go browsing help output on its own. So guidance that lives anywhere else needs a signpost in that file, and the signpost has to name the kinds of task it covers, because an agent won’t connect “set up an automation” to a docs tool by itself.
This is the vendor’s own benchmark on its own product, so we’d read the exact percentages loosely. The direction is consistent across four models, though. We’d add a short, task-named pointer to the root AGENTS.md of any docs repository agents work in, then give an agent one task in that repository and watch whether it follows the pointer before its first edit.


