My MCP server changed its menu at the wrong time. Now it doesn't.

01What it does

Four commands, and no LLM anywhere. The checks are deterministic for the same inputs and environment, so they can sit in CI.

• session plays a scripted conversation (the scenario is the agent), lists the menu after every step, and reports each change with the step that caused it.
• snapshot checks a single menu, and starts a second copy of the server to see if it serves the same one.
• diff compares two releases the way API breaking-change checkers do: removed tools, new required inputs, and the token change for every conversation.
• history installs a package's published versions one by one and diffs them. It installs them the way you would today, so it records what those versions serve now, not what they shipped with. That turned out to matter.

02My own server wouldn't meet the new rule

The 2026-07-28 version of the MCP spec says the tool set must not vary "per-connection or as a side effect of other requests on the connection". The older protocol doesn't have this restriction, and most servers I tested still use it.

On my server, session showed every unlock adding tools at the end of the list, the way I'd designed it. It also caught what I hadn't: every unlock was local to that connection. A fresh connection still saw the old menu. I'd need to change that behaviour before adopting the new protocol. My own tool caught it.

In the mode that unlocks more at once, the second unlock added 75 tools in one step, about 21,000 tokens, most of the menu. Those 21,000 are only the new tools. The cache had to redo everything from the start of the prompt. In my setup, that meant 40–57K tokens of prefix had to be processed again.

It caught smaller things too. One tool's description ran past Claude Code's cut and was sliced mid-word. Another tool's schema repeated the same block five times; defining it once and pointing to it saved about 900 tokens in every conversation, and diff confirmed the tool still accepts the same input.

And my labels were wrong. I'd marked every tool as reaching outside the server, the label meant for tools that touch the open web or other people's systems. toolmenu doesn't call those on its own, so session --auto skipped 80 of my 82 read-only tools. They only read my own systems, and now they say so.

03The wrong time, on other people's servers

Mid-conversation isn't the only wrong time.

04Where it was wrong

The first run on real servers produced a lot of false positives. The rule for invented IDs went from 48 findings to 4 once I fixed what it misread.

The rule for instructions buried past the point where a client cuts a description did worse. On my own server it was right only about half the time. It couldn't tell "don't retry this" (an instruction) from "never throws" (a description), and it missed the exact case it was built from.

Where clients cut was my own assumption too. I'd set it to 280 characters because that's where my client cut descriptions. Most don't: Claude Code cuts at 2,048, and some clients send the whole thing. I've since removed the cap in mine. The version check had the same flaw: it trusted a number servers don't maintain. The filesystem server reports itself as 0.2.0 in every release.

The token counts were wrong, and by the most on the biggest menus. toolmenu counted the whole tool definition, including parts clients never send the model, like icons and output schemas. On GitHub's official server, that made the menu look five times its real size. One of my own findings, a patch release that doubled a menu, came from that bug.

diff, the release comparison, waved through breaking changes nested inside a tool's inputs, like a field inside another field becoming required.

The biggest one was about the cache itself. Both toolmenu and I treated a tool added at the end of the list as safe; it was how I'd designed my own server's unlocks. But in the client I was testing, the tool list comes first in the prompt, so changing it at all, even at the end, invalidates the cached prefix. What keeps the cache is new tools arriving inside the conversation instead, the way clients with tool search bring them in. That difference determines whether the cached prefix survives. I caught it days before the release.

Assumptions are just that: assumptions.

Each of those is fixed now. On my server, the buried-instruction rule went from about half right to 36 of 39. That's one server, not a benchmark. The cut is whatever your client actually does. The version check uses the release version, not the one the server reports. The token counts estimate only the parts sent to the model, and diff is tested by breaking real servers' schemas thousands of ways. The cache one came from a question, checked against the docs. The rest came from real data, not from thinking harder.

It also says when it's guessing now. A finding that comes from reading words, not from something it measured, is marked "unsure". Every finding and every error ends with what to do next; a server that logged to the wrong place used to get just "Request timed out". I also run the checks against public servers before changing the rules.

05Where menus are going

The hosted Atlassian and Sentry servers I checked don't change their menus at all. They keep a small fixed menu: a tool that searches a catalog of operations, and one that runs what it finds. This is called search and execute.

A fixed menu is exactly what the cache wants, and the wrong time goes away. But the changes don't: they move into the catalog. A stable menu can hide an API that keeps changing. Behind Sentry's 9 tools are more than 60 operations, and a breaking change to one of them may not show up in the menu. So toolmenu reads the catalog too. In a run in September it found all 154 of Atlassian's.

A fixed menu was the answer for my server too, so I changed the default. It now lists its whole menu from the start and never changes it, and session finds nothing. The unlocks are still there, switched off, for clients that load every tool up front and have no way to search or filter them.

06What it can't catch

It sees what the server sends. It can't see what the client does with the menu (sorting, filtering, merging several servers), a feature the agent reaches for that has no tools at all, or whether the agent actually picks the right tool. That last one needs evals with a model, and toolmenu stays deterministic on purpose.

07Try it

npx toolmenu init -- node dist/server.js
npx toolmenu session --init -- node dist/server.js

The first snapshots your menu and writes a GitHub workflow that comments on every pull request. The second writes a starter conversation from your menu, unlocks included, to run with session. Servers behind OAuth, tools with no read-only label and the rest are in the README. The package is on npm.

08Why I built it

I built it because it might help other people discover, quickly, what took me time to discover. And I kept one rule from building it: if a rule can't point to a failure someone actually hit, it doesn't ship.