What is Agent Experience (AX)? Measuring AI agent usability

More and more developers meet a new SDK or API through an AI agent first. If the agent picks a competitor, or calls an API that has been retired, the technology is seen as “broken”, however readable its documentation is for humans.

Waldek Mastykarz, Principal Developer Advocate at Microsoft, has published an article that frames this problem as “Agent Experience (AX)”. This post walks through the definition, the way to measure it, and the measured results, then asks what people who use Claude Code or GitHub Copilot can take from it.

Definition and origin of AX#

The term comes from Introducing AX, written by Mathias Biilmann of Netlify in January 2025. He defines AX as “the holistic experience AI agents will have as the user of a product or platform”, proposed as a successor to user experience (UX) and developer experience (DX).

Biilmann’s questions are concrete.

• Can an agent easily get permission to act on a platform on the user’s behalf?
• Does the platform have organized, well-described APIs the agent can operate?
• Are there machine-oriented docs and context that let LLMs and agents use it correctly?

The same article uses Netlify’s integration for ChatGPT’s GPT store as an example: sites created on Netlify from ChatGPT exceeded 1,000 per day. That figure was Netlify’s own claim at the time, not a current number.

Microsoft’s article builds on this definition and puts it this way: practicing AX means improving that experience systematically, and measurement is how you know whether it improved. The scope is not limited to coding agents in an editor. It includes things like ChatGPT calling an MCP server.

What to measure: being chosen, and being used correctly#

Microsoft’s article reduces AX measurement to two questions.

1. Propensity (how likely you are to be chosen): Does the agent find and pick the technology? Give it a task that names no product, such as “add email sending to my app”, and see what it picks.
2. Efficacy (whether it is used correctly): Once chosen, does the agent use it correctly? Give it a task that names the product, and see whether it calls the current API or a retired one.

Both are scored on the quality of the result and on the cost it took. Cost is the item that is easiest to overlook.

In another article, the same author compares Claude Sonnet 4.6 and Sonnet 5 in GitHub Copilot Chat on VS Code (Windows). Sonnet 5’s token prices are 33% lower across the board: input drops from $3.00 to $2.00 per million tokens, and output from $15.00 to $10.00. Even so, on three scenarios that update a SharePoint Framework (SPFx) project (15 runs per model), the cost per run rose 3.7 times, from $0.55 to $2.01, because token consumption grew 10 times.

On a design task (12 scenarios, 60 runs per model), by contrast, Sonnet 5’s average cost was $0.47, 12% below Sonnet 4.6’s $0.54. The direction depends on the task. The article’s conclusion is to measure on your own tasks rather than rely on unit prices.

Quality is not simple either. On the design task, the Idiomatic metric, which checks whether the output follows conventions, was 90% for Sonnet 4.6 and 78% for Sonnet 5. On the SPFx update, the opposite held: the share of runs that passed the first gate, which checks whether the instructions were followed, was 60% for Sonnet 4.6 and 100% for Sonnet 5. The data say you cannot assume that newer and cheaper means better.

Where you can intervene, and where you cannot#

The article on the AX stack splits the layers between a developer’s instruction and the resulting code into the model, the harness (agent apps such as Copilot and Claude Code), agent extensions, and the technology’s surface (CLI, SDK, API).

• Model: Training data cannot be changed. A model trained on old documentation will emit retired idioms. If a competitor has more training data, the model leans toward it.
• Harness: This is the vendor’s domain and is hard to see from the provider’s side. The same MCP server can give different results in different harnesses.
• Everything beyond that: Documentation; extensions such as skills, MCP servers, instruction files, and custom agents; APIs, SDKs, and CLIs; and error messages are all yours to change.

Documentation is what to fix before extensions. A trained model does not know about doc updates until the next model ships. An agent that reads the docs during its work picks up the update on its next run. The same holds for interfaces. The article gives an example: a scaffolder with no pinned version created a project with a July 2020 version, and the agent took that as a success. Agents trust what comes back, so they report success even when the result contains errors.

Agent-friendly-looking fixes are hypotheses until measured#

The point Microsoft’s article stresses most is that a list of best practices is only a starting point; whether a fix works is unknown until you measure. The table below summarizes results from different experiments. It mixes measurements from the SPFx update with those from a synthetic CLI.

Fix	Measured result (experiment)
Change the CLI’s arguments to JSON input	With plain arguments, every model profile was correct in all 5 runs. With JSON input, Claude Haiku 4.5 was correct in only 2 of 5, and every model’s cost per task rose 4 to 11 times (multi-service deployment with the synthetic CLI podctl)
Add a note in the docs pointing to the right tool	Even a direct pointer was used in only 1 of 5 runs, but a warning that named the approach that fails was used in 5 of 5 (SPFx update)
Add another doc source (the context7 MCP server)	No noticeable improvement. In 3 of 5 runs the tool was not loaded, and in the other 2 it was not called (SPFx update)
Switch to a model with a lower token price	As in the previous section, cost rose 3.7 times (SPFx code update)

The proposal to switch CLI arguments to JSON input rests on a plausible hypothesis: agents are used to handling structured data. The article that measured it used the synthetic CLI podctl and had agents run a nested multi-service deployment with more than 30 values, once with plain arguments and once with JSON. The results are in the table above; the rewrite made things worse.

What worked for the SPFx update was not adding a skill. On the update task, the agent at first passed only 30 of 80 configuration checks (Claude Sonnet 4.6, 5 runs). Telling it to use the CLI for Microsoft 365 raised that to 75, but this only helps people who know the CLI or have installed the skill. Looking at the agent’s steps, it fetched the SPFx release notes during the work. So the author added a sentence to the release notes warning that manual updates tend to fail, and rewrote the migration guide to explain the changes and point to the CLI. After that, the CLI was used every time without any instruction in the prompt. It also works for developers who have not installed the skill.

Starting in your own environment#

This approach is useful even for developers who have no product documentation of their own. For an agent, your repository or an internal library is also a “technology surface”. From here on, I apply the article’s content to the position of a team using Claude Code or GitHub Copilot; that part is my interpretation.

Microsoft’s article suggests five steps.

1. Write a few tasks that developers actually hand to agents, in the words they use. Mix ones that do not name the product (propensity) with ones that do (efficacy).
2. Run each several times in a bare baseline, with no extensions and the same settings. Look at where the agent went wrong and what it read along the way.
3. Fix the cause. Start with the docs and interfaces the agent already uses, and add new extensions last, because extensions only help developers who install them.
4. Run the same tasks again, compare quality and cost per task, and decide whether it was a lift or a drag (worse or no change).
5. Treat the conclusion as valid only within the agent configuration you measured (OS, harness, model, extensions). Measure again when the configuration changes.

If you already distribute skills or MCP servers, the article says to start at step 2 and compare with and without them.

Before adding an instruction file such as CLAUDE.md or AGENTS.md, look at what the agent actually reads. That order is the main point of the article’s measurements. Added explanations do not help if they are not read, while fixing the release notes and migration guide the agent was already reading got the CLI chosen with no extra instruction file.

Limits and how to read this#

Microsoft’s article itself says the results hold within the scenarios and agent configurations measured and may differ elsewhere. The numbers come mainly from one technology, SPFx, and from runs in GitHub Copilot Chat on Windows, with small samples of around 5 runs per condition. Reading “JSON input is bad” as a general rule goes too far; it is better read as an example of how something added without measuring can miss.

The author also evaluates AX for Microsoft products, and the technology discussed is Microsoft’s own. This article does not tell us whether the same pattern shows up for non-Microsoft technology.

The term AX itself is still new, and the vocabulary (bare baseline, lift and drag, propensity and efficacy) is used within Microsoft’s series. These are not definitions that have settled across the industry, which is worth keeping in mind.