An AI agent that stops calling its tools does not throw an error. It answers from the model’s memory, confidently and plausibly, and the output still looks fine in the UI. I wanted a regression test that catches that, so I wrote kagent-agentevals : a converter from kagent session records into the trajectory format agentevals scores, plus a golden-suite runner that exits non-zero when an agent’s behavior changes.
The conversion is where the work turned out to be. agentevals wants a flat list of OpenAI-format chat messages. kagent records what its agents do as Google ADK events. Getting from one to the other is not a field rename, and three of the reasons only showed up when I ran it against real sessions instead of reading the schema.
What kagent actually records
kagent writes one row per ADK event, with the event JSON in event.data. A single event carrying a tool call looks like this:
{"author": "github_assistant",
"invocation_id": "e-c51ee408-...",
"partial": null,
"content": {"role": "model",
"parts": [{"function_call": {"id": "toolu_01...",
"name": "search_users",
"args": {"query": "themsquared"}}}]}}
That is the readable version. ADK serializes every optional field, so the actual row has around forty keys set to null around the four that matter. Dumping a session’s raw events to a terminal is unusable, which is why the first thing the tool grew was a view that accounts for each event and says why anything was dropped:
kagent-evals extract <session-id> --summary
# AUTHOR ROLE CONTRIBUTED WHY NOT
0 system no content
1 user user text "Who is themsquared?"
2 github_assistant model call ask_user
3 github_assistant user filtered: adk_request_confirmation
4 user user filtered: adk_request_confirmation
5 github_assistant user result ask_user
6 github_assistant model text "I don't currently have a tool to look …
7 github_assistant no content
8 system no content
9 user user text "Try again"
10 github_assistant model call search_users
11 github_assistant user result search_users
12 github_assistant model text "Here's what I found for **themsquared*…
13 github_assistant no content
14 events, 8 contributed → 8 messages, tools: ask_user, search_users
Fourteen rows, eight of which contribute anything. That is a real session: someone asked an agent about a GitHub user, the agent asked a clarifying question first, then retried and searched. Everything you would want to assert on is in there. So is a lot that you do not.
Why the role field lies about who called the tool
Look at events 3, 5 and 11 above. Their content.role is user, and every one of them carries the agent’s own function call or function result.
That is not a bug in kagent, it is how the human-in-the-loop confirmation round-trip is modeled: the approval request and its answer flow back through the user side of the conversation. But if you map content.role onto the OpenAI role field, the agent’s tool calls end up attributed to the person, and every comparison you write after that scores the wrong speaker. The trajectory looks plausible and is wrong.
The fix is to stop reading the role at all and decide from the part type:
if isinstance(response, dict): # function_response -> tool message
flush()
trajectory.append({"role": "tool", "content": ..., "name": name})
elif isinstance(call, dict): # function_call -> assistant tool_calls
buffer.tool_calls.append(entry)
elif isinstance(text, str) and text.strip():
if author == "user": # only plain text consults the author
trajectory.append({"role": "user", "content": text})
else:
buffer.text.append(text)
A function_call part is always an assistant action. A function_response part is always a tool message. Only plain text needs to ask who was speaking, and for that author is reliable in a way content.role is not.
Why adk_* tool calls have to come out
Events 3 and 4 in that table are adk_request_confirmation. The ADK runtime injects that tool to run the approval handshake. The agent never chose to call it.
Leave it in and it appears as a tool call in the trajectory, which means every reference trajectory you write has to include a piece of runtime plumbing that has nothing to do with the agent’s behavior. Worse, it appears conditionally, only on turns where a confirmation happened, so references written against one session break against the next.
So anything matching the adk_ prefix is dropped by default. The distinction is the prefix, not a judgment about the tool: ask_user is a real kagent tool the agent genuinely decided to call, and it stays in.
Why tool output arrives double-wrapped
kagent’s builtin tools return {"result": ...}. MCP tools return the richer MCP shape:
{"content": [{"type": "text", "text": "{\"total_count\":1,...}"}], "isError": false}
Hand either of those to an LLM-as-judge evaluator and the judge spends its attention on the envelope. Both get unwrapped, and the isError flag is preserved as a [tool error] prefix on the message content, so a failed tool call still reads as a failure rather than as data.
This one bit me in a way worth mentioning, because it is the kind of bug that ships. My first unwrap only handled single-key dictionaries. The MCP envelope has two keys, content and isError, so it sailed straight through unwrapped and I only noticed because the tool message in the output still had isError in it.
What the agentevals match modes actually compare
This is the part I would have gotten wrong from the documentation, and it changed how I wrote the suite. agentevals offers four trajectory match modes, and their names oversell what they inspect. I read the scorers to be sure:
subset,supersetandunorderedextract tool calls from both trajectories and compare only those. Roles, message order and every piece of text are ignored entirely.strictadds message count, roles, and per-message tool calls with argument matching. It still never compares assistant prose.
So none of the match modes grade what the agent actually said. That is not a criticism, it is the right design for a deterministic check, but it means two things in practice. First, the content fields in a strict reference trajectory are documentation for whoever reads the suite next, not assertions. Second, if you need the wording graded, you need a trajectory LLM-as-judge
evaluator, which is a different tool with a different cost profile.
Knowing that let me add a third evaluator type that asserts on tool names alone, with no reference trajectory to transcribe:
evaluators:
- type: tools_used
mode: superset
expected: [get-weather-by-city_get-weather-by-city]
It builds a synthetic reference containing exactly those tool calls and delegates to agentevals. Because it only ever supplies names, it forces argument matching off. I learned that the hard way: I had set that default in the YAML parser rather than in the evaluator, so the two unit tests that constructed the spec directly in Python failed against a session they should have passed. The invariant belonged in the code, not in a config convention.
One suite, three sessions
Here is what makes it a test rather than a demo. The assertions never change. Only the session does.
The suite says three things about a weather agent: the answer has to come from the weather tool, the agent stays inside its tool budget, and it looks up the city the user actually asked about. Against the real captured session:
kagent-evals run demo/suite.yaml --fixture demo/fixtures/weather-grounded.jsonl
answer-is-grounded-in-a-tool-call [session fixture]
4 messages, tools: get-weather-by-city_get-weather-by-city
PASS called-the-weather-tool trajectory_superset_match true
stayed-within-its-tool-budget [session fixture]
4 messages, tools: get-weather-by-city_get-weather-by-city
PASS no-unexpected-tools trajectory_subset_match true
looked-up-the-city-the-user-asked-about [session fixture]
4 messages, tools: get-weather-by-city_get-weather-by-city
PASS strict-with-args trajectory_strict_match true
3/3 cases passed
Now the same suite against a session where the agent skipped the tool and answered from memory:
answer-is-grounded-in-a-tool-call [session fixture]
2 messages, tools: (none)
FAIL called-the-weather-tool trajectory_superset_match false
stayed-within-its-tool-budget [session fixture]
2 messages, tools: (none)
PASS no-unexpected-tools trajectory_subset_match true
looked-up-the-city-the-user-asked-about [session fixture]
2 messages, tools: (none)
FAIL strict-with-args trajectory_strict_match false
1/3 cases passed
Caught, exit code 1, CI stops. The answer that agent produced was a fluent, specific, entirely invented weather report.
Two checks that pass when they should not
Notice the middle case still passed on the ungrounded session. That is worth sitting with, because it is the most useful thing the demo taught me.
A subset check asks whether the agent called anything outside the approved set. An agent that called nothing satisfies that trivially, since the empty set is a subset of everything. A tool allowlist cannot catch an agent that did no work. Catching that needs a superset check asserting the tool was called at least once. A suite that only allowlists is half a suite.
The second one has the same shape. Point the suite at a session where the agent did call the weather tool, but for Paris when the user asked about London, and only the strict case fails. Both tool-name checks pass, because they compare names and ignore arguments. If the arguments carry the meaning, and for a lookup tool they usually do, you have to say so explicitly.
Both of those are pinned by tests in the repo, so if the behavior ever changes the narration fails in CI rather than going stale on me in front of someone.
Check the data, not just the source
The most expensive mistake I nearly made had nothing to do with agentevals.
I started by reading a months-old checkout of kagent to work out the storage format, and that version serialized session events as A2A protocol.Message objects, with tool calls encoded as data parts carrying kagent_type metadata. I had most of a converter written against that shape before I queried the running cluster and found it storing raw ADK events instead. Different envelope, different field names, and a much closer fit to what agentevals wants.
Reading the source told me what some version once did. Only the database told me what the cluster in front of me was actually writing. For anything that consumes another system’s persisted records, the stored bytes are the contract, and the fixture in the repo is a real captured session precisely so the converter stays pinned to data the runtime produces rather than to my reading of it.
There is a related caveat I have written about before. The trajectory being scored is the runtime’s own record of what the agent did, and an agent’s self-reported record is not automatically trustworthy . These evals catch behavior drift in a record produced by the system under test. That is genuinely useful for regressions and it is not the same thing as a witness.
Quickstart
No cluster and no API key. The sessions are JSONL fixtures in the repo, which is also what CI runs on every push.
git clone https://github.com/themsquared/kagent-agentevals
cd kagent-agentevals
pip install -e .
make demo
make demo walks the whole story with pauses so you can talk over it. make demo-fast runs straight through.
Against a live cluster it is the same suite with one flag changed:
kagent-evals sessions --agent my-agent
kagent-evals extract <session-id> --summary
kagent-evals extract <session-id> --raw -o demo/fixtures/mine.jsonl
kagent-evals run demo/suite.yaml --fixture demo/fixtures/mine.jsonl
Capture a session once and it becomes a permanent regression test for that agent’s behavior. One thing to do before you commit a capture: read it. A raw session record can carry request metadata from whoever was chatting, so the CLI prints a reminder on every --raw, and there is a test that fails if a fixture in the repo contains a JWT.
What I would add next
The LLM-as-judge path is wired and unit-tested but I have not run it against a real model yet, so the repo says so rather than implying otherwise. Two evaluators there are worth having: reference-free grading of whether a trajectory is coherent and efficient, which the deterministic modes cannot express, and grading of the final answer’s wording, which no match mode touches.
The other direction is the boring, valuable one. A capture from every agent, a suite per agent, run nightly, with the tool budget asserted in both directions. That is a small amount of YAML and it is the difference between finding out an agent stopped grounding its answers from your eval suite versus finding out from a user.
Code, the demo, and the talk track are at github.com/themsquared/kagent-agentevals .