Surge AI’s Dayjob: Tax benchmark landed on Hacker News on 2026-10-28 and sat at the top of the page for a day (247 points, 154 comments ). The number that stuck: Claude Sonnet 5.5, the best of the three frontier models tested, passed 10.8% of 50 real professional tax assignments. Fable 5.1 scored 9.6%, Opus 5.5 scored 9.2%. The grading wasn’t a toy rubric either: licensed tax experts built the cases and scored each attempt against criteria requiring a 95% match.
The interesting part isn’t the headline number. It’s why the models failed. I built a small repo, themsquared/agent-verify-gate , that reproduces the shape of that failure mode with a toy agent and shows the one trajectory check that would catch it, scored from a recorded trace with no LLM judge and no extra agent run.
Disclosure: I work at Solo.io, which builds agentgateway and kagent. agentevals-dev, used below, is a separate open-source project that several Solo engineers contribute to.
What Dayjob: Tax actually tested
Surge AI’s benchmark gave agents real client requests with supporting records and asked them to own the assignment end to end, not answer a single defined question. The scale: 50 environments across eight practice areas (individual and family tax, small-business tax, partnerships, property and investments, estates and trusts, international tax, corporate transactions, and tax review and disputes), backed by 1,789 workspace files, a median of 30 per assignment and a max of 123. Most of those files are PDFs, with Excel workbooks, Word documents, emails and images mixed in.
Three tax professionals with backgrounds at PwC, Deloitte, Exxon, Baker Tilly, Moss Adams, and EY built the cases and ran every attempt through a three-layer expert review for correctness, completeness, and solvability. An attempt passes only if it satisfies 95% of the case’s criteria, covering calculations, supporting evidence, consequential issues, and the final deliverable.
That’s a materially harder bar than most agent benchmarks: it isn’t “did the model produce a plausible-looking answer,” it’s “would a reviewing tax professional sign off on this.”
Where the models actually failed
Surge AI’s own case studies point at a specific pattern, not generic incompetence:
- In a condo-sale case, one model missed a filing-eligibility deadline in all five attempts, while a different model caught it in all five.
- In a divorce case, a model based its recommendation on an assumption about the client’s marital status in all five attempts, instead of verifying it against the evidence files in the workspace.
Neither failure was an arithmetic mistake. Both share the same shape: the agent had a fact it could have checked and didn’t. It answered from a general rule instead of the specific case in front of it. That’s not a knowledge gap you fix with a bigger model or a better prompt. It’s a missing verification step, and it is exactly the kind of thing you can check mechanically from what the agent actually did, without re-deriving the tax law yourself.
A minimal trajectory check for the same failure class
I built a toy version of this pattern to show the mechanism. A fictional “tax advisor agent” gets asked whether a client can still elect out of installment-method reporting on a property sale, and by what deadline. The correct answer depends on a fact the agent has to check: whether the client’s return extension was filed. Get that wrong and you give the unextended, earlier deadline when the client actually has months more.
agentevals
scores agent behavior from OpenTelemetry traces: point it at recorded or live telemetry, and its deterministic evaluators (tool-trajectory matching, response matching) or LLM judges score what the agent actually did, no reruns required. I recorded two versions of the same question as traces and scored both against one eval case specifying check_filing_deadline as the required tool call before the final answer.
python3.13 -m venv venv
source venv/bin/activate
pip install agentevals-cli
The trace that verified first:
agentevals run traces/pass_verified_deadline.json \
--eval-set eval_sets/eval_set_deadline_check.json \
-m tool_trajectory_avg_score
[PASS] tool_trajectory_avg_score 1 PASSED 1 12ms
The trace that skipped verification and answered from the general rule:
agentevals run traces/fail_unverified_deadline.json \
--eval-set eval_sets/eval_set_deadline_check.json \
-m tool_trajectory_avg_score
[FAIL] tool_trajectory_avg_score 0 FAILED 0 11ms
Invocation 1 trajectory mismatch:
Expected:
- check_filing_deadline({"client_id": "demo-client-01", "tax_year": 2025})
Actual:
(none)
That’s the whole mechanism. No LLM judge had to know anything about installment-sale elections. The check only needs to know that this class of question requires this tool call before the answer is trustworthy, and it reads that straight from the trace agentevals already ingests.
The gotcha: agentevals-cli silently has no wheel for Python 3.9
Building this, pip install agentevals-cli failed with ERROR: Could not find a version that satisfies the requirement agentevals-cli (from versions: none) on this machine’s default python3. The error gives no reason. The actual cause: agentevals-cli’s PyPI metadata sets Requires-Python: >=3.11, and the default python3 here is 3.9.6. pip’s resolver silently drops every candidate instead of saying so plainly. The fix is a newer interpreter:
/opt/homebrew/bin/python3.13 -m venv venv
source venv/bin/activate
pip install agentevals-cli
If you hit the same bare “no versions” error on any package with a python_requires floor, check python3 --version before anything else.
What this does and doesn’t prove
This repo is a minimal, original stand-in for the failure class Surge AI’s benchmark surfaced, not a reproduction of their tasks or grading. Their rubric covers calculations, supporting evidence, consequential issues, and the deliverable as a whole, reviewed by tax professionals. A tool-trajectory check only tells you whether one specific, named verification step happened before the answer. That’s a narrower, cheaper, and more auditable signal, and it’s one you can run on every production invocation, not just on benchmark day.
The broader point: a 10.8% pass rate on real professional work is a finding about agent reliability, not a finding you only get to act on once. Whatever gate the agent would need to pass in production (a tool call, a format, a required citation) can be scored the same way, continuously, from telemetry the agent already produces.
Try it
git clone https://github.com/themsquared/agent-verify-gate.git
cd agent-verify-gate
python3.13 -m venv venv && source venv/bin/activate && pip install agentevals-cli
agentevals run traces/fail_unverified_deadline.json --eval-set eval_sets/eval_set_deadline_check.json -m tool_trajectory_avg_score
Full repo, traces, and eval set: themsquared/agent-verify-gate . For the OTel plumbing that gets agentevals scores back into your existing observability pipeline as standard events, see agentevals v0.10.0: Scores Come Back as OpenTelemetry Events . For scoring a real kagent agent’s trajectory the same way, see Regression Tests for kagent Agents with agentevals .