// #AI Agents #Robotics #Evals #Thoughts // 7 min read

Eight Hours, Not One Task

Eight hours, not one task: Andra Keay says commercial robots need a full shift. Agent evals should too, with a queue, faults, and a soak score.

Commercial robots, and production agents, have to last a shift, not a demo. Andra Keay’s bar is eight hours of continual work, recovery from errors, and a whole work process, including getting to and from the task. A one-task eval from a clean container cannot see the failures that show up after hour six, so if you would be nervous leaving it alone over lunch, you have not tested it yet.

Every soak chamber I’ve worked around had a sticky note on the door. The one I remember best just said “LUNCH?” Somebody wanted to know if the run would survive the hour nobody was watching it.

That’s the real test. Not whether the unit works. Whether it still works after hour six, with nobody standing there.

What did Keay say commercial robots actually need?

Andra Keay runs Silicon Valley Robotics and writes Robots & Startups. The October 31 issue is a post-BARS reading list: 29 faculty talks at five minutes, 40 student talks at 45 seconds, 13 industry talks at two minutes, plus two short keynotes. She pulls papers from agentic world models to the Berkeley QUAD hand. Then she says the research frameworks make it hard to work on what commercial deployments actually need:

8 hour continual operation performing tasks with high accuracy, robustness to perturbation, error recovery, safety around humans (idiots) and most importantly, integration into a whole work process, not simply single or disconnected tasks.

And then the line I keep rereading: “Integrating into the workplace and getting to and from the task is part of the process and rarely included in the job metrics.”

She points at the DARPA Robotics Challenge in 2015 as the last time a benchmark really forced that, and at the new NIST Loco-Manipulation Challenge, launching at IEEE Humanoids 2026, as the attempt to pick it back up.

That is a customer-down spec, not a task-up spec. A grasp taxonomy, a locomotion run, a single pick, those are tasks. An eight-hour shift in a workplace is a process: walk to the station, do the work, recover when something moves, do not hurt the person who walked through the aisle, go to the next job without a researcher resetting the world.

The same issue is why the hard part is the hands and why duty cycle decides whether a robot pays. A capable demo that cannot hold a shift is a lab result. Capable isn’t cheap until the hours are real. Changeover time is the spec is the same argument on the other side of the aisle: the time between useful work is part of the work.

Why do single-task agent evals miss the same failures?

Swap “robot” for “agent” and the paragraph still holds.

Most agent evals I see, including the trajectory evals I wrote up for kagent, score one task from a clean start. Fresh container, fresh context, one prompt, one grade. That’s a useful regression test. It’s also the demo, not the shift.

What breaks in production rarely breaks on task one:

  • Leftover state. Task 40 inherits a temp file, a half-applied migration or a stale cache from task 39.
  • Recovery loops. A tool call fails, the agent retries, and nothing caps the retries. Every call passed and nothing got done is what that looks like when the call never even fails.
  • Time. Tokens expire, rate limits reset, the context window fills. None of that exists in a two minute run. The house agent rereading its rulebook is a time problem: the cache dies between wakes, and the next hour costs full price.
  • The walk to the task. Finding the right ticket, the right repo, the right credential. Keay’s “getting to and from the task.” Most evals hand the agent the task already staged.

One run is an anecdote is about repeating the same task. This is about stretching the task until it looks like a job. You can have a tight five-run median on a 90-second prompt and still know nothing about hour six.

If you monitor a humanoid like a service, you already know the pager does not care about the first pick. It cares about the fleet after lunch. The pager goes off and nobody wakes up is the human version of a soak that nobody scored.

What would a shift-length agent eval look like?

A shift-length eval, built like a soak test:

  1. A queue, not a prompt. Load a realistic mix of work, a few hours’ worth, and let the agent pull from it. No environment reset between tasks.
  2. Scheduled faults. Kill a tool server at hour two. Expire a token at hour three. Drop a malformed ticket in the queue. Robustness to perturbation, on a timetable.
  3. Score the run, not the task. Interventions per hour. Time to recover from each injected fault. Task quality in the last hour compared to the first.
  4. Count the overhead. Log the steps spent finding and setting up work separately from the work itself. If setup is half the run, that’s the finding.

None of this is exotic. It’s what a soak chamber does to hardware. We just haven’t been putting agents in the chamber.

The sticky note is the spec. If you’d be nervous leaving it alone over lunch, you haven’t tested it yet.

What soak test plan would I actually run?

If you already have a one-task suite, keep it. Then add a chamber run next to it.

  1. Define the shift. Wall clock, not token count. Eight hours is Keay’s number for robots. For an agent, even two or three hours without a reset is a different experiment than a two-minute prompt. Write the duration down before you like the results.
  2. Build a queue from real work. Tickets, PRs, pages, tool calls, the mix your production agent actually sees. Include the walk: look up the repo, find the credential, open the right environment. Do not start the agent on a gold prompt with the files already in /tmp.
  3. Do not reset the sandbox between items. Leftover files, leftover git state, leftover browser cookies are the point. If you must recycle a pod, keep the session record, the same split as the sandbox is the fixture, not the record.
  4. Inject faults on a clock. Hour 1: nothing. Hour 2: kill a tool. Hour 3: expire a token. Hour 4: a malformed ticket. Hour 5: a duplicate ticket that looks like the last one. Publish the schedule with the result.
  5. Cap recovery. If the agent retries a failed call without a ceiling, the soak should fail the run, not wait until morning. Funnel’s 254 identical SQL calls would have been a soak finding, not a cute footnote.
  6. Score three curves. Interventions per hour, minutes to recover from each injected fault, and quality in the last hour versus the first. A flat first-hour pass rate with a last-hour collapse is a fail, even if the median task still “passes.”
  7. Separate setup from work. Tag traces as “walk” or “job.” Keay’s complaint is that job metrics skip the walk. If half the tokens are spent finding the ticket, print that as the headline.
  8. Leave the room. The lunch sticky note is not a joke. If a person has to baby-sit the eval, you scored a demo. Schedule the soak when nobody is watching, then read the traces.

I would still keep kagent trajectory evals for the one-task regressions. They catch an agent that stops calling tools. They do not catch an agent that calls tools all afternoon and slowly poisons its own workspace.

Put the two next to each other on the board: unit test, then soak. Hardware people already know which one you trust before a ship date.

Frequently asked questions

What did Andra Keay say commercial robot deployments need?

In her October 31, 2026 Robots & Startups issue, Keay listed 8 hour continual operation performing tasks with high accuracy, robustness to perturbation, error recovery, safety around humans and, most importantly, integration into a whole work process rather than single or disconnected tasks. She added that getting to and from the task is part of the process and rarely included in the job metrics.

Why do single-task agent evals miss deployment failures?

A single-task eval starts from a clean state, runs one job and stops. Most production failures show up later: state left behind by the last task, a retry that never stops, a credential that expires mid-run, or a handoff that drops context. None of those exist in a run that lasts one task.

What is a shift-length eval for an AI agent?

A run that feeds the agent a realistic queue of work for hours without resetting its environment, injects faults on a schedule, and scores interventions per hour, recovery time and drift in output quality across the run, not just pass or fail per task.

What benchmark does Keay point to for whole-process robot work?

She says the DARPA Robotics Challenge in 2015 was the last time a benchmark really forced getting to and from the task, and that the new NIST Loco-Manipulation Challenge, launching at IEEE Humanoids 2026, is the attempt to pick that up.

How is a soak test different from a demo?

A demo is one task from a clean start with someone watching. A soak test is hours of mixed work, nobody at the chamber door, and a score for what happens after hour six: leftover state, recovery, expired tokens, and the walk between jobs.

Disagree or have a war story? Reply on X