· 3 min read
The agent that investigates our sites every night started leaving half-finished runs behind, always at the same turn. What I found when I went digging.
Last week I wrote about the AI agent that investigates our sites every night. This week it broke. This is the more useful post.
The symptom: runs that ended after exactly twelve turns. No error. No conclusion. Nothing filed. Just rows in the database insisting a run was still going while nothing, anywhere, was running. I found four of these husks before I stopped calling it a fluke.
The agent works in legs. A long investigation spans several short-lived processes, each doing a stretch of turns and handing off to the next. Twelve turns was exactly two legs. Whatever was killing runs was killing them during the second handoff, every single time.
The culprit was a platform detail I thought I understood. Our functions get a wall-clock budget per process, and we were starting leg three from inside leg two while leg two held its connection open. So leg three never got a fresh clock. It inherited whatever was left of its parent's, and by the third handoff there was nothing left to inherit. The platform reaped the process mid-handoff. From the code's point of view nothing failed. It just got unplugged.
I did not figure this out from first principles. I figured it out by noticing that the dead runs all had nearly identical lifespans and working backwards from there. Debugging is mostly noticing.
The fix was making the handoff an actual handoff: acknowledge fast, do the work detached, every leg starts with its own clock. Each leg also watches its own elapsed time now and stops early on its own terms instead of the platform's. Boring fixes. Boring fixes are the good kind.
While reading the dead runs I found a second bug, and this one bothers me more.
One run ended cleanly on a site that had plenty of sessions worth looking at. The agent investigated nothing and filed nothing. In the transcript, its final turn contained internal reasoning and then no message and no tool call. Our loop saw an empty reply and decided the model was finished.
It was not finished. It ran out of room mid-thought, and the truncation looked identical to a deliberate stop. We had built the agent equivalent of assuming the coworker who went silent in a meeting must have completed all their tasks.
So now the loop pushes back. An empty reply gets one nudge: you said nothing, either act or say you're done. A second empty reply ends the run, and the record says so in plain words. Silence no longer gets to look like a decision.
Both bugs made the agent do too little, so before closing out the week I asked what too much would look like. Easy to imagine. One bad night, one misfiring detector, and the agent files twenty issues before sunrise. Twenty issues at once is not triage. Nobody reads the fifteenth one.
Hence the nightly cap, small on purpose. If the agent finds more real problems than the cap allows, the extras wait for tomorrow. They were live and unnoticed for weeks already. One more night changes nothing.
None of this made the agent smarter. The model was never the problem. All three bugs lived in the seams, in a wall clock and in the meaning of an empty string, places no eval will ever look.
Spend your paranoia on the seams.
