# Teaching an agent to skim

Canonical: https://flusterduck.com/blog/teaching-an-agent-to-skim
Published: 2026-07-23 · The Flusterduck team

Our nightly agent reads every session from the day before deciding what to investigate. It can only afford that because of the least glamorous part of the system.

The [nightly agent](/blog/ai-investigates-our-site-every-night) has one advantage over a human reviewer, and it is not intelligence. It reads everything. Every session from the day, before it forms a single theory about what's wrong.

That advantage has a price problem. A raw session is thousands of tokens of event JSON. A busy site does thousands of sessions a day. We charge a flat rate. Run that math and one night of reading raw data costs more than the customer pays us in a month. People ask what the hard part of building this was and expect to hear about prompts. It was this. It was always this.

## Sessions become lines

We compress each session into one short line of a rigid, code-like notation carrying the shape of the visit: which pages in what order, roughly how long, and every friction signal pinned to the element where it happened. A flight recorder summary instead of the cockpit video.

The notation is boring on purpose. Every line has the same skeleton, so a few hundred sessions read like a table, and a model is extremely good at spotting the row that doesn't belong in a table. The agent doesn't need detail at this stage. It needs to notice that eleven sessions stalled on the same form field yesterday, and it needs that noticing to cost basically nothing.

We tried plain aggregates first, obviously. Even cheaper. A counter saying "43 rage clicks on /pricing" turned out to hide the one thing that matters: whether that was 43 people mildly annoyed or one person losing a fight with a table for ninety seconds. Different bugs. You can only tell them apart if every session keeps its own line.

## Then it digs

Skimming produces a suspicion, and the agent is not allowed to stop there, because a compressed line is a summary and summaries lie by omission.

The suspicious leads get the expensive treatment: raw events for those sessions, the rendered page, a comparison against the page before the last deploy. A typical night skims the whole day for pennies and spends actual dollars on two or three deep dives. This is an index and a table scan wearing new clothes. Most of what gets called agent engineering is old database ideas wearing new clothes.

## The version that failed

Our first brief was English. A cheap model narrated each session in a sentence or two and the smart model read the narrations. It demoed beautifully. It worked terribly.

Partly cost: narration burns tokens on connective tissue like "the user then proceeded to," which carries nothing. The worse problem was that the cheap model phrased the same behavior differently on different days, which broke pattern-matching across sessions. The entire point of reading everything is that repeated problems line up. Prose does not line up.

So we stopped treating the brief as writing and started treating it as a file format that happens to pass through a language model. That one reframe improved the agent more than any model upgrade we've made. I find that slightly deflating to admit, given the year everyone has had about model capabilities. It's still true.

If you're building an agent on production data, here's the claim I'll defend: your agent's ceiling is set by what it can afford to pay attention to. Context windows keep growing. Attention is still billed by the token. The system that reads all the evidence cheaply beats the system that reads a sample brilliantly.

Make the evidence cheap.