An AI investigates our website every night

Every night around 7:30 UTC, an AI agent reads a compressed brief of the day's sessions on our sites, decides what looks suspicious, pulls the raw evidence, and files issues when it can prove them. We built it, so you'd think we would trust it. Honestly, we expected to spend the first month rejecting nonsense.

Instead it keeps embarrassing us. Two examples from our own properties.

The accordion that failed at every width

The agent flagged our FAQ accordion for overlapping nearby links. Fine, responsive bugs happen. The part that got our attention was the reasoning: it had evidence of the overlap at 416px, at 1920px, and at 2560px, and it concluded that a bug reproducing at mobile, laptop, and ultrawide widths at the same time is not a breakpoint problem. It's a stacking problem. It recommended auditing the component's z-index and positioning, and adding regression tests at all three widths.

It was right. The bug was in the component's stacking context, not in any media query. A human would have loaded the page on their own laptop, seen the overlap, fixed it for that width, and plausibly shipped a fix that solved a third of the problem. The agent's advantage wasn't intelligence. It was that it had already seen the failure at three widths before forming a theory.

The table that begged to be clicked

Second one. On a comparison page, the agent noticed a visitor rage clicking table cells 55 times in ninety seconds. The cells had hover styling, so they looked interactive, and they did nothing. That alone is a decent finding. But it also measured the contrast on the table's "No" cells: 2.26 to 1, against the 4.5 accessibility threshold. Its diagnosis tied the two together: people could barely read the negative cells, so they clicked them hunting for detail that didn't exist, and the hover styling promised them it was there.

We had looked at that table many times. It looked fine to us, because we knew it wasn't clickable and we knew what the faint cells said. Familiarity is a rendering bug in the human reviewer.

What we learned running it on ourselves

A few things surprised us enough to change how the system works.

First, the floor matters more than the ceiling. Early on we cared about how smart the write-ups were. In practice the feature that builds trust is the boring one: the agent cannot file an issue unless the sessions it cites actually exist in the database, with a minimum of verified evidence behind every claim. An AI that occasionally hallucinates a bug is worse than no AI, because you stop reading the real ones.

Second, we rate everything. Every issue it files can be marked confirmed or rejected by a human, and we track its precision. Some nights it files nothing, which was hard to accept at first and is obviously correct: a quiet night should produce silence, not filler.

Third, it never gets bored. The two bugs above had been live for a while. No user reported either one. No dashboard flagged them. The nightly agent found both because it looks at all the evidence every day with the same patience, which is the one thing a team demonstrably cannot do.

This is the product we sell, so read the enthusiasm with that in mind. But the underlying lesson doesn't require buying anything: the bugs that cost you conversions are mostly ones you shipped confidently and stopped looking at. Whatever process you use, human or machine, the trick is having something that keeps looking after you've stopped.