What AI is actually good for in security research

I spent a weekend pointing an AI agent at a widely-used open-source library — the kind of dependency that sits three levels down in a Gemfile.lock you’ve never read — to see whether it could find a real vulnerability. It could. The bug is a credential leak, it’s reproducible on every released version, and I’ve reported it privately to the maintainer. I’m not naming the library here, for reasons I’ll come back to at the end.

The finding is the least interesting thing that happened. What I actually learned is where an AI agent earns its keep in this kind of work and where it will confidently walk you off a cliff, and those two things turn out to be the same underlying trait viewed from different angles.

The trap: an agent that agrees with itself

If you ask a model “is this code vulnerable?”, it will find you a vulnerability. It wants to be helpful, and “here is a subtle flaw in your security-critical code” is a very helpful-shaped answer. Left to narrate freely, it produced a tidy paragraph about how the library forwarded sensitive data across a trust boundary, complete with a plausible line-number citation and a severity rating. It read like a real advisory.

It was also, at that point, worth nothing. The model had reasoned about the source code and told me what the code looked like it did. That’s a hypothesis dressed as a finding, and the gap between those two things is where every bad security report lives. A library’s documented behavior, its actual behavior, and the behavior a language model infers from reading it are three different objects, and they disagree in exactly the cases that matter.

So the rule for the whole exercise became: the model is allowed to generate hypotheses; it is not allowed to confirm them. Confirmation only counts if it shows up on the wire.

The harness: assert on bytes, not on prose

The thing that made the agent useful was a test harness that took its claims and forced them through reality. It stands up two throwaway servers on the loopback interface — one playing a “trusted” host, one playing an “attacker” host that records everything it receives — and drives the library at them with a marked secret in the credentials:

SECRET = 'CANARY-a1b2c3d4-DO-NOT-LEAK'

Then every check is a question answered by observation, not by argument: did the bytes containing CANARY-a1b2c3d4 arrive at the server they were never supposed to reach? The recording server is raw TCP on purpose, not a real web framework, because I wanted it to be able to send malformed and hostile responses that a normal server library would politely refuse to emit. You cannot test how something handles a broken redirect if your test server won’t send you a broken redirect.

Two details mattered more than I expected. The first is that the “trusted” and “attacker” servers both live on loopback but are addressed by different host strings — because host-string comparison is exactly the logic under test, and if both sides had been literally identical the whole test would have been meaningless while appearing to pass. The second is the canary. A unique, searchable token in the secret means “did it leak?” is a substring search across every byte recorded anywhere, not a judgment call. There’s no interpreting a canary. It’s in the wrong place or it isn’t.

This is the part I’d want anyone doing similar work to steal: make the ground truth a physical event the model doesn’t get to characterize. The agent can write the hypothesis, propose the check, even write the server code. But the verdict comes from a byte that either showed up or didn’t, and no amount of fluent explanation moves it.

Where the agent was genuinely good

Within that frame, the model was excellent at things that are tedious for a human and well-suited to a machine.

Generating the adversarial cases. It enumerated variations I’d have gotten bored of writing by hand: the obvious version of the attack, then the protocol-relative version, then the one with a duplicated header, then the same idea against a different credential-carrying mechanism. Breadth of “here’s another shape of the same trick” is real work and the model is tireless at it.

Tracing intent through history. The most convincing part of the eventual report wasn’t the proof of concept — it was the model reading a decade-old commit that had added a partial defense, quoting the commit message where the author stated the security goal, and showing that the implementation only ever reached one of the several ways the dangerous input arrives. “This is a known bug class, the maintainer clearly tried to defend against it, and here is precisely where the defense falls short of its own stated intent” is a far stronger and fairer claim than “I found a bug.” A human could assemble that. Most humans wouldn’t bother. The agent did it in minutes.

Killing its own findings. Once the harness existed, I had the agent turn it on every hypothesis, including the ones it was proud of. One candidate finding — an XML-parsing issue that would have been a great headline — simply didn’t reproduce. The default parser on a current stack doesn’t do the dangerous thing, and the wire capture showed a harmless result where the theory predicted a file read. Into the report it went, as a disproven finding with the evidence. A report that only contains hits is a report that never ran a control.

Where it lied to me

Now the other side, because it’s the important side.

It invented a leak that was really my mistake. At one point a check went red and the model happily wrote up the confirmed vulnerability. When I forced a re-check, the leak evaporated — the “credential” the attacker server had recorded was an artifact of how I’d set up that specific test, not something the library did. The model had no instinct that the result was too convenient. It got the answer it was steered toward and reported it. If I’d trusted the first green-to-red transition, I’d have shipped a false accusation about someone’s code.

That’s the whole thing in one incident. The same eagerness that makes the model good at generating attack variations makes it want every check to confirm. It has no skin in being wrong, no professional embarrassment waiting on the other side of a bad report. You have to supply that yourself, structurally, because it will not come from the model.

It reached for severity it hadn’t earned. Left alone, the model’s instinct was to rate everything High and to describe the worst-case exploitation as though it were the default case. The real severity depended on conditions I couldn’t verify from the library alone — it needed a particular kind of application, doing a particular thing, for the bug to bite. The model was happy to assume all of that into existence. Grounding the severity meant explicitly separating “this is what the code does” (certain, tested) from “this is what it would take to exploit” (conditional, stated as conditions).

Line numbers drift. Cited locations were roughly right and precisely wrong often enough that I stopped trusting any file-and-line reference until I’d opened the file. Useful as a pointer, worthless as a citation.

The division of labor that worked

By the end the working relationship had a clear shape, and it’s not “AI finds bugs.”

The model is a hypothesis engine with enormous breadth and zero judgment about whether it’s right. It reads more code than I would, remembers more bug classes than I do, generates more variations than I’d have the patience for, and traces intent through history beautifully. It is also structurally incapable of being disappointed when a finding turns out to be real, which is the exact emotion that makes a human security researcher careful.

So the human’s job moves. It’s not to find the bug; the machine is better at surfacing candidates. It’s to build the thing that tells the truth — the harness, the canary, the insistence that confirmation is a physical event — and then to be the adversary who assumes every green result is a measurement error until it survives a second look. The judgment didn’t go away. It concentrated into designing the test and distrusting the result.

I got one solid, reportable finding out of the weekend, plus a couple of lesser ones and one satisfying disproof. Every single one of them was generated by the model and earned by the harness. Neither half would have gotten there alone: without the model I wouldn’t have enumerated the cases, and without the harness I’d have a beautifully written report full of things that aren’t true.

On not naming it

You’ll notice I haven’t said which library this is. That’s deliberate, and it’s the last thing worth saying about doing this responsibly.

The finding is real and it’s unpatched, which means writing “library X leaks your credentials, here is exactly how” is not a blog post — it’s a disclosure, and disclosure has an order of operations. The maintainer got a private report with a proof of concept and a suggested fix before I wrote a word of this. I told them I’d hold the details until there’s a patch or a reasonable amount of time passes, and this post is me keeping that promise: everything here is about method, and nothing here is a recipe against a named target.

If and when it’s fixed — or the window I offered lapses — I’ll write the other post, the one with the name and the proof of concept and the timeline. The gap between the two is the difference between security research and just publishing an exploit, and the AI, for all its help finding the thing, has no opinion whatsoever about which one you become. That part is still entirely on you.