The topic of How to verify AI-discovered vulnerabilities aren't just training data echoes is currently the subject of lively debate — readers and analysts are keeping a close eye on developments.
This is taking place in a dynamic environment: companies’ decisions and competitors’ reactions can quickly change the picture.
Last month a friend DM'd me a screenshot. An AI security agent had "discovered" a vulnerability in a popular open-source project. The agent walked through exploitation steps, suggested a patch, the whole nine yards. Looked legit.
Then someone pointed out the CVE ID it kept almost-quoting was from years earlier.
This is going to keep happening. As we wire LLMs into vulnerability research workflows, we run into a problem that doesn't have a clean analogue in traditional static analysis: the tool you're using may have already seen the answer in its training data, and it cannot reliably tell you which findings came from reasoning and which came from memory.
I've spent the last few months adding AI-assisted triage to a security workflow at a contracting gig. Here's what I've learned about not getting fooled.
If a CVE was disclosed before a model's training cutoff, the model has very likely seen a description of the bug, the patch, and probably someone's analysis of it. When you point that same model at the vulnerable file, it isn't always finding the bug — sometimes it's recognizing it.
The tricky part: the model usually can't tell you which is which. It generates the same confident output either way. There's no internal flag for "I retrieved this from memory" versus "I derived this from the code in front of me."
This is the same phenomenon that makes LLMs unreliable for leaked benchmark questions — if the benchmark made it into training, the model "solves" it by recall. The security version just has higher stakes.
Here's the rough process I run on any AI-flagged finding before it gets escalated. None of this is exotic — it's stuff I wish I'd been doing from day one.
Before you trust any finding, fuzzy-match the bug fingerprint against known CVEs. The NVD publishes JSON data feeds you can pull locally:
If you get a hit above ~0.6 similarity, your "discovery" is almost certainly a memorized CVE. SequenceMatcher is dumb but it catches the obvious cases. For better recall use sentence embeddings (the sentence-transformers library works fine) but start with the dumb thing — it's faster to debug.
Git history doesn't lie. If the model says "this buffer overflow in parse_packet," run blame on the offending lines and check what the file looked like at different points in time:
If a fix landed for this exact code path years ago and the model is "discovering" it against modern source, you've already got your answer. Either the bug is fixed (and the model is recalling the pre-fix version), or there's a regression — which is worth knowing either way, but it's not a novel discovery.
Here's a trick that's saved me a lot of time. Run the analysis again with the package name and obvious identifiers redacted. Replace function names with hashes:
If the model still flags the same vulnerability class on the anonymized code, the finding is probably grounded in the code in front of it. If it suddenly can't find anything, you were getting recall.
This isn't bulletproof — distinctive code structure can still trigger memory — but it filters out a lot of noise. I haven't tested this thoroughly against every model family, so calibrate your threshold against findings you already know the answer to.
Even when an AI tool does genuinely identify a real bug, you usually can't tell from the output alone whether it reasoned its way there or got lucky with memorization. That isn't a bug in any specific tool — it's a property of how these models work. The validation step isn't optional and it isn't going away.
The good news is that the validation is straightforward. The bad news is that I keep meeting teams who skip it because the AI sounded confident.
Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink.
For further actions, you may consider blocking this person and/or reporting abuse
DEV Community — A space to discuss and keep up software development and manage your software career
Built on Forem — the open source software that powers DEV and other inclusive communities.
Why it matters
News like this often changes audience expectations and competitors’ plans.
When one player makes a move, others usually react — it is worth reading the event in context.
What to look out for next
The full picture will become clear in time, but the headline already shows the dynamics of the industry.
Further statements and user reactions will add to the story.