What these articles never say is how many hallucinated bugs the LLM found that either weren’t real or were actually exploitable. The LLM didn’t find these with any confidence it highlighted areas of interest that actual security researchers then needed to investigate and confirm or rule out.
What these articles never say is how many hallucinated bugs the LLM found that either weren’t real or were actually exploitable.
It literally wouldn’t matter if it did.
The fact that it found exploitable bugs means that these bugs need to be addressed. To be clear, I care much more about the security flaws and fixing them than how they were discovered.
I feel like you missed the forest for the trees.
The question is how many were made up?
I saw that, and you’re right, I wasn’t answering that question. What I was saying was that I thought the question was irrelevant and ignoring a bigger issue.
I disagree that its ignoring the bigger problem, which is that slop like this is overwhelming devs to get fixes out ASAP faster than they can fix.
So now we have AI big reports feeding AI big fixes in a lot of projects.
The assumption that what AI finds is correct in the first place is… Probably wrong.
It makes stuff up all the bloody time, so how many of these bugs were made up, or not actually bugs?
In my experience, when given real world data to run on, they don’t hallucinate that often. Its when you ask it to regurgitate stored info that its off by a wild amount sometimes. Fixing or comparing code is like AI 101, unlike code generation where it may be incorrect.
The term 0-day vulnerability in security has a specific meaning. It means that it was not found by security researchers and is instead being actively exploited on the day that it’s discovered. I’m pretty sure none of these by definition or zero days
“Chrome’s record landed after Google overhauled its bounty program to cope with a flood of AI-generated reports.”
How did they change it?