Research · Open for participation
We are assessing 100 SaaS applications built primarily by AI coding assistants, using one fixed test plan, and publishing the frequency of what we find. The methodology is below. The results are not, because we have not collected them yet.
Plenty of security companies publish research with impressive numbers and no method. We are doing it the other way round: protocol first, results when they exist. If you are reading this before the study closes, the honest answer to “what did you find” is that we are still finding it.
Protocol
Published in advance so the method cannot be adjusted after the data starts arriving.
Production or pre-production SaaS applications where the majority of application code was generated by an AI coding assistant. Self-reported at intake, verified during scoping by reviewing commit patterns and the team's own account of how the product was built.
100 applications. Results will be published at 100, with an interim report at 25 if the pattern distribution is already stable. We will publish the achieved sample size, not a rounded one.
A fixed test plan applied identically to every participant: data-layer policy review, two-tenant authorization sweep, built-bundle secret extraction, endpoint enumeration including routes the interface never calls, and rate limit testing on authentication and paid-API endpoints.
Every finding classified against CWE and, where applicable, the OWASP Top 10 for LLM Applications. Severity assigned by a consultant using exploitability and business impact, with the rationale recorded so the distribution can be audited.
No participant is named, and no finding is published in a form that could identify an application. Published data is aggregate: frequency by category, severity distribution, and anonymised code patterns rewritten to remove identifying detail.
Findings go to the participant first, in full, before anything is aggregated. Participants receive their individual report regardless of whether the study proceeds, and may withdraw their data from the aggregate at any point before publication.
Full methodology, the achieved sample size, the frequency table and the limitations section published openly on this site under a permissive licence. No gate, no email capture, no summary-only version.
Limitations
Stated up front, because a limitations section written after the results is a limitations section written to protect them.
Prior work
This study is not the first look at AI-generated code security. These are the results it builds on, attributed to the researchers who produced them.
Analysed Copilot completions across scenarios drawn from the CWE Top 25 and found a substantial proportion of generated code contained security-relevant weaknesses.
Why it matters hereThe first large systematic look at assistant-generated code. Establishes that the question is empirical rather than theoretical.
Participants with access to an AI assistant produced less secure solutions on several tasks, and were more likely to believe their code was secure than participants without one.
Why it matters hereThe confidence gap matters more than the defect rate. A developer who believes the code is fine does not review it.
Consensus categorisation of LLM application risk, including prompt injection, improper output handling, excessive agency and unbounded consumption.
Why it matters hereThe reference taxonomy we classify against, so results are comparable with other work rather than using a private scheme.
Ranked list of the weakness types most prevalent and impactful in reported vulnerabilities.
Why it matters hereClassification baseline. Using an established scheme is what makes a frequency table meaningful to anyone else.
Summaries above are our characterisation of other researchers’ work, not quotations. Read the original papers before citing them; we have linked the titles and venues so you can find them.
Because publishing a protocol before collecting data is what separates research from marketing. It means we cannot quietly adjust the method once we see which findings would make a better headline, and it lets anyone reading the eventual results judge whether the design supports them.
Participation is free and you receive a full individual report. The exchange is that we may use your anonymised results in the aggregate, and you can withdraw that permission at any point before publication. We will also, obviously, know your product well enough to offer you further work. You are under no obligation to take it.
No, not even with permission. Naming any participant makes the others identifiable by elimination in a sample this size, and a study where being included implies you had findings is a study nobody sensible joins.
You get told immediately, not at the end of the study, and you get the reproduction steps and the fix. Aggregate publication happens long after, and only in a form that cannot identify you. If you would rather withdraw entirely at that point, you can.
Then we publish that. A finding that AI-built applications are no worse than hand-written ones would be a genuinely useful result and we would report it as readily as the alternative. Committing to publish before you know the answer is the point.
You get a full security assessment at no cost and your results before anyone else sees an aggregate. We get one more data point toward a study worth citing.
No sales sequence. A scoping call and a written proposal cost nothing.