


Contents (13 sections)
- The real problem: AI finds flaws everywhere
- Four decisions that change an AI audit
- 1. A ledger of what was not looked at
- 2. The one who checks is never the one who found
- 3. A doubt has no severity
- 4. The budget pays for verification first
- What Cloudflare's numbers say
- The sandbox: why most leads will stay "needs validation"
- The blind spot: the audited repository talks to the agent too
- Model safeguards: a refusal is not a verdict
- How we would use it
- Key takeaways
- Sources
On 4 November, the French site IT-Connect published a tutorial on Cloudflare's security-audit skill: a set of instructions that turns a coding agent, Claude Code for instance, into a security auditor. The skill is not new. It has been online since 18 June 2026, the day Cloudflare explained how it hunts for flaws in its own repositories, and it was reworked in September. We read it file by file, in its 14 September version, together with the two Cloudflare articles behind it and the Claude Code documentation. The tutorial explains how to install it. This article looks at what it is worth, where it stops, and how we would use it.
The real problem: AI finds flaws everywhere #
Finding flaws is no longer the hard part for an AI model. Sorting them is. Cloudflare puts it in one sentence in its Project Glasswing review, published in May: "Ask a model to find bugs, and it will find them, whether the code has any or not."
Independent measurements point the same way. In September 2025, Semgrep ran Claude Code and Codex against eleven open-source Python web applications. Claude Code found 46 real vulnerabilities, but with a 14% true positive rate: six alerts out of seven were wrong. On one application, three identical runs produced 3, then 6, then 11 different findings.
Maintainers pay for that noise. In January 2026, Daniel Stenberg closed the curl bug bounty. Before 2025, more than 15% of the reports it received were real flaws. From 2025 on, fewer than 5%. He wanted to "remove the incentives for submitting made up lies".
So an audit skill is not worth what it finds, but what it refuses to claim. That is where Cloudflare's skill is interesting.
Four decisions that change an AI audit #
The IT-Connect tutorial describes the six phases of a full audit well. Rather than repeat them, here are the four design choices that, on reading, set it apart from a plain "find me flaws in this code".
1. A ledger of what was not looked at #
Before any hunting, the skill builds a coverage ledger, coverage-ledger.json: every entry point in the code, crossed with the trust boundary it crosses and the type of attack to test. Each cell has a state, and a program shipped with the skill checks the ledger after every update. An area outside the requested scope is marked out_of_scope, never "covered". A quick pass must present itself as partial coverage. The instructions spell it out: never imply that one run exhausts the target.
It sounds like paperwork. It is actually the only honest way to answer the question a manager asks after an audit: "And the rest is fine?" A report without a ledger cannot say what it did not look at.
2. The one who checks is never the one who found #
Every lead found by a hunting agent goes to a fresh agent whose job is to prove it wrong. The claims that survive are checked a second time, in the source code, by agents that took no part in the discovery. Cloudflare goes further in its internal system: discovery and validation run on two different models.
3. A doubt has no severity #
Every lead ends in one of three states: confirmed, needs validation (needs_validation) or rejected. Only a confirmed flaw gets a severity, and that severity cannot exceed the demonstrated impact. A lead that needs validation is not a low-severity flaw: it is a precise hypothesis, blocked by a precise fact, for example the behaviour of a proxy that is not in the repository. The skill also rules out two classic sources of noise: a deviation from best practice is not a flaw, and neither is a missing second protection when the first one already blocks the attack.
4. The budget pays for verification first #
You can cap an audit in number of agent calls. The skill then reserves the calls needed for reconnaissance, coverage reviews and validation first, and gives the hunting agents only what is left. If the budget is not enough, the audit is declared incomplete, with the exact reason, rather than validating less. That is the opposite of the usual reflex, which is to hunt as much as possible and sort afterwards.
What Cloudflare's numbers say #
The published skill is the starting point, not the full tool. In its 18 June article, Cloudflare describes the system that grew out of it, applied to 128 repositories. That system is not published; Cloudflare only hopes it "will follow shortly". Its numbers still say a lot about the method:
- out of 20,799 raw leads, about 12,057 survived validation, and 5,442 duplicates were then merged; 7,245 actionable findings remained for the teams;
- as hunting improved, the validation rejection rate fell from 40% to 11%;
- for a repository of about 30,000 lines, a full run takes 3 to 4 hours and produces around a hundred leads;
- a single run finds about half of the flaws found across several runs, and each additional pass costs about half as much as the first.
And this sentence, which should open every audit report, human or not: "We don't claim a false-negative rate for our system. There's no labeled set of every real bug in a codebase." In other words, an empty report does not prove that code is safe. This is not a weakness of this particular skill: it is a limit of every audit, which the skill is honest enough to write down.
The sandbox: why most leads will stay "needs validation" #
Confirming a flaw often means running code: a test, a small demo application, a crafted file. The skill only accepts that inside a sandbox enforced by the operating system, with four conditions: no external network, an empty environment filled from an explicit list of variables, the target and toolchain read-only with writes limited to a scratch folder, and limits on CPU, memory, processes and time. If a single condition is missing, it runs nothing and leaves the lead as needing validation.
IT-Connect notes that the Claude Code sandbox "is not enough", without going into detail. Here is what the official documentation says, condition by condition, with default settings:
- External network: covered. On Linux, commands run in a separate network namespace, and traffic goes through a proxy whose list of allowed domains starts empty.
- Empty environment: not covered by default. Commands inherit Claude Code's variables, "including any secrets". You can remove some by naming them one by one, but there is no built-in list of secrets.
- Read-only target: not covered by default. Writes are allowed in the working directory, which is the audited repository if you start Claude Code there. Reads are open across the whole machine, credential files included, unless configured otherwise.
- Resource limits: absent. The documentation describes no CPU, memory, process or time limit.
Two more details. The sandbox only covers shell commands: Claude's file tools, MCP servers and hooks run outside it. And if the platform does not support it, Claude Code runs commands without a sandbox by default, unless failIfUnavailable is set.
So the skill is right to stay cautious, and that is a strength: it prefers a blocked lead to a confirmation obtained on your workstation. To go as far as execution, you need a disposable virtual machine, with no network route to your other machines and no credentials at all. Cloudflare itself runs its system's tests in a sandbox based on Linux namespaces, and notes that inside Docker the container's seccomp and AppArmor protections have to be relaxed. One more reason for a dedicated VM rather than a container on a shared machine.
The blind spot: the audited repository talks to the agent too #
This is the point we have not seen addressed anywhere. An agent that audits code reads that code, and for a model, reading text already means being exposed to instructions. A comment, a README or a configuration file can address the agent rather than the developer.
This is not theoretical. In September 2025, Checkmarx researchers showed that a well-phrased comment was enough for a Claude Code security review to report "0 new vulnerabilities" on malicious code. In another test, a payload split across several files ended up running on the developer's machine during the agent's tests. In August 2025, Johann Rehberger published CVE-2025-55284: an injection placed in a source file made the agent send the contents of a .env file to an outside server, hidden in DNS requests. The flaw has been fixed since version 1.0.4. Anthropic itself warns that its own GitHub security-review action is not hardened against prompt injection and should only be used to review trusted pull requests.
Cloudflare's skill handles prompt injection in the audited code very well when that code is itself an AI application: one of its guides is devoted to it. But in none of its fifteen instruction files did we find a sentence telling the auditor to treat the repository's text as plain data. And the architecture summary, written from reading the repository, is copied verbatim into every hunting agent's instructions. A booby-trapped sentence that made it into that summary would therefore be handed to every agent.
On the Claude Code side, two documented mechanisms make things worse when you start the agent from the repository being audited, as most tutorials do:
- At startup, Claude Code loads the
CLAUDE.mdfiles of the current folder and its parents, and those of subfolders as soon as it reads a file there, which an audit does constantly. These files are instructions for the agent: a malicious repository does not need to be any subtler. - A repository can declare hooks in
.claude/settings.json, that is, commands run automatically at certain points in the session. Once you accept the folder in the trust dialog, those hooks run, and they run outside the sandbox, with all your rights. In non-interactive mode (claude -p), there is not even a dialog.
An audit agent that reads an unknown repository, on a machine holding your keys, with network access, combines the three ingredients Simon Willison calls the "lethal trifecta": private data, untrusted content, and a way to communicate externally. France's cybersecurity agency, ANSSI, says the same thing in its guide on securing agentic AI systems, published on 30 October: treat the agentic system's inputs and outputs as untrusted, isolate its execution environment, and keep secrets out of the model's context.
Model safeguards: a refusal is not a verdict #
IT-Connect describes an audit interrupted by the model's safeguards. That is consistent with Anthropic's policy, updated on 6 October: general-purpose models have protections that "block most cyber work", but they can still be used for code review, patching known issues, vulnerability finding in your own source code and alert triage. Offensive work goes through the Cyber Verification Program, on application. On a benchmark of attack scenarios, Anthropic reports that Claude Opus 5.5 without that program blocked every task on the first prompt.
Cloudflare, for its part, observes that these refusals "aren't consistent": two equivalent tasks can get opposite answers depending on how they are phrased. In practice, a refusal says nothing about your code, and the absence of a refusal is not an authorisation. On your own code, a narrow scope and a clearly defensive request, which is what this skill is by design, are the right approach. Working around a refusal is not.
How we would use it #
We already use coding agents to review our code and configurations, with a dedicated account, the fewest rights possible and no production action without human approval. We have not yet run this skill on our repositories. Here is how we would go about it, and what we would advise a development team or a small business that maintains its own software:
- On your own code, at the right time. Before a release, or after work on something sensitive: authentication, permissions, file uploads, separation between customers. Not on code whose source you have no right to analyse.
- Small first. The
quickprofile, a precise scope, a budget in number of agents. A full audit with no limit can start dozens of agents. - On an identifiable state. A clean commit, output outside the repository (the skill's default), and a session started outside the audited repository if you did not write it yourself.
- Several passes. Successive runs add up: the skill reads previous ledgers and targets uncovered areas. With half the flaws found per pass, a single pass is a sample, not an audit.
- Read "needs validation" leads as a list of questions. They concern what the repository does not show: the proxy, headers, the web application firewall, the identity provider configuration. They are questions for the person who knows production, not for the agent.
- Fix as usual. Every fix goes through a branch, tests and a pull request. That is the rule we apply to all our repositories since we blocked direct pushes to the main branch.
What the skill does not replace: a penetration test on the real environment. It works on source code and rightly refuses to probe a deployed service. Everything that depends on the deployment stays out of its reach.
Key takeaways #
Cloudflare's skill does not bring a stronger model. It brings discipline: count what you did not look at, have every claim checked by someone whose job is to tear it down, refuse to give a severity to a doubt, and spend on verification first. That is exactly what was missing from the AI audits that swamped curl's maintainers. It lacks a clear instruction about the audited repository's text, and a sandbox most workstations do not provide. Both can be made up for by the environment you run it in.
Now that models find old flaws in bulk, as we wrote about Mythos and GPT-6 Astra, the question is no longer whether someone will comb through your code, but who will do it first. Want to audit your application before someone else does, or frame how coding agents are used in your team? Let's talk.
Sources #
- Cloudflare, security-audit-skill repository (MIT licence, online since 18 June 2026, version read: commit c1c8a8c of 14 September 2026): README and SKILL.md.
- Cloudflare, Grant Bourzikas, Build your own vulnerability harness (18 June 2026): internal system, figures, false negatives.
- Cloudflare, Grant Bourzikas, Project Glasswing: what Mythos showed us (18 May 2026): false positives, inconsistent refusals.
- IT-Connect, Florian Burnel, Cloudflare security-audit : prendre en main la skill d'audit de sécurité pour agents IA (4 November 2026, in French).
- Semgrep, R. Gaucher, V. Ermilov, C. Gibler, Finding vulnerabilities in modern web apps using Claude Code and OpenAI Codex (2 September 2025).
- Daniel Stenberg, The end of the curl bug-bounty (26 January 2026).
- Claude Code documentation: sandboxing, permissions and folder trust, CLAUDE.md files, security.
- Checkmarx Zero, Ori Ron and Dor Tumarkin, Bypassing Claude Code: How Easy Is It to Trick an AI Security Reviewer? (4 September 2025).
- Johann Rehberger, Claude Code: Data Exfiltration with DNS (11 August 2025), CVE-2025-55284.
- Anthropic, claude-code-security-review: prompt injection warning.
- Simon Willison, The lethal trifecta for AI agents (16 June 2025).
- ANSSI, Sécuriser un système IA agentique, Les Essentiels series, V1.0 (30 October 2026, in French).
- Anthropic, Expanding the Cyber Verification Program (6 October 2026).






