Security
The Bug Hunter May Be Free but The Tokens Are Not
What the cloudflare security skill does and it's true cost to operate.
In June, Cloudflare released the skill that seeded its own system for finding security flaws. A skill is a packet of instructions that turns an AI coding agent into a specialist. This one turns the agent into a security auditor. It started at about 450 lines and grew in six weeks into a harness that scans 128 Cloudflare code repositories. The code is under the MIT license, and the project had about 55,800 GitHub stars by November 1. Most of the praise stops at the word free. That is half the story as the license costs nothing. But every run costs tokens, the units AI vendors bill by, plus hours of machine time. Cost is the loudest complaint in early reviews.
This article explains what the skill does and why it is worth the money. Then it shows how to cut the time and tokens it takes with:
- A tuned harness
- Models you run yourself and
- Focused decision models such as Jev
We are biased toward open models, where they fit and we'll be sure to say where that bias might have limitations.
What the skill does
The skill runs six phases, laid out in the Cloudflare repository.
- Recon agents map the code. They write an architecture file and a coverage ledger. The ledger works like a building inspector's room list. It records which parts of the code have been checked for which kind of attack.
- Isolated hunters take items from the ledger and look for flaws. Critics then review, looking for gaps the hunters missed.
- Each candidate goes to a fresh verifier whose only job is to disprove it.
- Each finding lands in one of 3 piles: - Confirmed means a full trace and an observed result. - Needs validation means one fact is still open. - Rejected means it was disproved.
- New agents check the final claims against the source once more.
- Reports are written from the verified records.

The skill also ships eleven attack playbooks. They cover:
- prompt injection and agent tools
- supply chain and release
- cloud identity and containers
- and tenant isolation
Why the method matters more than the model
Cloudflare's post makes a point that is easy to skip. The company says its approach centers on the harness, not the model, and it runs whatever frontier model is best at the time. Different models find different bugs. So Cloudflare uses one model to discover and a different one to validate. Validators may disprove a finding but never file one.
The numbers show why the disprove step pays. Across its fleet, the discovery harness produced 20,799 raw candidates. About 12,057 survived validation. After duplicates and low risk items were folded, 7,245 actionable findings went to teams. Without the second opinion engineers would have chased thousands of ghosts.
An outside test points to the same results. In a small blind test filed on the project, the skill reached a median of 90 percent precision and never fell for planted decoys. It also missed a known flaw in a pinned dependency in all three rounds. That is by design, since it does not guess on outside facts so keep your dependency scanner. Be sure to understand the limits too. The tool reads source code and never probes live systems. Without an isolated sandbox it will not run the target code, so more leads end up as needs validation. It is a first pass, not a replacement for a professional penetration test.
Where the time and tokens go
Total tokens equal the number of agent calls times what each call reads and writes. Total time equals the longest chain of steps that must wait on each other. Think of a taxi meter. The fare grows with the number of trips, the length of each trip and the rate per mile. You can cut any of the three. Cutting trips saves the most, because a trip you never take has no length and no rate.
The skill takes many trips by design. Each phase launches fresh agents, and fresh agents must reread context. Users on a public forum reported about 150,000 tokens on a small project and about one million on a medium one with nothing to show. The main instruction file alone is 22 KB and loads on any security request. The project's own README says a single run finds roughly half of what repeated runs find.
Cloudflare's internal numbers set a baseline. A typical repository of about 30,000 lines took three to four hours for the first pass and about 14 hours end to end, including draft fixes. Most compute went to the hunt. Each extra gap filling pass cost roughly half the first hunt.
The common reaction is to buy a bigger plan or a bigger model. First principles says cut in order. Cut the trips. Then cut what each trip reads. Then cut the price per token.
Seven ways to cut time and tokens
Scan the change, not the whole city. The skill can be scoped to named paths, one subsystem or the difference between two versions of the code. It also has three modes. Quick runs one hunter wave, one critic pass and one verifier per candidate. Standard runs the full workflow. Deep splits the work finer and keeps going until critics come back clean. One reviewer suggests quick on every pull request and deep each quarter. Think of a home inspector for a new house addition. They check the new addition, not the whole house, each time you build. Set a budget cap as well. The skill counts agent calls and refuses to start if the budget cannot fund a real pass.
Keep the map and reuse it. Runs build on each other. Confirmed findings on unchanged code carry forward and get checked again, so you do not rescan everything. A modified harness can store the architecture file and ledger by commit. Cloudflare's advice is to save everything in a small database keyed by run, repository and stage before adding more agents. A good map also cuts waste later. When Cloudflare gave its recon better context, the share of findings rejected at validation fell from 40 to 11 percent.
Trim what each agent reads. Cloudflare keeps each agent below a quarter of its context window to avoid made up answers. Load only the playbooks that fit the target. A Go web service does not need the mobile app playbook. A modified harness can hand each hunter only the files in its ledger unit. Search helpers such as jevgrep go further and find code by what it does. More on that below.
Pay once for shared text. Many agents read the same opening text. Prompt caching stores that text after the first read. On Anthropic's API, cache reads cost 10 percent of the normal input price. On your own GPUs, vLLM's prefix caching does the same job, and its designers call it almost a free lunch. The rule is simple. Put stable text such as the skill, the architecture file and the playbook first. Put the part that changes last. Think of a briefing binder which is left for others to review reprinting for every meeting.
Let code do code's job. Cloudflare's report stage is a script with no model at all. Validation starts with plain code that checks format and file paths. Duplicate checks start with simple indexes, and an agent reasons only over the short list. Every check a script can do is a model call you do not pay for. One caution though, Cloudflare found that wiring in traditional static analysis early had little payoff, and its hunters rarely used it. Measure before you assume a filter helps.
Match the model to the job. Not every step needs the strongest model. The hunt does. Bookkeeping, sorting and summarizing do not.

Read it top down. Each tier costs more per call than the one above it, so work moves down only when it must.
Open models have earned a place here. On a public CyberGym leaderboard, a benchmark where agents reproduce real flaws, open weight models hold the top four rows. GLM-5.3 ties GPT-5.6 Sol at 84.5 percent. The page warns that most rows are vendor self reports. In a private test in April, an independent researcher found GLM 5.1 ahead of Claude Opus 4.6. A mid sized Qwen model small enough for local hardware also did well. Bearing in mind, those were single runs.
My bias is to run the many cheap steps and the second opinion on an open model inside your walls. That gives you Cloudflare's model diversity at low cost and keeps findings at home. Keep a frontier model for the hunt until an open model proves itself on one of your repositories with known bugs. One caution for public agencies, check your policies on open model restrictions.
Spend time where work can run side by side. Time comes from waiting. Hunters can run in parallel and Cloudflare uses a pool of 50 to 200 workers with a task cap per repository. Local models remove vendor rate limits for bulk steps so kill dead runs early. Cloudflare flags hunts that return no findings and spawn no new tasks, because they often signal a crashed dependency. Stop running once a unit stops producing findings.
Where Jev fits, and where it does not
Jev is a new kind of model from TypeSafe AI, launched September 15. You cannot chat with it. You send it the state of a system and typed questions. It returns choices, scores and probabilities that software can use at once. Where chatbot is a tour guide, think of Jev as an elevator to your outcome.
The price fits harness gates. Published pricing is 4.2 cents per million input tokens with free output, with a 64,000 token window and vendor reported replies in 70 to 500 milliseconds. A harness asks thousands of small questions. Is this file in scope? Does this finding match one we already have? Did this hunt go shallow? Jev was built for questions like these.
jevgrep, an MIT licensed tool by David Zhang, uses Jev to find code by asking what it does. Its own tests are candid about mixed results. On ten tuned Python tasks, agent cost fell 28.6 percent before counting Jev, and total cost fell 25.8 percent after. In the next version, Jev's own cost fell, yet combined cost came out 2 to 3 percent higher. Single runs on ten tasks prove little either way.
Now the risks. Jev's weights are closed, and it runs as a hosted service. jevgrep sends eligible source code to Jev through the provider you choose. It filters likely secrets, but only on a best effort basis. In a security audit, your code and your findings form a map for attackers. TypeSafe's own documents list weakness with adversarial instructions and irrelevant context, and audited code can hide hostile text in a comment. The bold claims, up to 193.6 times faster and 444.6 times cheaper than frontier models, come from TypeSafe's own tests so understand by design they are favorable.
So use Jev where a wrong answer is cheap and the code is not secret. Pin a version. Measure cost per confirmed bug before and after. For sensitive code, a small open model on your own hardware can play the same gate role with no data leaving.
Your options
| Route | Speed to start | Where your code goes | Skill you need | Main risk |
|---|---|---|---|---|
| Run the skill as is on a frontier API | Fast | The model vendor | Low | Bills grow with repo size, and one run finds about half |
| Tune the skill with scope, modes, caching and a budget | Fast | The model vendor | Medium | A narrow scope can hide bugs that cross modules |
| Hybrid harness with open models in house and a frontier hunter | Slow | Mostly you | High | GPUs, tuning and model choice fall on you |
| Add decision models such as Jev for gates | Medium | Jev's provider | Medium | A closed hosted model sees code, and savings are unproven at scale |
Our bias is to start with the second route this month. It costs only discipline. Move to the hybrid route when volume is steady enough to keep GPUs busy. Treat Jev as an experiment on code that is not sensitive.
A cadence keeps the meter honest. Light and frequent near the code, heavy and rare where the stakes are highest.

The real limit is fixing, not finding
Finding costs tokens but fixing costs people. Cloudflare drafts patches with AI, but a human reviews every branch and nothing merges on its own. About ten critical and high bugs reached production in about five days. The rest rolled out over 15 to 20 days.
That sets the true budget. If your team can fix 20 bugs a month, a run that finds 200 does not make you ten times safer. It builds a queue faster than anyone can empty it, and a long list of known open flaws is a risk in itself. Size discovery to what your team can absorb. Then judge every change in this article by one number, cost per confirmed and fixed bug.
Four questions for your team this week
- What did our last AI code audit cost per confirmed bug?
- Which repositories change most, and do we scan only the change?
- Which audit steps could a script or a small local model do instead?
- When an agent reads our source code, where does that code go?
More from Cloudflare worth bookmarking
The skill is not Cloudflare's only open gift. Its public post mortems show how a large operator explains its own failures. The Learning Center explains attacks and defenses in plain terms. CFSSL is a toolkit for working with certificates. Cloudflare Radar shows traffic and attack trends across its network.
Additional Reading & References
- Build your own vulnerability harness, Dan Jones, Alexandra Godoi and Grant Bourzikas (Cloudflare)
- security-audit-skill repository and README, Cloudflare (GitHub)
- Cloudflare security-audit-skill review, agent vulnerability hunting, andrew.ooo
- Cloudflare Security Audit Skill, Nicolas de Ville (notes)
- Cloudflare security audit skill, GitHub repo, install, and limits, Vibe Eval
- Prompt caching, Anthropic (Claude documentation)
- Automatic Prefix Caching, vLLM project (vLLM documentation)
- CyberGym benchmark scores, BenchLM
- Benchmarking open weight models for security research, Noah Lebovic (Dual Use)
- TypeSafe AI Launches Jev for Structured Software Decisions, Let's Data Science
- What Is Jev? Why System One Models Matter for Enterprise AI, HatchWorks AI
- Jev model specifications, CometAPI
- jevgrep, find code by asking what it does, David Zhang (GitHub)
© 2026 Symvoulos LLP. You may quote or excerpt this article with attribution and link to the original URL as well as referenced material. Republication in full, adaptation, translation, or use as the basis of derivative or secondary works requires attribution. Removal or alteration of attribution is prohibited.
Stathy Touloumis is President of Symvoulos (Symv), a boutique technology consultancy serving all sectors of business and government entities including regulated enterprises. His experience spans over 30 years with most recent to state, local, and public entities across infrastructure modernization, security, service delivery strategy, and the governance questions raised by AI adoption. Symvoulos’ expertise starts where most stall: multi-year cost models, risk vectors and staffing plans an agency can defend in a budget hearing. Be sure to follow us at symv.ai
Want to talk about your own case?
Share a few details. A person on our team will reply.
Start a conversation