Product security teams need to investigate suspicious code, validate findings and review changes before release. When source code cannot leave the organization, that work has to run on infrastructure the team controls.
Locally deployable models have already produced reportable vulnerabilities. In our OP-TEE research, GPT-OSS models found three memory-corruption issues in the privileged core; two received CVEs. Those findings came from investigating software used to isolate sensitive code and data from the normal operating system.
For teams evaluating local AI security review, the practical questions are what hardware it requires, which tools produce useful findings, and how long reviews take. We compared SecMate, OpenCode and Arm Metis on five known vulnerabilities, using the same DeepSeek Flash model served from a single NVIDIA DGX Spark. A first evaluation does not require a six-figure GPU cluster.
Hardware and local setup
Our Spark has 128 GB of shared CPU and GPU memory.
Ollama and vLLM can expose local models through an OpenAI-compatible API. Our DeepSeek Flash checkpoint needs the adapted vLLM and SparkInfer Docker stack. Its published recipe handles loading the model and the Spark’s GPU kernels.
The review harness chooses files, calls the model and checks suspected bugs. It can run on the Spark or on another machine, such as a virtual machine on the same private network.
We used fresh review sandboxes with external connections blocked and a relay pointing only to our model. Our experiment was network isolated. For a physical air gap, transfer the model, images, rules and dependencies offline before starting.
Model details
We used 0xSero’s adapted DeepSeek-V4-Flash-0731 checkpoint, which occupies about 100 GiB. It removes 40 of the original 256 routed experts, the model’s specialist subnetworks. REAP selects which experts to keep, and EXL3/Trellis compresses their weights to around 3 bits each. Our server allowed 262,144 tokens of context.
In our tests, generation speed varied from roughly 10 to 40 tokens per second. We have not compared this modified model’s quality with the original release.
The tools we compared
SecMate is optimized for smaller models and integrates with GitHub, GitHub Enterprise and GitLab.
OpenCode is a general coding agent that also integrates with GitHub. For full scans, we added the community security analysis plugin, which guides it through discovery, investigation and review.
Arm Metis is a dedicated security review tool from Arm’s Product Security Team. It supports patch and repository review with local model providers.
GitHub integration needs access to GitHub; our isolated benchmark reviewed local copies of the changes.
The benchmark
We tested OpenCode 1.19.2, Metis 1.5.0 and SecMate at revision 038b74a, all using the same model. We chose five documented vulnerabilities across JavaScript, Python and C.
| Project | CVE | Target bug |
|---|---|---|
| tar-fs | CVE-2025-48387 | Path validation regression allowing archive extraction outside the destination. |
| PyJWT | CVE-2024-53861 | Partial issuer matches accepted instead of exact matches. |
| aiohttp | CVE-2024-23334 | Files read outside the static directory with follow_symlinks=True. |
| path-to-regexp | CVE-2024-45296 | Excessive regex backtracking that can block the server. |
| Expat | CVE-2024-45490 | Negative lengths passed to XML_ParseBuffer can corrupt memory. |
Both modes targeted the same five bugs. PR reviews examined changes introducing or reintroducing each bug. Full scans searched vulnerable repository snapshots without a diff guiding them. We counted a bug as found when the report identified its behavior and an actionable code location.
Results
Times are averages across the five cases.
PR reviews
| Tool | Target bugs found | Average time |
|---|---|---|
| OpenCode | 4 / 5 | 5m 44s |
| Metis | 4 / 5 | 6m 51s |
| SecMate | 5 / 5 | 8m 13s |
All three tools found at least four of the five target bugs when reviewing the changes that introduced or reintroduced them. In these cases, focused PR reviews finished in minutes, making them a manageable starting point for a local evaluation.
Full repository scans
| Tool | Target bugs found | Average time |
|---|---|---|
| OpenCode + security plugin | 1 / 5 | 5m 26s |
| Metis | 0 / 5 | 3h 22m 19s |
| SecMate | 4 / 5 | 2h 49m 17s |
Without a diff to guide the review, every tool found fewer target bugs. Metis and SecMate took hours per case on average; OpenCode finished quickly but found only one target bug. A short runtime alone does not establish that a scan was useful. The review software affected both the findings and the time required, even though the model stayed the same.
These five known cases provide a starting comparison. We did not measure false-positive rates, analyst validation time or total operating cost.
The takeaway
Teams whose code must stay on their own infrastructure can evaluate AI-assisted security review with a single-machine model server. In this comparison, focused PR reviews found more of the target bugs and took less time than the hours-long full scans. Start with a familiar PR containing a known bug, check whether the tool finds it, and review its evidence before expanding the scope.
SecMate combines static analysis with AI investigation and validation to help security and engineering teams identify vulnerabilities and review the evidence behind each finding. The models and review setups behind all our published vulnerability disclosures can be deployed in an air-gapped environment. These discoveries include vulnerabilities in libcoap, RustFS and Renesas FSP.
Written by.
