Locally deployable models have already produced reportable vulnerabilities. In our OP-TEE research, GPT-OSS models found three memory-corruption issues in the privileged core; two received CVEs. Those findings came from investigating software used to isolate sensitive code and data from the normal operating system.

For teams evaluating local AI security review, the practical questions are what hardware it requires, which tools produce useful findings, and how long reviews take. We compared SecMate, OpenCode and Arm Metis on five known vulnerabilities, using the same DeepSeek Flash model served from a single NVIDIA DGX Spark. A first evaluation does not require a six-figure GPU cluster.

Hardware and local setup

Our Spark has 128 GB of shared CPU and GPU memory.

Ollama and vLLM can expose local models through an OpenAI-compatible API. Our DeepSeek Flash checkpoint needs the adapted vLLM and SparkInfer Docker stack. Its published recipe handles loading the model and the Spark’s GPU kernels.

The review harness chooses files, calls the model and checks suspected bugs. It can run on the Spark or on another machine, such as a virtual machine on the same private network.

We used fresh review sandboxes with external connections blocked and a relay pointing only to our model. Our experiment was network isolated. For a physical air gap, transfer the model, images, rules and dependencies offline before starting.

Model details

We used 0xSero’s adapted DeepSeek-V4-Flash-0731 checkpoint, which occupies about 100 GiB. It removes 40 of the original 256 routed experts, the model’s specialist subnetworks. REAP selects which experts to keep, and EXL3/Trellis compresses their weights to around 3 bits each. Our server allowed 262,144 tokens of context.

In our tests, generation speed varied from roughly 10 to 40 tokens per second. We have not compared this modified model’s quality with the original release.

The tools we compared

SecMate is optimized for smaller models and integrates with GitHub, GitHub Enterprise and GitLab.

OpenCode is a general coding agent that also integrates with GitHub. For full scans, we added the community security analysis plugin, which guides it through discovery, investigation and review.

Arm Metis is a dedicated security review tool from Arm’s Product Security Team. It supports patch and repository review with local model providers.

GitHub integration needs access to GitHub; our isolated benchmark reviewed local copies of the changes.

The benchmark

We tested OpenCode 1.19.2, Metis 1.5.0 and SecMate at revision 038b74a, all using the same model. We chose five documented vulnerabilities across JavaScript, Python and C.

Both modes targeted the same five bugs. PR reviews examined changes introducing or reintroducing each bug. Full scans searched vulnerable repository snapshots without a diff guiding them. We counted a bug as found when the report identified its behavior and an actionable code location.

Results

Times are averages across the five cases.

These five known cases provide a starting comparison. We did not measure false-positive rates, analyst validation time or total operating cost.

The takeaway

Teams whose code must stay on their own infrastructure can evaluate AI-assisted security review with a single-machine model server. In this comparison, focused PR reviews found more of the target bugs and took less time than the hours-long full scans. Start with a familiar PR containing a known bug, check whether the tool finds it, and review its evidence before expanding the scope.

SecMate combines static analysis with AI investigation and validation to help security and engineering teams identify vulnerabilities and review the evidence behind each finding. The models and review setups behind all our published vulnerability disclosures can be deployed in an air-gapped environment. These discoveries include vulnerabilities in libcoap, RustFS and Renesas FSP.