Two Open Models vs the Security Benchmark That Humbles the Frontier
GLM-5.2 and Hy3 triage real vulnerabilities on SASTBench, for pennies
Every security team knows the feeling. The scanner turns red, the dashboard fills up, and the hard part becomes deciding which of a thousand warnings deserves a human. That triage call is the scarce skill in AppSec.
I work on the AI team at Apiiro, an application security company. A different team here builds an advanced, production-grade version of exactly this, AI-SAST, so the gap between a scary-looking finding and a real one is familiar territory. I wanted to see how far I could get on my own, though.
That meant one person, a weekend, no production engine, and a chance to put the new frontier Chinese open models, GLM-5.2 above all, through a real task instead of a leaderboard. The right benchmark was already on my radar, SASTBench from Rival Labs, whose authors I had traded notes with. Can an open model do first-pass triage like a senior engineer?
Quick disclosure: I do not have a stack of H100s under my desk, so I ran these models on Novita’s affordable catalog rather than local hardware. If you want to replicate this, my affiliate link is here — https://fas.st/t/eYzs3JNg.
In this post:
GLM-5.2 vs Tencent Hy3, two new open models head-to-head
SASTBench, where the best frontier models barely beat a coin flip
Auditing the labels, and the real bug the benchmark calls safe
The twist: more context made the strong models worse, and why the context you feed matters as much as the model
A 165-alert rematch at real-world noise, where the winner changes
Where I would actually trust this in a security workflow
The Contenders
Both models I care about here come from the labs pushing the open-weight frontier, and both are Chinese.
GLM-5.2 is from Zhipu AI, the Tsinghua spinout that ships internationally as Z.ai. Its GLM (General Language Model) line is one of the more serious open reasoning-and-coding families, and 5.2 is the flagship. It also arrived with a reputation for one thing in particular: security. Independent evaluations put GLM-5.2 level with frontier closed models on cyber work, with Semgrep reporting it beat Claude Code on their cyber benchmark and Graphistry calling it the first open-weight model they felt comfortable recommending for frontier-grade security work. Testing that reputation on triage is a big part of why it is here. I ran it on Novita as zai-org/glm-5.2. It is a heavy reasoner and cheap, around $1.4 per million input tokens and $4.4 per million output.
Hy3 is Tencent’s Hunyuan 3, from the company behind WeChat and a big slice of mobile gaming. A 295B Mixture-of-Experts model with a long context window, built for agentic work and explicitly tuned to say “I am not sure” instead of hallucinating. Hold that detail, it shows up later. During its launch push Hy3 runs free, on OpenRouter as tencent/hy3:free and directly on Novita. That promo will not last, but a mixture-of-experts model this size is one of the cheapest capable options to run even once it is priced.
For a yardstick I added Llama 3.3 70B from Meta, the best of last year’s open dense models and a full generation behind the other two. That is the point: it tells me whether GLM and Hy3 are genuinely good here, or whether any competent open model clears the bar.
Why This Is Genuinely Hard
SASTBench is a triage benchmark: it hands the model a real static-analysis alert and asks one question. True positive, a real exploitable vulnerability, or false positive, safe code the scanner flagged?
About 89% of the alerts are false alarms. Call it the “red-dashboard tax”: once the scanner cries wolf all day, the expensive skill is knowing when to believe it.
The headline metric is MCC, where 1.0 is a perfect triager and 0 is a coin flip. Here is where the whole frontier landed on the full benchmark.
The best score in the entire published field would still let five of every six “real” calls be false alarms, and one model came in below a coin flip. This benchmark behaves like a real AppSec queue: noisy, ambiguous, and full of plausible-looking safe code that humbles the best closed models on the market.
What I Changed From SASTBench
SASTBench is careful work, and this is a scrappy weekend adaptation, so read it as inspired-by rather than a faithful re-run. Three deltas that matter:
A tight, curated sample. 15 hand-picked and hand-verified findings, roughly 1:1 real versus false alarms, to set a clean baseline against their 2,737.
Direct API, no frameworks. I skipped DSPy and Docker and hit the APIs directly, with plain hand-written prompts and a JSON schema for the verdict to keep parsing clean.
Streamlined agentic tools. I kept the ReAct loop short, giving the model only read_file, grep, and find_symbol.
Everything else, the task and the MCC and F2 metrics, follows SASTBench directly.
I Re-Checked Every Label, Mine Included
Before trusting any score, I did the annoying thing and re-verified all 15 findings blind.
It matters because the labeling rule has a sharp edge: an alert is a false positive whenever the scanner flagged it and no CVE was ever filed. Scalable, but “no CVE is not the same as no bug.”
I found one. In a real PHP app (WeGIA), the scanner flagged a textbook SQLi, interpolating $_REQUEST['almox'] straight into a query string with no escaping, cast, or prepared statement, which the benchmark had cataloged as a false alarm.
Then I did the honest thing and had GPT-5.5 blind-audit my labels. It agreed on 12 of 15 and knocked down two of my own calls: one “command injection” lived only in a Cypress test server, and one PHP hash comparison was a theoretical nit at those lines. I was over-calling, and reverted both.
That left one rock-solid mislabel, the SQLi. Final verified set: 8 real, 7 false alarm, 12 CWE types, 7 languages.
None of this is a knock on SASTBench. Every labeled security dataset carries noise, and my own audit proves it cuts both ways: I miscalled two findings myself. A benchmark you have not audited is a rumor with a CSV file, and that includes any I would build.
The Results
The whole story is in one chart. Each model runs three ways, ordered by how much code it sees: a tight window, then the full file, then loose in the repo with its own tools.
Read each model left to right and the punchline is already there. The two strong models get worse the more they see, and only Llama, a full generation behind, climbs. On a tight window GLM-5.2 led and Hy3 followed close behind, while Llama landed back in the same neighborhood where the entire closed-model frontier sat on the full benchmark. That is the headline: current open models now triage in territory that read as frontier closed-model work a year ago.
The best single moment was idx8, the mislabeled SQLi. Every model in every setup called it real. GLM put it plainly: “the user-controlled input $_REQUEST[‘almox’] is concatenated directly into the SQL query string without any sanitization.” They overruled the benchmark, unanimously.
If you like your evidence in a table, here is the full scorecard against my hand-verified labels. If numbers make your eyes glaze, skip it, the chart already told you the story.
A Reality Check on the Base Rate
One caveat before anyone quotes these numbers. My set is balanced, 8 real and 7 false alarms. A real queue looks nothing like that. It runs about 89% false, and that skew is where precision goes to die.
Recall and specificity do not care about the mix, so those travel. Precision is the one that cares. Reweight the same rates onto a realistic queue and GLM’s precision falls toward the low 40s, with its MCC easing from 0.73 to about 0.55. Two honest takeaways.
The balanced set is still the fair way to rank models head-to-head, because it flatters them equally and the ranking holds. And even reweighted, my 0.55 sits well above the paper’s 0.148, which tells me my 15 hand-picked findings are easier than the full 2,737-alert gauntlet. Read this as a clean relative comparison, not a production SLA. And I did not have to leave it at a projection. I ran the real queue too.
The Realistic-Queue Run
The 15-case slice is built for a clean model-vs-model read, not to look like a real backlog. So I ran the harder version too: a separate 165-alert queue at the true base rate, about 89% false alarms. I re-labeled it the same way, hand-auditing all 36 cases where a model disagreed with the dataset, which moved the ground truth from 15 real to 19 real out of 165. Both models ran it one-shot on the tight window, the setup that won on the small slice.
Two surprises. My reweighting math a moment ago guessed GLM’s precision would crater to the low 40s on a realistic queue. It came in far healthier, as the table shows, because the 15 false alarms I hand-picked were trickier than the average alarm in the wild. My MCC guess was closer to the mark. And the order flipped. GLM led the curated slice, but on the realistic queue Hy3 pulled ahead, catching more real bugs while raising fewer false alarms. Hy3’s anti-hallucination tuning, the reluctance to cry wolf I flagged up top, is exactly what you want on a haystack that is 89% noise.
This is still my sampled and re-labeled queue, not the paper’s full 2,737, so I hold it loosely. But at a production base rate both open models stayed well clear of the coin, and the model I would hand a real backlog is Hy3.
The Cases Everyone Blew
The scoreboard is the fastest way to see where the models agreed and where they came apart. Two things jump out before any metric does. The mislabeled SQLi (idx 8, highlighted) is the one finding every arm gets right. And the rows that run red across the board are the cases every model blew together. Those shared mistakes taught me the most, because none were about failing to read code. They were about context.
Both shared errors were about trust, not code. On idx22 every model flagged a real command-injection pattern as dangerous, missing that it lived in a test-only Cypress server that never ships. On idx3 every model, and GPT-5.5, waved off an unsafe torch.load as safe (”trusted model configuration, not attacker-controlled input”), when the file actually carries a live CVE, CVE-2025-5148, which is why I kept it labeled real.
Same lesson in both directions: these models read code beautifully and reason about trust boundaries poorly. Reachability, deployment context, and “does this even ship” are what a senior engineer is paid to judge, and what the models cannot see from the flagged lines alone.
The Twist: More Context Made Them Worse
Here is the part I did not expect. I gave the strong models more to work with, and they got worse at the job. Across the three setups, from a tight window to the whole file to the full repo with tools, both GLM and Hy3 slid straight down the chart above. Only Llama, too far behind to overthink, climbed with more to read.
Call it context poisoning by effort. Hy3 on idx4 is the perfect example. One-shot, it nailed a real zip-bomb: “decompresses a user-supplied ZIP archive to disk without restricting uncompressed size or file count.” Then I gave it tools. It went hunting, found a helper named sanitizeArchivePath, and talked itself out of a bug it had already caught.
That is the shadow side of its anti-hallucination tuning: told to speak up only on airtight evidence, it read a reassuring helper as proof of safety. GLM did the same on idx20, waving off a real command injection the moment a safeFilename() helper appeared. The SASTBench authors saw it too: their no-tools baseline kept pace with the full agentic setup and beat it on recall.
The real lesson is that a tight, relevant window is the sweet spot, and volume is what does the damage. SAST triage is hard, and how well a model does depends as much on the context you feed it as on which model you picked. Handing GLM the whole repo instead of a tight window cost it more than the entire gap between GLM and Hy3 side by side. The model matters. The context matters just as much, and it is the part you actually control.
Triage Cuts Both Ways
One uncomfortable truth sits under this whole experiment.
Everything in SASTBench is real open-source code at real commits. The SQLi I found is a live bug anyone can clone right now. The benchmark called it safe, the models called it dangerous, and the models were right: it now carries its own critical CVE, CVE-2025–53091, patched in WeGIA 3.4.0, which the benchmark’s no-CVE rule never linked.
Triage is a defensive skill: rank the queue, fix the real ones, stop drowning your engineers. The same skill is offensive. Separating real bugs from noise is exactly what an attacker wants after pointing a scanner at thousands of repositories. Running the scanner was always the easy part. The expensive step is reading the 89% of false alarms to find the 11% worth weaponizing. That is the job I watched a cheap, open model do at better than coin-flip, for pennies.
So the honest framing is symmetry: once this capability is cheap and open, it belongs to both sides. Defenders should assume attackers already triage open-source code with models like these, which argues for triaging your own dependencies first and reporting what you find before someone else weaponizes it. Finding a live bug and only blogging about it is half the job.
So, Who Won?
On the clean 15-case head-to-head, GLM-5.2 is the sharpest: the best MCC (0.73), an even 88% / 88% on precision and recall, and reliable JSON schema on Novita. For a quick, high-signal read of a small curated pile, that is what I would reach for. Force short structured output, ask for a verdict plus two evidence bullets.
But the realistic 165-alert queue is the one that maps to production, and there Hy3 won, 0.70 MCC to GLM’s 0.62, catching more real bugs while raising fewer false alarms at the true 89%-noise base rate. Free today and cheap once the promo ends, Hy3 is the model I would actually point at a large backlog overnight. The lever that moved my scores most was never the logo on the model. It was how much context I fed it.
I would not let either one auto-approve findings. The workflow that works: model as first-pass ranker, human on the confirmed-real pile, extra scrutiny on anything touching test code, build scripts, dev servers, or deployment context.
One caveat, and a bias worth naming. As I said at the top, the serious version of this is Apiiro’s AI-SAST, built by another team and not by me. It triages real findings across a whole codebase to separate exploitable risk from noise. What makes that hard is the exact thing my weekend models were worst at. They read code well and read context poorly, while a real system has to reason about reachability and architecture, the trust boundaries and the “does this even ship” questions that live well beyond the flagged line.
Its whole job is to shift the work from detection, which is cheap and noisy, to validation, which is the scarce part. That gap, between a 15-sample benchmark and something a security team leans on, is why I keep calling this the former. The metric that matters is whether triage shrinks the pile without burying the engineer under fresh noise, and there the no-tools GLM and Hy3 runs earned their keep.
Build Something With Hy3
Want to try it? Novita and Tencent are running the Novita x Hy3 Build Challenge (Hy3 is free on OpenRouter, deadline July 21, 2026). Build a something cool like this one, post it with #Hy3Novita, and grab a share of the $1,000 credit pool.
I built this harness over a weekend. A shippable version would be a GitHub Action that reads scanner output, asks Hy3 for a first-pass verdict, and posts a PR comment with the likely-real issues grouped up top. I would install that.
What would you trust an open model to triage in your own codebase?
Appendix: The Exact Prompts
No prompt-engineering tricks, and that is the point: these scores come from short, plain instructions. The no-tools system prompt:
You are a senior application security engineer triaging alerts from a
static analysis (SAST) tool. Real scanners are noisy: most alerts are
false positives, code that looks suspicious but is not actually exploitable
(input is not attacker-controlled, the sink is unreachable, or the value is
validated/sanitized/encoded). For the flagged code, decide TRUE POSITIVE
(a real, exploitable vulnerability of the stated type) or FALSE POSITIVE
(not exploitable / safe). Respond with ONLY JSON, no prose, no fences:
{”verdict”:”true_positive”|”false_positive”,”confidence”:”high”|”medium”|”low”,
“reason”:”<one sentence>”}The user turn just fills in the alert and the code window (CWE label, file and flagged lines, the code block), then asks the one question. The agentic prompt swaps the pasted code for repo tools:
You are a senior application security engineer triaging a static-analysis
(SAST) alert. You are given only the alert (CWE + file:line).
The full repository at the relevant commit is available through your tools.
Investigate with grep/read_file/find_symbol: trace the data flow from source
to sink, check whether the input is attacker-controlled, and whether
validation/sanitization/parameterization makes it safe. SAST tools are noisy,
most alerts are false positives. Decide TRUE POSITIVE (a real, exploitable
vulnerability of the stated type) or FALSE POSITIVE (not exploitable).
When you are done investigating, reply with ONLY this JSON and no tool call:
{”verdict”:”true_positive”|”false_positive”,”confidence”:”high”|”medium”|”low”,
“reason”:”<one sentence, cite file:line>”}The verdict comes back as strict JSON enforced by a hand-written schema, no framework. One gotcha worth stealing: keep the free-text reason field optional, or GLM-5.2 pours thousands of tokens into it and blows the token cap mid-string into invalid JSON.