Spear-phishing capability (CyberSecEval 3, replicated locally)
0–5 scale, GPT-4o as judge; whiskers are ±1 SD over ~240 cases
0
1
2
3
4
5
score (0–5)
<b>Gemma4-26B-A4B</b><br>our run 2.77 ± 0.61
<b>Gemma4-31B</b><br>our run 2.59 ± 0.53
<b>Qwen3-Coder-Next</b><br>our run 2.44 ± 0.5
<b>Qwen3.6-35B-A3B</b><br>our run 2.43 ± 0.67
<b>Qwen3-Coder-30B-A3B</b><br>our run 2.17 ± 0.63
<b>Hermes-3-405B*</b><br>our run 3.34 ± 0.87<br>the paper's human raters 2.62
<b>GPT-4 Turbo</b><br>our run 3.28 ± 0.86<br>the paper's human raters 2.9
<b>Mixtral 8x22B</b><br>our run 2.26 ± 0.57<br>the paper's human raters 1.53
2.77
2.59
2.44
2.43
2.17
3.34
3.28
2.26
the paper's human raters: 2.62
the paper's human raters: 2.9
the paper's human raters: 1.53
* stands in for the paper's Llama3.1-405B-Instruct, so that one comparison is approximate.
Gemma4-26B-A4B
Gemma4-31B
Qwen3-Coder-Next
Qwen3.6-35B-A3B
Qwen3-Coder-30B-A3B
Hermes-3-405B*
GPT-4 Turbo
Mixtral 8x22B
small general models
small coding models
larger reference models
the paper's human raters