CroissanthologyLLM cybersecurity risks in the long run are probably exaggerated

LLM cybersecurity risks in the long run are probably exaggerated

18 Feb 2026 also on Substack

Defense disproportionately benefits from capability advances

Are LLMs-in-cybersecurity defense or offense dominant? Opus 4.6 saturated Anthropic's cybersecurity benchmarks,¹ and Claude Code has been used for successful cyberattacks at least once. To develop context windows which jailbreak Opus 4.6 is not terribly difficult and if you do it once you can do it forever because jailbreaking is automatable. In 2026, the amount of apps and websites which are going to release with poor security due to being vibe-coded will be through the roof. LLMs are relatively cheap, and extremely scalable. Classifiers are just as easy to jailbreak as the main models themselves, such that Anthropic intervening whenever an API flags something as hacky is not going to work.

1From the system card:

1.2.4.3, On cyber risks
The RSP does not define a formal capability threshold for cyber risks at any AI Safety Level. However, Claude Opus 4.6 has saturated all of our current cyber evaluations, achieving ~100% on Cybench (pass@30) and 66% on CyberGym (pass@1). Internal testing demonstrated qualitative capabilities beyond what these evaluations capture, including signs of capabilities we expected to appear further in the future and that previous models have been unable to demonstrate. The saturation of our evaluation infrastructure means we can no longer use current benchmarks to track capability progression or provide meaningful signals for future models. We are prioritizing investment in harder evaluations and enhanced monitoring for cyber misuse, even in the absence of formal RSP thresholds.

But it's 2026 AD! The first, freebie step in any problem is “well have you tried describing the problem to a model?”.

We now have tools like Shannon, which take advantage of the information asymmetry between defense (where you can evaluate your entire code base) and offense (where you're stumbling around in the dark). As with every agent harness, it will take very little time for the eye of Sauron of the AI labs to lift their gaze to Shannon and replicate it. [UPDATE: seems to've taken 3 days as of writing this post.] When might the Claude Code system prompt direct Claude to do exactly as Shannon does? “Vibe-coded apps are far more vulnerable than professionally hand-coded ones” doesn't have to be a forever rule. Until now, pentesting a codebase cost thousands and would only happen every ~year or so because of how much of a big deal it felt. Now you could run ~Shannon every time you commit for a handful of dollars. It seems highly LLMs will make hacking into a black box a little bit easier and make protecting a white box WAY easier.

As models become more powerful, defense gains more and more of an advantage. A formally verifiable system cannot by definition be hacked. Eventually, once cybersecurity becomes a solved game at a given level of codebase size, defense will permanently win. 50 bucks and a few hours to make yourself invulnerable—Shannon's current cost—baked into the very model harness you're using to make your software is so trivial.

Depending on how fast AI progress is and how steerable the models we create will be, it's likely classic cybersecurity won't be a problem for legitimately ironclad reasons soon enough.

This DOES NOT include “five dollar wrench” solutions like “trick a human into letting us in". Those will predictably become the new bottlenecks for “cyber”attacks, and it's plausible that in this arena offense is dominant.