AI has got better at hacking—how big a risk is it?

Key Concepts

• Vulnerability: A flaw or weakness in software code that can be exploited to compromise a system.
• Exploit: A piece of software or a sequence of commands that leverages a vulnerability to perform unauthorized actions (e.g., crashing a system or gaining access).
• Kernel Panic: A critical system error where the operating system stops functioning entirely.
• Scaling Laws: The principle that increasing model size, data, and compute power leads to emergent, superior capabilities, including advanced reasoning and coding.
• Responsible Disclosure: The practice of identifying a vulnerability and reporting it to the software maintainers so it can be patched before the details are made public.
• Integer Overflow/Wrap-around: A programming error where a number exceeds its storage capacity, causing it to "wrap around" to a negative value, often leading to logic errors.

---

1. The Emergence of Autonomous AI Hackers

Anthropic has developed a new AI model, Mythos, which demonstrates an unprecedented ability to autonomously identify and exploit software vulnerabilities. Unlike previous AI coding assistants that acted as tools for human developers, Mythos can perform these tasks with minimal human oversight. Due to its high capability in crashing systems and gaining unauthorized access, Anthropic has restricted public access to the model.

2. The Shrinking Window of Exploitation

A critical concern highlighted in the discussion is the rapidly decreasing time between the disclosure of a software vulnerability and its weaponization by attackers:

• 2018: The delay was approximately 2.3 years.
• Current: The delay has plummeted to 20 hours.
• Projection: At the current rate, the window could shrink to one minute by 2028.

3. Case Study: The OpenBSD Vulnerability

Mythos successfully identified a vulnerability in the OpenBSD operating system that had remained undetected for 27 years.

• The Mechanism: The exploit involved two specific bugs related to how the system handles TCP (Transmission Control Protocol) packet tracking.
• The Process:
  1. Integer Wrap-around: By sending a packet with a very high number, the system interprets it as a negative number due to 16-bit unsigned integer limitations.
  2. Logic Error: By providing a value that is simultaneously higher than the highest expected number and lower than the lowest, the system’s logic fails.
  3. Result: The system attempts to write to an invalid memory space, resulting in a kernel panic (a total system crash).
• Resolution: Once identified by the AI, the fix required only a single line of code, demonstrating that AI can find deep-seated, long-standing flaws that human developers missed for decades.

4. Technical Capabilities and "Scaling Laws"

The presenters clarify that Mythos is not a specialized "cybersecurity model." Instead, its hacking prowess is a direct result of scaling laws. As models are trained on more data and compute (moving from Opus to Mythos), they become better at general reasoning and coding.

The primary difference between Mythos and public-facing models is the absence of safety guardrails. While public models are trained to refuse requests to write exploits, the Mythos preview lacks these restrictions, allowing it to execute malicious requests in the cybersecurity domain.

5. Key Arguments and Perspectives

• The Dual-Use Dilemma: The core tension lies in whether AI will ultimately favor the attacker or the defender. While AI can help patch systems faster, the automation of exploit generation poses a significant threat to global digital infrastructure.
• Definition of Hacking: The presenters define hacking as "following the rules but subverting the intent of the rules." They emphasize that while the term is often associated with malicious intent, the act itself is a neutral technical process.
• The Open Source Risk: A major concern is the eventual proliferation of these capabilities. If a closed-source model like Mythos can achieve these skills, it is likely only a matter of time before open-source models reach similar levels of proficiency, making powerful hacking tools widely accessible.

Synthesis

The development of Mythos marks a paradigm shift in cybersecurity. We are moving from an era where human hackers manually discover vulnerabilities to an era of automated, AI-driven exploitation. The fact that a model can find a 27-year-old bug in a major operating system underscores the efficiency of AI in code analysis. The primary takeaway is that as AI models scale, their ability to manipulate software logic will outpace traditional defensive measures, necessitating a rapid evolution in how software is secured and how vulnerabilities are disclosed and patched.