Agentic Attackers, LLM Leaderboards, and Q-Day: A Conversation with F5 Labs

Key Concepts

• Cassie Leaderboard: A ranking system for Large Language Models (LLMs) based on security performance against individual attack prompts.
• ARS (Agentic Resistance Score): A security metric evaluating how well LLMs withstand attacks from autonomous agents rather than static prompts.
• Agentic AI: Autonomous systems capable of setting goals, using tools, and executing multi-step tasks to achieve an objective.
• PQC (Post-Quantum Cryptography): Cryptographic algorithms designed to be secure against the processing power of future quantum computers.
• Q-Day: The theoretical point in time when quantum computers become powerful enough to break current encryption standards (e.g., RSA-2048).
• Blind Sequence Projection: A technique used to extract data from models by manipulating input sequences.

1. LLM Security Research and Leaderboards

F5 Labs conducts extensive data science on attacker behavior to rank LLMs. The research highlights a critical trade-off between cost, speed, and security.

• Performance vs. Security: While models like ZAI/GLM 5 are highly cost-effective (offering three times the tokens of Claude Pro for half the price), they often rank poorly in security assessments.
• Methodology: F5 Labs uses proprietary methods to test models against various attack types, including "persona bullying" and "sugar-coated poison attacks," which mimic social engineering tactics applied at scale.
• Cassie vs. ARS:
  • Cassie: Focuses on individual prompts; currently contains over 130,000 prompts, with 10,000 added monthly.
  • ARS: Focuses on intent-based attacks. Autonomous agents are given a goal (e.g., data extraction) and a set of tools, then allowed to operate for a set time (e.g., 30 minutes) to attempt a breach. This is considered a more realistic simulation of modern threat actor behavior.

2. Real-World Applications and Threat Profiles

The transition from human-sent prompts to autonomous agent attacks is a significant shift in the cybersecurity landscape.

• Case Study: A major consultancy firm’s internal chatbot was compromised by an external researcher. The researcher used agentic AI to scan public APIs, identify keys, and execute a "blind sequence projection" attack to extract sensitive data.
• Scaling Attacks: Threat actors are increasingly using agentic AI not just to attack models, but to scale traditional cybersecurity attacks, making them more efficient and harder to detect.

3. Post-Quantum Cryptography (PQC) and Q-Day

F5 Labs is actively monitoring the global transition to quantum-resistant encryption.

• Current Adoption: Research indicates that only 8.5% of the top 1 million websites have adopted PQC.
• The Threat of Q-Day: Recent academic research suggests that the number of logical qubits required to break RSA-2048 encryption has been reduced by a factor of 10. This implies that "Q-Day" may arrive significantly sooner than the previously estimated 2030 timeline.
• Future Research: F5 Labs plans to update its scans to track how organizations are responding to these accelerated quantum threats.

4. Synthesis and Takeaways

The discussion emphasizes that as AI models become more integrated into business infrastructure, the security paradigm must shift from static prompt filtering to agentic defense.

• Actionable Insight: Organizations should prioritize security metrics like the ARS when selecting models, as cost-efficiency often masks vulnerabilities to autonomous, intent-driven attacks.
• Resource Availability: F5 Labs provides ongoing intelligence through their website (f5.com/labs), a monthly research newsletter, and a "Weekly Threat Bulletin" that utilizes autonomous agents to summarize the top five global threats.

Notable Quote: "It’s a really fascinating world we’re in at the moment where you’ve got traditional cybersecurity completely developing and kind of overlapping with LLM and AI security." — Dave Wbertson, Director of Threat Research at F5 Labs.