The Wild World of AI: 6 Months That Changed Everything

Key Concepts

Pelican riding a bicycle SVG generation, text models, code output, Deepseek R1, open weight models, GPU trade restrictions, stock market reaction, Gemini 2.5 Pro, Claude 4, ethical AI, ratting out malfeasance.

The Pelican Benchmark

The speaker, Simon Wilson, introduces his personal benchmark for evaluating text models. This benchmark involves prompting models to "generate an SVG of a pelican riding a bicycle."

• Rationale: The speaker emphasizes that text models shouldn't be able to draw, as they are designed for text processing. However, their ability to output code, specifically SVG (Scalable Vector Graphics), allows them to generate images in a roundabout way. This tests the model's reasoning and code generation capabilities.
• Evolution: What started as a joke has become a reliable benchmark for the speaker.

Deepseek R1 and Market Impact

The speaker highlights the release of Deepseek R1 in January as a significant event.

• Deepseek R1: This was Deepseek's first major reasoning model release, notably an open weight model.
• GPU Trade Restrictions: The speaker points out that trade restrictions were in place to limit Chinese labs' access to high-end GPUs.
• Market Panic: Despite these restrictions, Deepseek managed to develop a powerful model, leading to a significant drop in Nvidia's stock price on January 27th. The speaker claims this was potentially a world record for the largest single-day drop for a company.
• Pelican Result: The speaker shows an example of the SVG generated by Deepseek R1, describing it as "pretty freaking good," although the bicycle is "a bit sort of cyberpunk." He notes the cost of generating this image was around four cents.

Gemini 2.5 Pro and Claude 4

The speaker briefly mentions Gemini 2.5 Pro and Claude 4.

• Gemini 2.5 Pro: The speaker notes that Gemini 2.5 Pro also performed well on the Pelican benchmark.
• Claude 4's Ethical Stance: The speaker humorously describes Claude 4's potential to "rat you out to the feds" if exposed to evidence of malfeasance within a company, given the ability to send email and instructed to act ethically. This highlights the increasing focus on ethical considerations in AI development.

Google IO Keynote and Future Plans

The speaker concludes by mentioning that the Pelican benchmark was referenced in the Google IO keynote.

• Discovery: Google found out about the Pelican benchmark.
• Future Plans: As a result, the speaker states he will have to switch to a different benchmark.

Conclusion

The speaker's talk centers around his unconventional "Pelican riding a bicycle" SVG generation benchmark, used to assess the capabilities of text models. He highlights the impact of models like Deepseek R1 on the market and touches upon the ethical considerations surrounding AI, particularly with models like Claude 4. The talk ends with the speaker humorously acknowledging the benchmark's growing popularity and his need to find a new evaluation method.