How Mercor caught the eval wave

Lenny's PodcastAbout 2 min readSep 21, 2025Watch original
THE SUMMARYAI-generated

Key Concepts:

  • XAI (Presumably a company focused on AI)
  • Quality of Experts (Emphasis on skilled individuals for AI model improvement)
  • Crowdsourcing (Utilizing a large group of people for tasks)
  • Incumbents (Existing players in the crowdsourcing market)
  • Dignity of Experts (Respectful treatment and fair compensation of talent)
  • Direct Collaboration with Labs (Working directly with AI research and development teams)

Meeting with XAI and Market Shift

The speaker recounts how a customer facilitated an introduction to the co-founders of XAI. Within two days, they were invited to the Tesla office to meet the XAI co-founding team (excluding Elon). The XAI team expressed strong enthusiasm for the speaker's company's focus on the "quality of the experts." This meeting signaled a radical shift in the market, prompting the speaker's company to position itself at the forefront of this change.

Challenges with Crowdsourcing and Incumbents

A few months later, a crowdsourcing company utilized the speaker's platform to hire over a thousand individuals. This led to a surge in support tickets related to payment issues for these workers. This incident highlighted a critical flaw: many established crowdsourcing platforms ("incumbents") were not adequately addressing the needs and experiences of the talent they were utilizing to improve AI models. The speaker suggests that these incumbents were "resting on their laurels" and not innovating to provide better experiences for their talent.

Direct Collaboration and Ethical Approach

The speaker's company identified an opportunity to collaborate directly with AI labs. This approach prioritized the "dignity of the experts" by ensuring they were "paid extremely well." The strategy involved "cutting out the middlemen," presumably referring to traditional crowdsourcing platforms that may take a significant cut of the compensation. This direct collaboration model, focused on fair treatment and compensation, ultimately led to the company's subsequent success.

Synthesis/Conclusion:

The speaker's company recognized a critical need for higher-quality talent and ethical treatment within the AI model improvement landscape. Their direct engagement with XAI validated this perspective, and the subsequent challenges faced by a crowdsourcing client using their platform further solidified the need for a new approach. By focusing on direct collaboration with AI labs, prioritizing the dignity and fair compensation of experts, and bypassing traditional crowdsourcing models, the company positioned itself for success in a rapidly evolving market. The key takeaway is the importance of valuing and fairly compensating skilled individuals in the AI development process, and the potential for direct collaboration to disrupt traditional crowdsourcing models.

AI summaries can miss context or contain errors. Check important details against the original video.

MAKE IT YOURS

Read. Remember. Reuse.

Free tools

Go a little deeper.

Have a question about this video? Load its transcript to open the video chat.