How to Hire AI Engineers when EVERYONE is cheating with AI — Beth Glenfield, DevDay

Key Concepts

AI in technical recruitment, AI cheating services, LeetCode's diminishing relevance, AI coding assistants, productivity boost through AI, competition beyond compensation, evaluating collaboration with AI, workplace simulations, AI agents with diverse personalities, measuring collaboration skills, cost of bad hires, evolving engineering roles.

The Broken Technical Recruitment Process in the Age of AI

Beth Glenfield, speaking from an Irish perspective, argues that AI is fundamentally changing and breaking the traditional technical hiring process. She highlights several key issues:

• AI Cheating Services: Services like Clu, an AI cheating service founded by an ex-Columbia student, are gaining traction (heading towards $1 million ARR), enabling candidates to bypass traditional coding challenges.
• LeetCode Inadequacy: LeetCode success rates are high (e.g., 93% for Google/Meta interviews), but Glenfield argues that excelling at LeetCode doesn't guarantee the ability to build great AI products. LeetCode is not indexing for creative problem solvers, collaborative leaders, and those who can work with AI.
• AI Coding Assistants: The prevalence of AI coding assistants (used in 1 in 3 interviews) means that companies are often evaluating the candidate's ability to use AI tools rather than their core coding skills. Sam Altman's statement emphasizes the importance of learning to use AI tools effectively.
• Productivity Boost Claims: Mark Benioff's announcement that Salesforce is halting software engineer hiring due to a 30% productivity boost from AI raises questions about the changing nature of software engineering roles. Glenfield expresses a desire to see the data supporting this claim.
• Competition Beyond Compensation: Candidates are now prioritizing brand recognition, career prestige, and perceived stability, especially given recent layoffs. Startups struggle to compete with larger companies like Google in attracting top AI talent.

Shifting Focus: Evaluating Collaboration and Real-World Skills

Glenfield proposes a shift in how technical candidates are evaluated:

• From Puzzles to Business Scenarios: Instead of asking candidates to solve abstract puzzles, companies should observe how they collaborate with AI teammates on real-world business scenarios.
• Measuring Collaboration Skills: The focus should shift from memorizing algorithms to assessing skills like task delegation, handling ambiguity, and adapting to changing requirements.
• Demonstrating Engineering Culture: Companies can use the hiring process to showcase their engineering culture and assess how candidates would perform in their specific environment.

DevDay's Approach: Workplace Simulations with AI Agents

DevDay is reimagining technical hiring by using workplace simulations:

• Real-Life Simulations: Candidates work alongside AI agents with diverse personalities (e.g., perfectionist, pragmatist, security expert, junior developer).
• Business-Specific Features: Candidates build features specifically for the company's business domain, rather than solving contrived problems.
• Measuring Key Skills: The focus is on measuring collaboration with AI, handling ambiguity, communicating technical decisions (in pull requests, comments, tickets), mentoring others, and adapting to change.

The Cost of Bad Hires and the Evolving Engineering Role

Glenfield emphasizes the financial impact of bad hires, particularly for companies that cannot afford to make mistakes. A bad hire can cost a company 20-60K. She also highlights the evolving nature of engineering roles:

• AI Handling Mid-Level Tasks: Mark Zuckerberg predicts that AI will handle mid-level engineering work soon.
• Shift from Entry-Level to Advanced Roles: While entry-level engineering jobs may be declining, the demand for engineers with creativity, collaboration skills, and business judgment is increasing.
• Working with AI, Not Being Replaced: The key is to find engineers who can work effectively with AI, rather than being replaced by it.

Conclusion

The traditional technical hiring process is broken in the age of AI. Companies need to move beyond LeetCode and focus on evaluating candidates' ability to collaborate with AI, handle ambiguity, and apply their skills to real-world business scenarios. DevDay's approach, using workplace simulations with AI agents, offers a potential solution for identifying and hiring the right talent for the future of engineering. Glenfield encourages interested parties to reach out to discuss these ideas further.