Key Concepts
AI agents, Agenic AI, LLMs (Large Language Models), code generation, coding assistants, Windsurf, GitHub Copilot, Cursor, Apriel model, Kubernetes, Triton, snow skates, metadata, configuration, junior developers, productivity, training clusters, checkpoints, GPU clusters, multi-instance architecture, business rules, flow designer, prompt engineering, knowledge requirement, technology adoption, risk management, security, compliance.
Introduction
The episode of "The New Stack Agents" features Pat Casey, CTO of ServiceNow, discussing AI and agents, particularly focusing on the impact of AI on software development, the use of AI coding assistants within ServiceNow, and the company's approach to Agenic AI.
Pat Casey's Background and Early Days at ServiceNow
Pat Casey recounts joining ServiceNow as the second coder after Fred Luddy. He initially didn't fully grasp Luddy's vision of a platform enabling regular users to build business apps. Casey, skilled in traditional tools, questioned the limited configurability. However, he realized the power of empowering non-experts to create solutions themselves, which fueled early customer excitement and company growth.
Coding and the Dopamine Effect
Casey, now CTO, doesn't code as much but recalls the "dopamine hit" from solving complex problems through coding. He emphasizes the need for a customer or business reason to code, not just for intellectual pleasure. He contrasts this with the dopamine effect for AI model builders (recognition and impact) and feature developers (rapid iteration). He notes that prompt-level coding feels different, less about low-level control.
AI Coding Assistants at ServiceNow
ServiceNow is heavily investing in AI coding assistants but doesn't mandate their use. They initially used GitHub Copilot and then switched to Windsurf after a thorough evaluation. A study showed a 10% increase in stories completed per engineer after Windsurf training, aligning with Google's findings. Casey emphasizes that the tool is a means to an end: writing good code at a good rate. He believes modern tools, including AI, are essential for engineers.
Windsurf and Customer Adoption
Casey is somewhat unaware of the specific AI coding tools customers are using, but he hears about Cursor, Windsurf, and GitHub Copilot. ServiceNow customers often use the company's own AI tools within the ServiceNow development environment.
Resistance to AI Tools
While most engineers are technophiles, some resist AI tools, particularly when mandated. Casey compares this to historical resistance to tools like debuggers and IDEs. He acknowledges some developers prefer coding "with their own two hands," similar to those who still use celestial navigation.
Deployment and Management of AI at Scale
Some corporate entities express concerns about AI, particularly regarding information leakage when employees use tools like ChatGPT with company data. However, there's also pressure to increase productivity with AI. ServiceNow has seen a real productivity benefit, allowing them to get more done with the same people.
Overestimation of AI's Impact
Casey notes that initial estimates of AI's impact on productivity (30-40%) were higher than actual results with early tools (3-4%). Windsurf achieved around 10%. He believes engineers overestimate the proportion of their time spent actually coding, as corporate jobs involve meetings, training, and customer interaction.
Agenic AI and Autonomy
ServiceNow has an internal program called "Now on Now" where they use their own technology, including Agenic AI tools. One use case is question answering in the support portal, where AI directly answers questions instead of just providing links. Another is AI agents that research tickets, summarizing logs and third-party data to assist human agents. They aim to automate ticket closure for cases where the AI is highly confident in its findings.
Infrastructure for Running AI
ServiceNow, originally built on physical infrastructure, is transitioning to a hybrid model with 50% hyperscaler usage by 2030. For AI inference, they use servers with many GPUs. They have GPU hubs in North America, Europe, and Asia. They primarily use internally developed LLMs based on open-source models, but also use third-party LLMs like OpenAI and Claude in some cases, allowing customers to choose.
Apriel Model
The Apriel model, developed with Nvidia, is a small, fast 15 billion parameter model focused on complex reasoning. Casey emphasizes the trade-off between large foundation models (high accuracy but slow) and smaller, faster models (acceptable accuracy). The art is finding the right balance for specific applications.
Kubernetes and AI Stack
Casey states that deploying and running AI models hasn't been overly complex. ServiceNow uses a multi-instance architecture where each customer has a unique software stack. When an agentic action is needed, the ServiceNow instance communicates with a Kubernetes service (LLM router) that directs the request to the appropriate LLM (e.g., Apriel) running on Triton. Triton manages the GPUs.
Training Clusters and Checkpointing
For training, ServiceNow uses large GPU clusters with interconnects. They use a checkpointing and restart approach to handle GPU failures during long training runs.
Agenic AI on the Cluster
Agenic AI, particularly the Apriel model, is used for reasoning tasks within ServiceNow. Model density is limited by memory footprint. Larger models may span multiple GPUs, requiring larger hosts.
Zurich Release and VIP Coding
The Zurich release includes VIP coding, which provides AI assistance for developers building applications on the ServiceNow platform. The goal is to simplify the configuration and customization of ServiceNow using AI.
Configuration and its Evolution
Casey discusses the evolution of configuration from assembly code to vendor-specific languages to metadata-driven systems. While metadata simplified things, it also increased the breadth of knowledge required. They are now trying to reduce this knowledge requirement with AI.
Impact of LLMs on Hiring Interns and Junior Developers
ServiceNow still hires interns and junior developers, but potentially less than before. AI coding tools are particularly effective at tasks typically assigned to junior developers, potentially reducing demand in that sector. Casey advises students to consider their passions and the evolving landscape of the tech industry.
Conclusion
Pat Casey provides a detailed look into ServiceNow's approach to AI, from internal tooling to customer-facing features. He highlights the importance of productivity gains, the challenges of managing AI at scale, and the evolving role of developers in an AI-driven world. He remains optimistic about the long-term benefits of AI but acknowledges the need to navigate the disruptions it will bring.
AI summaries can miss context or contain errors. Check important details against the original video.
