TNS Agents Livestream: Pat Casey, ServiceNow

Key Concepts

• AI and Agentic Technologies: Focus on AI-powered tools and autonomous agents for code development and customer support.
• Generative AI for VIP Coding: Using AI to assist developers in creating applications within the Service Now platform.
• LLMs (Large Language Models): Utilizing LLMs for various tasks, including code generation, reasoning, and natural language processing.
• Service Now Platform: A metadata processing engine that relies heavily on configuration and metadata-driven development.
• Windsurf: An AI coding assistant tool deployed across Service Now's engineering team.
• Apriel: A small, fast LLM developed by Service Now and Nvidia, optimized for complex reasoning tasks.
• Snow skates: Service Now's internal Kubernetes infrastructure for deploying and managing services, including AI models.
• Now on Now: Service Now's internal program for using its own technologies within the company.
• Configuration vs. Code: Evolution of software development from coding to configuration and the increasing complexity of metadata management.

1. Pat Casey's Background and Early Days of Service Now

• Pat Casey, CTO of Service Now, recounts his early involvement with the company, being the second person to work on the code after Fred Luddy, the founder.
• He describes his initial skepticism about Fred Luddy's concept of a platform for building business apps, finding it limited in configurability compared to existing tools.
• Casey highlights Fred Luddy's vision of a browser-based platform that was easy for regular people to use, emphasizing its full deployment on the internet.
• He recalls the excitement of early customers who were empowered to do things themselves that previously required experts, fueling the company's growth.
• "Regular people could do meaningful stuff on it" - Pat Casey, highlighting the core value proposition of Service Now.

2. Current Coding Activities and the Dopamine Effect

• As CTO, Casey no longer writes code directly but focuses on leadership and strategic tasks.
• The last code he wrote was related to the cache manager, a performance-critical component.
• He acknowledges missing the "flow state" of deep coding but emphasizes the need for a customer or business problem to drive his coding efforts.
• Casey discusses the "dopamine effect" for engineers working with AI, varying based on their position in the stack.
  • Model builders experience satisfaction from creating and sharing models.
  • Feature developers gain satisfaction from rapid iteration and faster concept-to-prototype cycles.
• He notes that vibe coding, while efficient, doesn't fully satisfy the part of the brain that enjoys low-level coding tasks like assembler.

3. Service Now's Approach to AI Coding Assistants

• Service Now is investing heavily in AI coding assistants but not mandating their use.
• They were early adopters of coding assistants and have their own AI tools integrated into the platform.
• The company conducted a thorough evaluation and chose Windsurf, deploying it to approximately 7,000 engineers.
• A 10% increase in productivity (measured as stories per unit time per engineer) was observed after training on Windsurf, consistent with Google's reported findings.
• Casey emphasizes that the tool is a means to an end, focusing on the quality and rate of code production rather than tool usage itself.
• "I sincerely believe you're going to be a better developer with modern tools" - Pat Casey, advocating for the adoption of AI tools in engineering.

4. Customer Adoption of AI Tools

• Casey mentions hearing about customers using tools like Cursor, Windsurf, and GitHub Copilot.
• Service Now's customers often use the company's own AI-powered tools within their development environments.
• He clarifies that Service Now, as an enterprise company, does not monitor individual customer workflows or business cases.

5. Resistance to AI Tools and Security Concerns

• Casey acknowledges that some developers are resistant to using AI tools, but this is less common than expected.
• He attributes this to engineers generally being "technophiles" who recognize the productivity benefits of these tools.
• He notes that "silly mandates" requiring AI tool usage can lead to passive-aggressive behavior.
• Casey discusses the concerns of compliance and security teams regarding the risks of using AI tools, such as information leakage.
• He highlights the pressure on companies to "do more with less" and the expectation that AI will help improve productivity.

6. Subjective vs. Objective Productivity Gains

• Casey notes that initial estimates of productivity gains from AI tools were overly optimistic (30-40%).
• Actual gains observed after deploying first-generation tools were closer to 3-4%.
• Windsurf has shown a more significant 10% productivity increase.
• He suggests that engineers may overestimate the proportion of their time spent actually coding, as corporate coding jobs involve various other tasks.

7. Agentic AI and Autonomous Agents in Service Now

• Service Now uses its own AI and agentic tools internally through the "Now on Now" program.
• One use case is a question-and-answer system that directly answers customer queries using knowledge base articles, deflecting tickets.
• Another use case involves agents that research tickets, gather information from various systems, and provide summaries to support engineers.
• The next step is to automate the closure of tickets where the research agent is highly confident in its findings, potentially managing 15,000 cases per month automatically.
• "15% of a big number is still a big number" - Pat Casey, emphasizing the value of even partial automation.

8. Infrastructure for Running AI Models

• Service Now was primarily built on physical infrastructure and is transitioning to a hybrid model with hyperscalers.
• The company has invested in GPU pools in North America, Europe, and Asia, with six or seven GPU hubs currently.
• Service Now generally runs its own internally developed LLMs, often based on open-source models with additional training.
• Third-party LLMs like OpenAI, Claude, and Gemini are used in some out-of-box products, with customers having the option to choose their preferred model.
• Apriel, a small and fast LLM, is used for agentic reasoning tasks.

9. Kubernetes and Model Deployment

• Service Now uses a multi-instance architecture where each customer's production environment is a unique software stack.
• Agentic tasks are handled by a Kubernetes service (LLM router) that directs requests to appropriate LLMs based on metadata.
• Triton services manage connections to the underlying LLMs and allocate GPUs.
• Casey states that deploying and running AI models has not been significantly more complex than existing infrastructure management.
• For training, Service Now has built a large training cluster but also buys training time from other providers.

10. Apriel Model Details and Reasoning Capabilities

• Apriel is based on the Mistral model and has been trained to improve its complex reasoning abilities.
• It is a small (15 billion parameter) model designed to be fast and efficient for reasoning tasks.
• Casey discusses the trade-offs between large foundation models (which can do almost everything slowly) and smaller, faster models.
• He emphasizes the importance of finding the right balance between model complexity and performance for specific applications.

11. Kubernetes Infrastructure and GPU Management

• Service Now's Kubernetes infrastructure ("snow skates") is homegrown and runs in its own data centers.
• Snow skates provides a consistent abstraction layer across different environments, including physical data centers and hyperscalers.
• For inference, GPU faults are rare, but for training, checkpointing and restarting are used to handle GPU failures.
• Casey acknowledges the need for more resilient GPU clusters that can tolerate failures without requiring restarts, but this remains an unsolved problem.

12. Zurich Release and VIP Coding

• The Zurich release of Service Now introduces VIP coding, allowing developers to use AI to assist in building applications on the platform.
• Service Now's product is described as a metadata processing engine, where the behavior of an instance is defined by metadata.
• The build assistant helps developers create business rules, playbooks, and flow designer flows by using natural language prompts.
• The goal is to make it easier for customers to configure and deploy Service Now's products and get value from them quickly.

13. Evolution of Configuration

• Casey traces the evolution of configuration from assembly code to vendor-specific languages to metadata-driven development.
• He notes that while metadata simplified development, it also increased the breadth of knowledge required.
• The addition of AI tools adds another layer of complexity, requiring knowledge of models and prompting techniques.
• Service Now is focused on reducing the knowledge requirement and making it easier for customers to configure and use the platform.

14. Impact of LLMs on Hiring

• Service Now still hires interns and junior developers, but the numbers are slightly lower than in previous years.
• AI coding tools are particularly effective at tasks typically assigned to junior developers, potentially reducing demand in that area.
• Casey advises students considering CS to be aware of the changing landscape and potential disruptions.
• He believes that the technology revolution will ultimately be positive for humanity but requires careful management and choices.

15. Conclusion

• Casey expresses optimism about the future of technology and the potential for AI to improve quality of life.
• He acknowledges the challenges and disruptions that will occur during the transition but emphasizes the importance of making good choices.
• He encourages those who enjoy solving hard problems to embrace the opportunities presented by this technology revolution.