Spotlight on Shopify

Key Concepts

• Agentic Tools: AI assistants ideal for adaptive decision-making, iteration, and autonomy, particularly in exploratory or ambiguous tasks.
• Structured Workflow Orchestration: Using AI for intelligent completion of components within predictable, well-defined steps, emphasizing consistency and oversight.
• ROAST: An open-source Ruby-based workflow orchestration tool developed at Shopify to standardize and streamline AI-driven processes.
• Cloud Code: An AI-powered coding assistant used at Shopify, particularly effective for individual developer tasks and integrated with ROAST for larger workflows.
• Deterministic vs. Non-Deterministic Processes: The core distinction between structured workflows (deterministic) and agentic tools (non-deterministic) and the benefits of combining them.
• Interleaving Workflows: Combining agentic and structured workflows to leverage the strengths of both approaches, minimizing errors and maximizing efficiency.
• Coding Agent: A tool within ROAST that wraps Cloud Code, allowing for automated workflows with narrower scopes for the AI agent.

Challenges of Scale at Shopify Engineering

• Shopify is a large Ruby on Rails shop with a massive codebase: millions of lines of code, approximately 5,000 repositories, and around 500,000 pull requests (PRs) generated annually.
• The core challenge is maintaining developer productivity at this scale, which is the focus of Obie Fernandez's augmented engineering group.

Two Approaches to Using AI

• Agentic Tools:
  • Ideal for scenarios requiring adaptive decision-making, iteration, and autonomy.
  • Shine when tasks are exploratory or ambiguous, relying on the LLM's reasoning and judgment.
  • Examples: Feature development, debugging, and tasks requiring ongoing adaptation.
  • Tools like Cloud Code are well-suited for these use cases.
• Structured Workflow Orchestration:
  • Better for tasks with predictable, well-defined steps.
  • Emphasizes consistency, repeatability, and clear oversight.
  • Leverages AI for intelligent completion of components within the larger workflow.
  • Examples: Migrating legacy codebases (e.g., Python 2 to Python 3), refactoring large systems, and automated testing.

ROAST: A Structured Workflow Orchestration Tool

• ROAST is an open-source workflow orchestration tool developed at Shopify to address the need for standardized AI-driven processes.
• It is implemented in Ruby, which is unusual for this type of tool, but it can interleave prompt-oriented tasks with bash scripts or other invocations.
• ROAST was created to address the problem of constant reinvention of the wheel within large organizations, where different teams implement similar workflows in different ways.
• ROAST helps standardize solutions and promote code reuse.

Relationship Between Cloud Code and ROAST

• The relationship between Cloud Code and ROAST is bidirectional.
• Using ROAST as a Tool for Cloud Code:
  • Cloud Code can call ROAST workflows to handle specific tasks, such as test grading.
  • This allows Cloud Code to leverage the deterministic nature of ROAST for certain steps in a larger workflow.
• Using Cloud Code as a Tool for ROAST:
  • ROAST includes a coding agent tool that wraps Cloud Code.
  • Workflows can be automated, with Cloud Code invoked in SDK mode to work on narrower scopes within the workflow.

Examples of ROAST in Action

• Automated Testing Generation and Test Optimization:
  • ROAST is used to address coverage gaps in Shopify's large codebase.
  • The process is broken down into steps, such as running coverage tools and generating reports.
  • Cloud Code can then be used to generate tests based on the coverage report.
• Sorbet Type Checking:
  • ROAST is used to improve the application of types in Shopify's Ruby on Rails monolith using Sorbet.
  • The workflow interleaves calls to Sorbet tools with prompts to Claude to address type deficiencies.
  • This approach is effective because Sorbet is not well-known by the models, and the tools used for type checking are not intuitive.

Benefits of Using ROAST

• Session Saving: ROAST allows users to save their session and replay from a specific step, which is a significant time-saver when debugging workflows.
• Function Caching: ROAST incorporates function caching, which speeds up workflow development by caching the results of function calls.
• Standardized Solution: ROAST provides a standardized solution for AI-driven workflows, which promotes code reuse and reduces the need for reinvention.
• Convention-Oriented Approach: ROAST takes a convention-oriented approach, similar to Ruby on Rails, which makes it easy to use for developers familiar with Rails.

Cloud Code Usage at Shopify

• Shopify was an early adopter of Cloud Code.
• Cloud Code has seen significant adoption within Shopify, with approximately 500 daily active users and 250,000 requests per second at peak.

Cloud Code SDK Tips

• When prototyping with the Cloud Code SDK, it can be useful to use the dangerouslySkipPermissions option to bypass permission checks.
• An example prompt for including a coding agent in a workflow is: "Use your code agent tool function to raise the branch coverage level of the following test above 90%. After each modification run rake test with coverage [path to the test]."

Conclusion

• Interleaving deterministic and non-deterministic processes, leveraging the power of Cloud Code and ROAST, has been a successful combination at Shopify.
• ROAST is taking off within Shopify as a standardized solution for AI-driven workflows.
• ROAST has a lot of cool features, such as inline prompts, inline bash commands, and a convention-oriented approach similar to Ruby on Rails.