Building Reliable Support Agents Using the Effect Typescript Library - Michael Fester

Key Concepts

• Effect: A TypeScript library for building robust, type-safe, and composable systems.
• LLMs (Large Language Models): AI models used for natural language processing and generation.
• Type Safety: Ensuring that data types are handled correctly to prevent errors.
• Composition: Combining smaller components to build larger systems.
• Concurrency: Managing multiple tasks simultaneously.
• Streaming: Processing data in real-time as it becomes available.
• Dependency Injection (DI): A design pattern that allows dependencies to be provided to a component rather than created within it.
• OpenTelemetry: A framework for observability, including tracing, metrics, and logging.
• Schemas: Data structures that define the format and types of data.
• Agents: Autonomous entities that perform tasks.
• Workflows: Deterministic multi-step processes.
• Retry Policies: Strategies for automatically retrying failed operations.
• Domain Specific Language (DSL): A programming language tailored to a specific domain.

14.ai's Architecture and Use of Effect

14.ai uses Effect across its entire AI-native customer support platform. The architecture includes:

• React Front End: Powers dashboards, agent IDE, knowledge management, insights, analytics, and SDKs.
• Internal RPC Server: Handles app logic, built on Effect RPC and a modified version of TanStack Query.
• Public API Server: Uses Effect HTTP with OpenAPI docs autogenerated from annotated schemas.
• Data Processing Engine: Syncs data from CRM, docs, and databases for real-time analytics and reporting.
• Agent Workflows: Written in a custom DSL built on Effect, allowing for mixing deterministic and non-deterministic behavior.
• Postgres Database: Used for both data and vector storage, with Effect SQL handling queries.

Everything is modeled using Effect schemas, providing runtime validation, encoding/decoding, type-safe input/output handling, and autogenerated documentation.

Agent Architecture and DSL

Agents are planners that take user input, create a plan, choose actions/workflows/sub-agents, execute them, and repeat until the task is complete.

• Actions: Small, focused units of execution (e.g., fetching payment info, searching logs).
• Workflows: Deterministic multi-step processes (e.g., cancelling a subscription).
• Sub-Agents: Group related actions and workflows into domain-specific modules (e.g., billing agent, log retrieval agent).

A custom DSL built on Effect's functional pipe-based system is used to model agent workflows. This DSL allows expressing branching, sequencing, retries, state transitions, and memory in a composable way.

Reliability and Error Handling

Reliability is crucial for mission-critical systems. 14.ai uses Effect to handle failures and ensure system resilience.

• LLM Provider Fallback: If one LLM provider fails (e.g., GPT-4), the system falls back to another with similar performance (e.g., GPT-3.5 Turbo).
• Retry Policies: Used to automatically retry failed operations, tracking state to avoid retrying failed providers too often.
• Token Stream Duplication: Token streams are duplicated, one directly to the user and one for storage (e.g., for analytics). Effect facilitates this.

Testing and Dependency Injection

Dependency injection is heavily used for testing.

• Mocking LLM Providers: DI allows swapping real LLM providers with mock versions to simulate failure scenarios.
• Service Swapping: Services and providers can be easily swapped with mock versions without affecting the internals of the system.

Developer Experience

Effect provides a good developer experience for building agentic systems.

• Schema-Centric: Input, output, and error types are defined upfront using schemas, which provide encoding/decoding, type safety, and automatic documentation.
• Dependency Injection: Services are provided at the entry point of the system, composed as needed, and easily mocked for testing. Dependencies are guaranteed at compile time.
• Modular and Composable Services: Services are modular and composable, making it easy to override behavior or swap implementations.
• Strong Guard Rails: Effect helps prevent common mistakes, allowing engineers new to TypeScript to become productive quickly.

Lessons Learned

• Discipline Required: Effect is powerful, but using it well requires discipline. It's easy to accidentally catch errors upstream and silently lose failures.
• DI Complexity: Dependency injection can be hard to grasp at scale, especially when tracing where services are provided across multiple layers.
• Learning Curve: Effect has a steep learning curve due to its large ecosystem of concepts and tools. However, once the initial bump is overcome, the benefits compound.

Conclusion

Effect helps build predictable and resilient systems, especially for LLM and AI-based applications where reliability and handling non-determinism are critical. It brings functional programming principles to TypeScript in a practical way for production use. Incremental adoption is possible, starting with a single service or endpoint.