How BlackRock Builds Custom Knowledge Apps at Scale — Vaibhav Page & Infant Vasanth, BlackRock

Key Concepts:

• AI Applications & Knowledge Apps
• Document Extraction
• Workflow Automation
• Q&A Systems
• Agentic Systems
• LLMs (Large Language Models)
• Prompt Engineering
• LLM Strategies (RAG, Chain of Thought)
• Context Limitations
• Model Limitations
• App Deployment (Inference Clusters, Burstable Clusters)
• Sandbox
• App Factory
• Extraction Templates
• Field Dependencies
• Transformation Logic
• Executors
• Human-in-the-Loop

1. Introduction and Context

• Infant (Director of Engineering) and Wyber (Principal Engineer) from BlackRock's data teams discuss scaling the building of custom AI and knowledge applications within BlackRock.
• BlackRock is the world's largest asset management firm. Portfolio managers and analysts process a large amount of information daily to develop investment strategies and rebalance portfolios.
• Investment operations teams are the backbone, ensuring smooth execution of investment manager activities, from data acquisition to post-trade activities.
• These teams build complex internal tools specific to their domains, making rapid app development crucial.

2. Types of Applications

• Applications fall into four categories:
  • Document Extraction: Extracting entities from documents.
  • Workflow Automation: Defining complex workflows and integrating with downstream systems.
  • Q&A Systems: Chat interfaces for information retrieval.
  • Agentic Systems: Autonomous systems for complex tasks.
• LLMs are leveraged to augment or supercharge existing systems in each domain.

3. Use Case: New Issue Operations Team

• The new issue operations team sets up securities in internal systems when market events occur (e.g., IPOs, stock splits).
• The team requires a tool to ingest prospectuses or term sheets, process them through a pipeline, consult domain experts (business, equity, ETF teams), and generate structured output.
• This output is then used by engineering teams to build transformation logic and integrate with downstream applications.
• The traditional process is lengthy, especially when introducing new model providers or strategies. Agentic systems have not been effective due to complexity and domain knowledge requirements.

4. Challenges in Scaling AI App Development

• Prompt Engineering:
  • Significant time spent on prompt engineering with domain experts.
  • Prompts for document extraction can become complex (e.g., describing financial instruments in multiple paragraphs).
  • Challenges in iterating, versioning, comparing, and evaluating prompts.
• LLM Strategies:
  • Choosing appropriate LLM strategies (RAG, Chain of Thought) varies based on the instrument.
  • Simple tasks like data extraction require different strategies depending on the document.
  • Corporate bonds (vanilla) can be handled with in-context models if document size is small.
  • Large documents (thousands of pages) exceed token limits, necessitating alternative strategies.
  • Iterative process involving prompt and LLM strategy experimentation is crucial.
• Deployment:
  • Traditional challenges: distribution, access control, federation.
  • AI-specific challenges: selecting appropriate cluster types (GPU-based inference clusters for overnight analysis of research reports, burstable clusters for other tasks).
  • Cost control considerations.

5. BlackRock's Solution: High-Level Architecture

• BlackRock developed a framework to accelerate app development, reducing the time from 3-8 months to a few days.
• Key components:
  • Sandbox: A playground for domain experts to build and refine extraction templates.
  • App Factory: A cloud-native operator that takes a definition and spins out an app.
• The data platform and developer platform handle data ingestion and orchestration.
• The framework federates pain points like prompt creation, extraction templates, and LLM strategy selection.
• Modular components enable rapid iteration, allowing domain experts to build apps quickly.

6. Sandbox Details

• The sandbox allows operators to quickly build and refine extraction templates and compare extraction results.
• It includes prompt template management with fields to extract, corresponding prompts, and metadata (data types).
• Operators require greater configuration capabilities beyond prompts and data types, including QC checks, validations, constraints, and inter-field dependencies.
• Example: In new security onboarding, a bond being callable requires values for call date and call price.
• Extraction templates define field names, data types, sources (extracted or derived), required status, and field dependencies.
• Document management involves ingesting documents from the data platform, tagging them by business category, labeling, and embedding.

7. Extraction and Workflow

• Operators run extractions and review the extracted values.
• Traditional tools often require manual processes (downloading CSV/JSON, manual transformation) to pass results to downstream processes.
• BlackRock's framework includes a low-code/no-code environment where operators can build transformation and execution workflows.
• This enables end-to-end pipeline execution.

8. Key Takeaways

• Invest heavily in prompt engineering skills for domain experts, especially in finance.
• Educate the firm on LLM strategies and how to select the right ones for specific use cases.
• Evaluate the ROI of AI app development versus off-the-shelf products.
• Human-in-the-loop is crucial, especially in regulated environments, to ensure compliance and accuracy. Design for human-in-the-loop first.
• The app factory component allows operators to take knowledge from the sandbox (extraction templates, transformers, executors) and build custom applications exposed to users.
• Users can upload documents, run extraction, and execute the entire pipeline without configuring templates or integrating results manually.

9. Q&A Highlights

• The framework targets investment operation domain experts building applications.
• Reusable components exist, but initiatives for CEO-level insights (asset/liability memos) may use different frameworks.
• Information security is addressed through multiple layers of controls and policies across the infrastructure, platform, application, and user levels.
• Different strategies and model providers are used based on the use case.

10. Conclusion

BlackRock has developed a comprehensive framework to streamline the development of custom AI applications, particularly in the investment operations domain. By focusing on modularity, empowering domain experts, and addressing key challenges in prompt engineering, LLM strategy selection, and deployment, BlackRock has significantly reduced app development time while maintaining necessary controls and human oversight. The emphasis on human-in-the-loop and ROI evaluation highlights a practical approach to leveraging AI in a highly regulated environment.