Function Calling is All You Need — Full Workshop, with Ilan Bigio of OpenAI

Key Concepts

• Function Calling: Enabling language models to interact with external tools and APIs.
• Agents: Autonomous entities that use language models and function calling to achieve goals.
• RAG (Retrieval-Augmented Generation): Enhancing language model responses with information retrieved from external sources.
• Memory: Storing and retrieving information to maintain context and personalize interactions.
• Delegation: Transferring tasks or conversations to other agents or models.
• Asynchrony: Performing tasks concurrently without blocking the main thread of execution.
• Tool Calling: The act of a language model deciding to use a specific tool or function based on the user's input.

History of Function Calling

• Text Completion (GPT-2, GPT-3): Initial models focused on continuing input text, making instruction following difficult.
• Instruction Following (InstructGPT): Models trained to perform tasks based on user instructions.
• User and Assistant Roles: Introduction of personas through post-training.
• External Tools: Allowing models to interact with external states and APIs.
• WebGPT: Early implementation of function calling using web search, trained to imitate user behavior.
• Meta's Tool Use: Teaching models to use any tool by analyzing log probabilities.
• OpenAI's General Function Calling (June 2023): Pre-training models to use tools based on a defined syntax.

Crash Course on Function Calling

• Purposes:
  • Fetching data (reading APIs, retrieval, memory).
  • Taking action (writing APIs, managing application state, workflow actions).
• Process:
  1. Tell the model which functions it can use.
  2. Provide user input.
  3. The model specifies the function it wants to use and its arguments.
  4. The developer parses the function call and executes the code.
  5. The result is provided back to the model.
  6. The model uses the result in its generation.
• Best Practices:
  • Write clear functions.
  • Apply software engineering best practices (intuitive functions, function of least principle, enums, object structure).
  • Explain the purpose of each parameter.
  • Use a system prompt and include examples.

Agents as Loops

• Agents are essentially loops that:
  1. Specify the tools.
  2. Call the model.
  3. Get the message.
  4. Print the message.
  5. Handle tool calls.
  6. Append the results.
  7. Repeat until no more tool calls are needed.

Memory Implementation

• Basic Implementation: Using a list to store and retrieve information.
• Function Definitions:
  • add_to_memory(memory_text): Appends information to the memory.
  • get_memory(): Retrieves information from the memory.
• Usage:
  • The model is instructed to call add_to_memory when the user provides factual information or preferences.
  • The model is instructed to call get_memory in the first turn.
• Persistence: Storing the memory bank in a local JSON file and reading it in at the beginning of each session.

Delegation

• Forms of Delegation:
  • Handoffs: Fully swapping the conversation to a different agent (replacing system prompt and tools).
  • Nested Calls: Calling one model from another.
  • Manager Tasks: Using asynchronous tasks to manage delegation.
• Implementation Example:
  • Creating a function delegate_to_smarter_model(task_description) that calls a more powerful model (e.g., GPT-4) to perform a task.
  • Using the OpenAI API to create a chat completion with the task description.

Asynchrony

• Problem: Blocking the main thread while waiting for delegated tasks to complete.
• Solution: Using asynchronous programming to perform tasks concurrently.
• Implementation:
  • Using asyncio to create asynchronous functions.
  • Creating a server and client using websockets to handle user input and model responses.
  • Using a message queue to ensure that messages are processed in the correct order.
  • Creating tasks with random IDs and allowing the model to check their status.
• Benefits:
  • The model can continue to interact with the user while tasks are running in the background.
  • Multiple tasks can be performed concurrently, reducing the overall time required.

Random Cool Stuff

• Real-Time API Tricks:
  • Stay Silent Function: Using a function to verify if the user is done talking, triggered by voice activity detection (VAD).
  • XML Tagging: Instructing the model to read a script according to XML tags to control speech patterns.

Super Unsafe Self-Tool Writing Agent

• Concept: An agent that can write its own functions and add them to its toolset.
• Implementation:
  • Using the exec function to evaluate Python code provided by the model.
  • Adding the new function to the agent's tool list.
• Example:
  • The agent is instructed to create a calculator function.
  • The agent writes the Python code for the calculator function.
  • The agent adds the calculator function to its tool list.
  • The agent can then use the calculator function to perform calculations.
• Caution: This approach is highly unsafe and should not be used in production environments.

Conclusion

Function calling is a powerful technique that enables language models to interact with external tools and APIs, creating more versatile and capable agents. By combining function calling with concepts like memory, delegation, and asynchrony, developers can build sophisticated applications that can perform complex tasks and provide personalized experiences. The workshop demonstrated these concepts through practical examples and live coding, emphasizing the importance of understanding the underlying principles and building from first principles.