François Chollet - Creating Keras 3

Key Concepts

• Keras: A high-level, user-friendly deep learning API written in Python, capable of running on multiple backends (TensorFlow, PyTorch, JAX).
• Deep Learning: A subset of machine learning based on artificial neural networks with multiple layers.
• LLMs (Large Language Models): Deep learning models, specifically transformers, trained on vast amounts of text data to generate human-like text.
• AGI (Artificial General Intelligence): Hypothetical AI with human-level cognitive abilities, capable of understanding, learning, and applying knowledge across a wide range of tasks.
• Transformers: A neural network architecture that relies on self-attention mechanisms to weigh the importance of different parts of the input data.
• Gradient Descent: An optimization algorithm used to train machine learning models by iteratively adjusting the model's parameters to minimize a loss function.
• Neuropsychology: The study of the relationship between the structure and function of the brain and psychological processes.
• Cognitive Developmental Robotics: A field that focuses on building robots that can learn and develop cognitive abilities in a way similar to young children.
• Few-Shot Learning: A machine learning approach where a model learns to generalize from a small number of examples.
• LoRA (Low-Rank Adaptation): A parameter-efficient fine-tuning technique for large language models.
• Kaggle: A platform for machine learning competitions and datasets.
• ARC (Abstraction and Reasoning Corpus) Challenge: An IQ test for machines designed to measure general intelligence.
• Hebbian Learning: A learning rule that states that neurons that fire together wire together.

Early Interest in Computers and AI

Francois Chollet's fascination with computers began in childhood, even before he owned one. While school didn't fuel his interest, he became determined around age 15 or 16 to invent true AI and use it to create robots, influenced by Isaac Asimov's "Robot" series. He initially explored neuropsychology lectures from MIT OpenCourseWare (around 2005-2006) but found them lacking a cohesive model of the brain, offering only observations.

Shift to Math, Physics, and Robotics

Disappointed with neuropsychology, Chollet pursued intensive math and physics studies in a "[FRENCH]" program (preparatory classes for the grandes écoles) to prepare for a career in science. He then joined an engineering school with a strong robotics department. He read "On Intelligence" by Jeff Hawkins and research by Pierre-Yves Oudeyer on artificial curiosity. He found "Artificial Intelligence, A Modern Approach" by Russell and Norvig too focused on algorithms and lacking explanations of thinking. He was drawn to cognitive developmental robotics, believing in embodied cognition.

Research in Tokyo

After engineering school, Chollet moved to Tokyo to conduct research in cognitive developmental robotics, focusing on unsupervised video data processing. He built modular, hierarchical representations of video feeds using matrix factorization (not gradient descent) for few-shot video segmentation and classification.

Keras Creation and Google

After his research, Chollet entered the tech industry, joining startups in New York City and San Francisco. At 25, he created Keras. He joined Google and continued working on Keras.

Keras: A Deep Learning Library

Keras is a Python-based deep learning library that works with TensorFlow, PyTorch, and JAX. It allows users to quickly build and train deep learning models.

Keras Origins

Chollet's interest in deep learning began around 2009. Inspired by the 2012 ImageNet competition, he started building a question-answering engine called QuickAnswers.io in 2014, which used an early form of Retrieval-Augmented Generation (RAG). To build this, he needed a tool for recurrent neural networks (RNNs), specifically LSTMs, and created his own library using Theano. In March 2015, he open-sourced this library as Keras.

Keras's Rise and Integration with TensorFlow

Keras gained traction due to the growing interest in RNNs and its ease of use. After Google released TensorFlow, Chollet rebased Keras to support TensorFlow. Recognizing Keras's popularity, the head of TensorFlow invited Chollet to join the Google team to work on Keras full-time.

Keras 3: Multi-Backend Support

Keras 3 is a complete rewrite of Keras to support multiple backends (TensorFlow, PyTorch, JAX). This allows users to choose the best framework for their model and hardware, potentially improving performance by 20-30%. Keras 3 ensures users always get the best performance for their model by switching backend based on which backend is going to be the fastest for their particular model architecture and hardware. It also enables interoperability with packages from different ecosystems. Keras 3 allows users to benefit from the entire ML ecosystem. If you have a Keras model, it's a Keras 3 model, it's multi backend, you can use it with packages from the TensorFlow ecosystem, like TFJS, TFLite and so on. You can use it with packages from the PyTorch ecosystem as well. You can even use it with packages from the JAX ecosystem.

Community and Open Source

Keras is a community-driven open-source project. Chollet emphasizes the importance of community contributions and guides contributors towards industry best practices.

New Book

Chollet is working on a new book that aims to build intuitive and actionable mental models of deep learning and AI. It is expected to be released in mid-2024.

Gemma and Keras Integration

Gemma is a new open-source LLM by Google. A Keras 3 implementation of Gemma is available, supporting multiple backends. It includes features for model-parallel distributed training and LoRA fine-tuning.

Kaggle Integration

Keras is integrated with Kaggle, a machine learning competition platform. Keras starter notebooks are provided for competitions. KerasCV and KerasNLP models are available on Kaggle Models, a model hub, allowing offline use in competitions and community sharing of fine-tuned models.

LLMs and the Hype Around AI

Chollet acknowledges the hype around LLMs and AI but emphasizes that LLMs are not a prelude to AGI. LLMs are based on curve fitting and memorization, leading to issues like hallucinations and limited generalization.

LLMs as Databases of Programs

Chollet describes LLMs as databases of programs, specifically vector programs, that can be retrieved and run based on prompts. This is useful for automating tasks but is not true intelligence.

Intelligence vs. Memorization

Chollet defines intelligence as the ability to adapt and efficiently acquire new skills in novel situations, citing Jean Piaget's quote: "Intelligence is what you use when you don't know what to do." LLMs lack this ability, scoring low on tests like the ARC challenge, which humans can easily solve.

Transformers and the Brain

Chollet sees some overlap between transformers and the brain, particularly in Hebbian learning. Transformers can embed semantically similar tokens together, similar to how the brain learns passively. However, most human learning is active, causal, and high-level.

Existential Risk and the Future of AI

Chollet is not worried about existential risk from AI, as LLMs are limited by their training data. However, he acknowledges potential concerns about the deployment of AI at scale, its impact on culture, and its effect on specific jobs.

The Path to AGI

Chollet believes that achieving human-level AI is still far off. LLMs may be part of the solution, but they are not the most essential part. The focus should be on developing systems that can perform few-shot learning and synthesize new programs to adapt to novel situations.