Why We Don’t Need More Data Centers - Dr. Jasper Zhang, Hyperbolic

Hyperbolic: AI Cloud for Developers - Why We Don't Need Just More Data Centers

Key Concepts:

• GPU Marketplace: An aggregation layer connecting GPU providers and users.
• HyperDOS: Hyperbolic Distributed Operating System, a Kubernetes-based software for cluster management.
• GPU Utilization: The percentage of time GPUs are actively used.
• Compute Matching: Efficiently connecting GPU supply with user demand.
• AI Workload: Different types of AI tasks, including training, online inference, and offline inference.
• Scaling Law: The principle that increased compute power leads to better model quality.

The Exploding Demand for GPUs and Data Centers

• AI is integrating into everything, making every company an AI company.
• This drives an exploding demand for GPUs and data centers.
• McKinsey projects a need for 4x more data centers by 2030, built in a quarter of the current time.
• Current data center capacity is 55 gigawatts.
• A median scenario predicts a 22% annual growth rate, requiring 219 gigawatts by 2030.

Challenges of Building More Data Centers

• High Cost: The first Stargate data center cost over a billion dollars.
• Slow Grid Connection: A seven-year wait to connect a 100-megawatt facility to the electrical grid in Northern Virginia.
• High Energy Consumption: GPUs and data centers consume 4% of total US electricity.
• Environmental Concerns: Significant CO2 emissions.
• Supply Deficit: Even if all planned data centers are built on time, a 15+ gigawatt deficit is projected in the US by 2030.

Low GPU Utilization and Fragmented Market

• Deoitte reports that GPUs sit idle 80% of the time for enterprises.
• Analysis shows 100+ fragmented GPU clouds.
• Many users struggle to find GPUs or face extremely high prices.
• Conversely, many GPUs sit idle in data centers and clouds.

The GPU Marketplace Solution

• A GPU marketplace or aggregation layer can connect different data centers and GPU providers to users.
• Hyperbolic is presented as an example of this solution.
• Hyperbolic is building a global orchestration layer.
• They invented HyperDOS (Hyperbolic Distributed Operating System), a Kubernetes-based software.
• Any cluster with HyperDOS installed becomes part of the Hyperbolic network within five minutes.
• Users can rent GPUs through spot instances, on-demand, long-term reservations, or model hosting.

Benefits of a GPU Marketplace

• Compute Matching: Solves the problem of matching GPU supply with user demand.
• Commoditization of GPUs: Reduces the time spent waiting for data center resources.
• Diverse Options: Provides users with different pricing and performance options.
• Cost Savings: Potential cost savings of 50-75%.
• Hyperbolic's H100 GPU cost is $0.99 per hour, compared to $11 on Google and $2-3 on Lambda.
• Reduced Supplier Vetting: Users can choose providers based on ratings and performance benchmarks instead of vetting multiple data centers.

Use Case Example: Startup GPU Needs

• Traditional Cloud: A startup reserves 1,000 GPUs for a year, then needs an additional 10,000 GPUs for a month, requiring another year-long reservation. After six months, they only need 500 GPUs but are stuck with the original commitment.
• Hyperbolic: The startup rents 1,000 GPUs initially, then rents an additional 10,000 GPUs for just one month. After six months, they release the idle GPUs on the marketplace.
• Cost Comparison: The traditional cloud costs $43.8 million, while Hyperbolic costs $6.9 million (6x saving).

Increased Productivity and Scaling Law

• The speaker argues that the benefits extend beyond cost savings.
• Scaling law dictates that more compute leads to better model quality.
• With the same budget, startups can increase productivity by 6x.
• This allows startups to train their own models instead of relying solely on closed AI models like OpenAI and Anthropic.

Evolution to an All-in-One AI Platform

• The GPU marketplace will evolve into an all-in-one platform for different AI workloads.
• This includes AI inference (online and offline) and training jobs.

Key Takeaways

• Focusing solely on building more data centers is insufficient. Smarter resource allocation is needed.
• A GPU marketplace can reduce costs and increase productivity.
• Reusing and recycling idle compute is more sustainable than solely building new data centers.

Hyperbolic OS Details

• HyperDOS is a Kubernetes agent.
• It can be installed on any cluster with Kubernetes, including personal computers.
• Hyperbolic uses a feudal model: a "monarch" server manages "barons" (data centers).
• Users request GPUs from the monarch server, which then provisions resources through the barons.