OpenAI has detailed how its internal storage platform, Habitat, grew from a small Python library into a distributed system handling more than 70 million requests every second.
According to OpenAI, Habitat now serves more than 500 petabytes of data across nearly 40 geographic regions and supports products used by more than one billion people each week.
The platform provides storage services behind products including ChatGPT, the OpenAI API and Codex, handling tasks such as retrieving data, routing requests, enforcing access controls and managing where information is stored.
From a Python Library to a Storage Service
Habitat originally launched alongside GPTs at DevDay 2023 as a Python client library connected to Azure Cosmos DB.
The goal was to give product developers a simpler way to store and retrieve information without requiring each team to manage details such as routing, encryption, authorization, caching and database connections.
As OpenAI’s products and internal services expanded, that approach became harder to manage.
By mid-2025, changes to Habitat could require coordinated updates across dozens of services. OpenAI eventually moved the storage logic into a standalone service, allowing infrastructure changes and security controls to be managed centrally.
Scaling Python Came With Performance Challenges and Eventual Move to Rust
OpenAI initially continued running the service in Python, despite the additional CPU and memory overhead involved in operating it at large scale.
One challenge was keeping response times consistent while handling large numbers of simultaneous storage requests. OpenAI found that Python’s asynchronous processing could introduce delays when CPU-heavy background work competed with requests waiting to be processed.
The company responded by limiting how many requests individual Python processes handled and spreading traffic across a much larger number of worker processes.
OpenAI also describes changes to connection pooling and load balancing after discovering that slower server processes could unintentionally receive more traffic, creating a cycle in which already overloaded systems became increasingly busy.
Tools including Envoy and Istio are now used to help manage connections and distribute traffic across the infrastructure.
OpenAI ultimately decided to move Habitat away from Python as its traffic continued to increase. The Python version of the service peaked at more than 20 million requests per second, according to the company.
In the second quarter of 2026, two engineers working with Codex and GPT-5.5 rewrote the service in Rust. OpenAI says the Rust implementation now handles approximately 95% of Habitat’s production requests.
The company reports that the new version is six times more CPU efficient and 15 times more memory efficient than the Python implementation, while also reducing average and high-end request latency.
OpenAI says it plans to retire the remaining Python service as the Rust migration is completed.
Habitat’s application service is only one part of the overall storage system. OpenAI says the platform currently serves more than 500 petabytes of data and processes over 70 million requests each second, using infrastructure that includes Azure Cosmos DB along with caching and other storage systems.
Rather than supporting complex database queries directly, Habitat uses a more restricted NoSQL-style interface designed to keep individual operations predictable and easier to scale.
OpenAI says more complex analytical and search workloads can be handled separately, helping prevent expensive queries from affecting the storage systems used by its online products.
The company plans to publish a second engineering article detailing the underlying storage layer and how it scaled Azure Cosmos DB and other parts of Habitat to support its current workload.
Leave a Reply