OpenAI says Habitat handles more than 70 million requests per second

OpenAI says its Habitat online storage platform now handles more than 70 million requests per second after evolving from a small Python library into a distributed service. The company says the system operates across almost 40 geographic regions, serves more than 500 petabytes of data and supports products used by more than 1 billion people each week.

Those figures come from OpenAI’s engineering account and have not been independently audited. The company did not enumerate the regions, define how it counts the 500 petabytes or publish raw data behind its throughput and reach claims. Its architecture diagram identifies ChatGPT, the API, Codex and internal services as Habitat clients.

That shared product footprint puts Habitat inside a broader service stack: an August outage affected ChatGPT, Codex and OpenAI API endpoints. OpenAI’s Habitat account does not connect that incident to the storage platform.

Habitat first launched to support GPTs at OpenAI’s November 2023 DevDay, according to the company’s current account. It began as a small Python client-side library connected through ChatGPT’s main server to one Azure Cosmos DB database. The library gave product teams a common interface while hiding schema lookup, routing, authorization, encryption, serialization, request shaping and connection pooling.

By mid-2025, OpenAI says, the client-library model had become difficult to operate. Protocol changes required coordinated deployments across dozens of services, while unrelated rollbacks could return older clients to production. The company moved the storage logic into a standalone Habitat service, creating one place to deploy changes, observe behavior, enforce access controls and audit requests while restricting direct access to storage resources.

The first service retained Python so the team could stabilize its APIs and unblock product work before attempting a rewrite. OpenAI describes that choice as deliberate technical debt: the extra network hop increased latency, while Python workers consumed more CPU and memory. The company says CPU-heavy work and background tasks could delay asyncio coroutine scheduling by hundreds of milliseconds and, in edge cases, several seconds. Python’s asyncio documentation describes the framework as suited to I/O-bound network code, while the threading documentation notes that CPython’s default global interpreter lock constrains CPU-bound parallelism in threads. Those documents support the general mechanism, not OpenAI’s Habitat measurements.

OpenAI says one tail-latency problem came from workers parsing a large feature-flag configuration at the same one-minute interval. The team reduced the configuration, refreshed it less often and added jitter so the work would not happen simultaneously. It also traced uneven load to last-in, first-out connection reuse in aiohttp: after traffic bursts, slower processes retained the newest connections and received more work. OpenAI says switching reuse to first-in, first-out reduced that imbalance.

The service now relies largely on Istio and Envoy for connection pooling and load-aware balancing, according to OpenAI. Envoy multiplexes HTTP/2 requests and applies rate limits and circuit breakers, mechanisms described in the project’s documentation for connection pooling and circuit breaking. That consolidation matters for Azure Cosmos DB because a direct-mode client can keep connections to every replica of each physical partition, with the connection count rising as partitions or concurrency increase, according to Microsoft’s connectivity documentation. Microsoft does not disclose Habitat’s configuration.

Habitat deliberately exposes a constrained NoSQL interface instead of arbitrary SQL. OpenAI says predictable request shapes keep unbounded scans, joins and fan-out queries away from the online path. Its object-and-edge model colocates each object with its edges, while more complex queries go to client-specific Rockset instances fed by near-real-time change data capture.

After deferring a migration for about a year, OpenAI says two engineers used Codex and GPT-5.5 to rewrite the Habitat service in Rust during the second quarter of 2026. At publication, the company said Rust handled 95% of production requests and the remaining Python service would be retired within weeks. It reported that the Rust implementation was six times as CPU-efficient and 15 times as memory-efficient as Python, with lower average and tail latency, but supplied no benchmark methodology, hardware baseline, raw results or independent reproduction. The company separately says the Python implementation peaked above 20 million requests per second and that the relevant overall workload grew more than tenfold year over year in each of the past three years.

OpenAI said a second engineering post would cover Habitat’s storage layer and multi-tenancy design; those details were not included in the first account.