status.openai.com web signal

OpenAI: Postgres replica failure peaked API, ChatGPT errors at 75%

TL;DR

  • OpenAI says requests to its API and ChatGPT failed at a peak 75% rate between 1:48 and 5:21 PT on November 21.
  • Both Postgres read replicas behind the Primary API service, which handles authentication across the entire OpenAI stack, failed during scheduled maintenance.
  • OpenAI has paused similar maintenance, retuned connection-churn settings, and shipped a feature flag to shift traffic off replicas by percentage.

Between 1:48 and 5:21 PT on November 21, requests to OpenAI's API and ChatGPT failed at rates peaking at 75% on paid and free ChatGPT, according to the company's post-incident writeup. The outage ran roughly three and a half hours and swept across Chat Completions, Fine-tuning, Embeddings, Images, Batch, Audio, Moderations, Realtime and the Files endpoint on the API side, plus Conversations, Login, Search, File uploads and Voice mode in ChatGPT.

The trigger, as OpenAI puts it: "This outage was triggered by maintenance on a PostgresDB service used by OpenAI's Primary API service that caused both read replicas to fail at the same time." The replicas were "load-bearing," and the Primary API itself "provides user authentication and service-to-service authentication across the entire OpenAI stack." When the replicas went, the login and inter-service handshake underneath the rest of the product went with them.

Attempts to revive the existing replicas did not work. The fix was "bringing two new replicas in the same region into service" in coordination with the provider, which the writeup does not name.

OpenAI says it has placed a moratorium on similar maintenance, retuned its database to reduce connection churn, and shipped a feature flag that can shift traffic away from replicas by percentage. Longer term, the company says it will add a pre-maintenance review process, hold adequate buffer capacity as hot standbys during maintenance windows, and build strategies against replica lag. It is the latest entry in a long run of AI infrastructure incidents we have been tracking.