OpenAI traces rare Rockset crashes to hardware corruption and an old libunwind bug

OpenAI describes an investigation into rare, confusing crashes inside Rockset, a C++ data infrastructure service used by ChatGPT for search, data plugins, and querying over conversations. Rockset maintains fresh indexes over workspace knowledge so ChatGPT can retrieve relevant information while answering questions or taking actions. Because its execution layer is written in C++, it can run efficiently and control memory closely, but it also inherits C++’s main risk: memory errors can corrupt program state and crash the process.

The crashes looked abnormal from the start. A normal C++ function appeared to finish execution, but when it returned, the program jumped to an invalid address. In some cases, the saved return address on the stack was NULL. In others, the stack pointer register looked shifted by 8 bytes, as if it had been modified during ordinary execution. That is not how typical compiled C++ code behaves. The stack pointer is normally adjusted in predictable places, such as a function’s entry and exit code, and OpenAI says the usual suspects, including inline assembly, setcontext, and longjmp, were not in use. The failure mode looked less like an ordinary application bug and more like something impossible happening at the machine-state level.

The team first approached the issue as a standard debugging problem. They inspected individual core dumps, which are snapshots of a program at the moment it crashes, and tried to infer what had corrupted the stack. Many crashes seemed to involve a method named DocumentTree::updateDocument. It looked as if that method had called some unknown function, the stack became corrupted while that function ran, and then the function returned to memory that was not executable code. But updateDocument is large and heavily optimized through inlining, so there were too many possible internal call sites to inspect one by one.

The investigation quickly became difficult because stack corruption damages the very evidence engineers usually rely on. OpenAI’s infrastructure uses folly’s fatal signal handler to record stack traces and uploads core dumps to Azure blob storage for later inspection. But when the stack is corrupted, stack traces can be incomplete, misleading, or missing. Application logs could not reliably identify all matching crashes: queries produced both false positives and false negatives. Manual inspection found more examples, but it was too slow and subjective to build confidence.

The important methodological shift was to stop treating the crashes as isolated mysteries and start treating them like a population-level problem. OpenAI frames this as “core dump epidemiology”: instead of deeply studying only a few cases, the team built a higher-quality data set covering the broader population of crashes. That made it possible to look for patterns across machines, services, stack shapes, and failure signatures. In other words, they moved from anecdotal debugging to statistical debugging: classify many failures, compare them, and ask whether they cluster around specific hosts, libraries, or conditions.

That broader view revealed that what looked like a single bug was actually two unrelated problems discovered at roughly the same time. The first was silent hardware corruption on one Azure host. In plain terms, one machine’s CPU was producing incorrect results, meaning software running on it could fail even if the code was correct. Silent hardware errors are especially dangerous because they may not immediately announce themselves as hardware faults; they can instead look like bizarre software behavior.

The second problem was an 18-year-old race condition in GNU libunwind, a widely used open source library for walking call stacks. A race condition means the outcome depends on unlucky timing between concurrent operations. In this case, a long-hidden bug in a low-level runtime component could contribute to misleading or broken stack-unwinding behavior during crash handling. The root cause was not one dramatic flaw in Rockset’s application code, but a combination of rare infrastructure-level failures that only became visible through systematic crash analysis.

The story also highlights why reliability work in large AI systems extends far beyond model training and inference code. Modern AI products depend on data systems that can search, index, and retrieve information at scale while the model is responding. If those systems crash unpredictably, user-facing AI features become less reliable. Rockset’s query processing leaves are replicated, so a single segfault has limited client impact, but OpenAI still treats every crash as evidence of a bug that must be understood and fixed.

A key lesson is that low-level bugs often require a different kind of evidence than normal feature bugs. A normal defect might be reproduced with a test case, traced through logs, and patched in a small code path. Stack corruption, hardware faults, and runtime library races are harder because they can invalidate the debugging tools themselves. The team had to collect enough core dumps, classify them carefully, and compare cases across the whole crash population before the pattern became clear.

The broader takeaway is that debugging critical AI infrastructure increasingly resembles incident science: engineers need strong observability, preserved crash artifacts, and population-level analysis to separate application bugs from hardware faults and deep dependency issues. OpenAI’s account is less about a single clever fix and more about building the right evidence base. By doing so, the team could identify silent CPU corruption on an Azure host and fix a very old GNU libunwind race condition that had remained unnoticed in a major open source component for years.