Network service provider Tailscale has found a 16-year-old programming bug in the SQLite database after a months-long investigation. Under specific conditions, the bug caused data corruption and significantly disrupted Tailscale's network stability for months. What makes it especially frustrating: SQLite had been the primary database since 2022 and was considered proven and reliable, with no changes made to the code.
Corruption with no apparent cause
In August 2025, the S3 backup pipeline reported the first database corruption incidents, which added up to 19 cases over the following six months. Tracking down the cause was particularly difficult because no changes had been made to Tailscale's own code, and the bug couldn't be artificially reproduced.
Tailscale's internal architecture spreads networks across various instances, each managed by its own SQLite database. For backup purposes, the system created full snapshots every few minutes. Whenever corruption occurred, the affected instance had to be stopped and repaired, during which new devices couldn't connect to the network and administrators lost access to the web console.
A race condition dating back to 2010
While investigating, Tailscale worked closely with the SQLite developers and systematically ruled out common theories such as faulty POSIX locks or thread-safety violations. A newly implemented transaction pipeline provided the decisive clue: in two incidents, it showed that data already successfully written and confirmed simply vanished without a trace in subsequent transactions, with SQLite issuing no error message. The metrics also indicated that checkpoints were incorrectly copying more pages than existed in the write-ahead log.
To make the bug visible at all, the SQLite developers, with financial support from Tailscale, built a new logging tool for the virtual file system called tmstmpvfs. It recorded every write and copy operation during the checkpoint process. The next time the failure occurred, the logs provided proof of a rare race condition in SQLite's own source code, known as the WAL reset bug. It had existed undiscovered in the code since version 3.7.0 from July 2010, roughly 16 years.
Aggressive checkpointing as the trigger
The bug occurred when a write transaction happened at a critical moment during a checkpoint. That threw off the copy process, causing pages to be incorrectly marked as copied when they weren't. Tailscale ran into this trap mainly because the company managed checkpointing manually and very aggressively, a usage pattern many other SQLite users likely never reach in this form. The case shows how even decades-old, widely used software can still harbor deeply hidden bugs that only surface under very specific, intensive usage.