How Meta's VirtualLog Indirection Decouples Consensus Reconfiguration From Ordering

When Meta engineers need to upgrade the consensus protocol beneath a live database, they no longer have to schedule downtime or migrate data. Instead, a control plane issues a single command to a virtualized log, seamlessly swapping the underlying replication engine while the database continues serving reads and writes.[1][2]

This capability powers Delos, the storage system that manages Meta’s highly critical control plane state. Because control plane services manage the fleet’s metadata and server reservations, they cannot rely on other infrastructure without risking circular dependencies.[1][2]

Delos currently processes over 1.8 billion transactions per day across all deployments. It acts as the foundational layer for systems like Twine’s Resource Broker, which manages server reservations for the entire Facebook fleet.[1]

Historically, distributed databases have relied on monolithic consensus protocols like Paxos or Raft. These protocols entangle the data plane—the ordering of incoming commands—with the control plane, which handles leader election and membership reconfiguration.[1]

That entanglement makes consensus engines notoriously difficult to upgrade. Once a database is deployed on a specific protocol, it is effectively locked into that implementation's performance and fault-tolerance characteristics for perpetuity, preventing it from adopting newer research.

Separating Materialization From Ordering

To break this lock-in, Delos first separates the materialization of data from the ordering of transactions. Each Delos server maintains a local copy of the database state using RocksDB, a high-performance storage engine optimized for fast storage media.[1][2]

When a server needs to modify state, it does not communicate directly with other database replicas. Instead, it appends a new update to a shared log, and to access state with strong consistency, the server synchronizes its local RocksDB copy with that log.[2]

The shared log exposes a minimal API consisting of append, checkTail, and readNext operations. This design hides the complexity of distributed coordination, allowing the database's materialization logic to remain entirely oblivious to network asynchrony and node failures.[1]

However, relying on a single, monolithic shared log implementation would simply shift the lock-in problem down one layer. To achieve true flexibility, Meta engineers introduced a layer of indirection called the VirtualLog, moving beyond the theoretical promises of decoupled storage into a shipped production system.[2]

Virtualizing the Shared Log

The VirtualLog presents the illusion of a single, continuous shared log to the database application above it. Beneath this abstraction, it maps different segments of the log's virtual address space to pluggable ordering protocols called Loglets.[1]

A Loglet is essentially a simplified replication engine that handles the data plane, sequencing incoming appends and storing them durably. Because the VirtualLog handles the complex control plane operations, Loglets do not need to implement leader election or membership reconfiguration.[1]

By extracting those responsibilities, the fault-tolerance requirement for a Loglet drops dramatically. It no longer needs to be continuously highly available for appends; it only needs to provide a fault-tolerant seal operation, which is theoretically weaker and simpler to implement.[3]

If a Loglet fails or experiences a network partition, the system does not attempt to repair it or elect a new leader within that Loglet. Instead, the VirtualLog simply seals the degraded Loglet and routes all subsequent appends to a healthy one.

The Mechanics of Reconfiguration

The mapping between the VirtualLog's address space and the underlying Loglets is managed by a separate component called the MetaStore. The MetaStore acts as a highly available configuration registry, exposing a single versioned register that supports conditional writes.[1]

When Meta needs to upgrade a Loglet or recover from a failure, the VirtualLog initiates a three-step reconfiguration process. First, a client issues a seal command to the active Loglet, ensuring that no further transactions can be appended to that segment.[3]

Once the old Loglet is sealed, a call to checkTail returns the definitive end of that segment. Finally, the reconfiguring client writes a new chain configuration to the MetaStore, pointing the next segment of the address space to a new Loglet.

Because reconfiguration happens far less frequently than regular command sequencing, the MetaStore's consensus protocol can prioritize simplicity over out-and-out performance. The entire transition happens in tens of milliseconds, completely hidden from the database application.[1]

From ZooKeeper to NativeLoglet

This architecture allowed Meta to deploy Delos rapidly without waiting for a custom, highly optimized consensus engine. The initial production version shipped in just eight months, utilizing a rudimentary Loglet built as a shim over an external ZooKeeper service.[1][2]

Four months later, Meta engineers hot-swapped the ZooKeeper Loglet for a custom-built NativeLoglet. The VirtualLog sealed the ZooKeeper segment and seamlessly routed new traffic to the NativeLoglet, achieving a 10-fold improvement in end-to-end latency without a single second of database downtime.[1]

In production, Meta uses the NativeLoglet in a converged form, meaning it runs on the same machines as the database layer. It implements the seal operation by simply setting a bit on a quorum of servers, and uses a central sequencer to assign positions to commands.[1]

An append is considered globally committed once a majority of the LogServers acknowledge it. This streamlined approach strips away the overhead of traditional consensus, allowing the NativeLoglet to process transactions with minimal latency.

Composing and Striping Loglets

Because Loglets share a uniform, minimal API, they are highly composable. Meta engineers can stack and stripe them much like physical hard drives in a RAID array, dynamically altering the system's performance properties without changing the database code.[1][2]

For example, a StripingLoglet can shard incoming appends across multiple underlying Loglets. This dramatically increases the system's overall write capacity, allowing a single Delos instance to support over one million appends per second.[1][3]

Striping also enhances fault tolerance. By rotating the sequencer across different underlying Loglets, Delos can double the failure threshold of an instance without sacrificing throughput, ensuring that the control plane survives multiple simultaneous machine failures.[1]

The indirection also supports disaggregated deployments, where the Loglets run on an entirely different set of servers from the database. Switching to a disaggregated Loglet allows Delos to scale throughput by up to 10 times when compute and storage demands diverge.[1][2]

Surgical Editing and Outage Recovery

The VirtualLog's mapping capabilities extend beyond simple performance upgrades, providing operators with surgical editing tools during severe outages. Because the log is virtualized, engineers can alter written entries simply by changing the metadata in the MetaStore.[1]

During one site-wide incident, a malformed poison entry caused hangs on all database learners attempting to process the log. Using virtual consensus, Meta engineers bypassed the standard append-only restrictions and deleted the entry by remapping the VirtualLog around it.[1]

Similarly, older, sealed segments of the log can be transparently migrated to a Loglet layered on cold storage. This prevents the primary consensus nodes from filling up with historical data, reducing the hardware footprint required for the active replication group.[1]

Ultimately, the VirtualLog abstraction transforms consensus from a rigid, monolithic commitment into a modular, hot-swappable component. While the industry often hypes zero-downtime architectures that still rely on heavy backend migrations, Delos achieves it by isolating the data plane from the control plane, ensuring its foundational databases can evolve without dropping a transaction.[1]