How Fencing Tokens Prevent Split-Brain Writes in Distributed Locks

Fencing tokens prevent split-brain data corruption by attaching a strictly increasing sequence number to every lock acquisition, allowing the underlying storage system to reject writes from clients whose leases have expired. When a system experiences a stop-the-world garbage-collection pause, a client can freeze, lose its lock, and wake up believing it still holds exclusive access.[1]

This vulnerability exists in any distributed architecture that relies on time-bound leases for mutual exclusion. A lock server grants a client permission to modify a shared resource for a specific duration, such as 30 seconds.[2]

If the client crashes, the lease expires, preventing the resource from being locked forever. However, time-to-live expirations introduce a fatal flaw when combined with the realities of modern runtime environments.[1]

Applications written in languages like Java or Go periodically halt all execution threads to reclaim memory. These garbage-collection pauses usually last 5 to 50 milliseconds, but under heavy load, they can stretch for 120 seconds or more.[1]

The split-brain write hazard

During a severe pause, the client process is completely suspended and cannot communicate with the network. The lock server observes the silence, assumes the client has failed, and allows the 30-second lease to expire.[3]

The lock is then safely granted to a second client, which begins processing the same task. The catastrophic failure occurs when the first client's garbage-collection pause finally ends.[1]

From the perspective of the suspended process, no time has passed, and it remains entirely unaware that its lease was revoked. It resumes execution exactly where it left off and issues a write command to the database.[1]

Because the second client is also actively writing to the same resource, the system enters a split-brain state. Two independent processes now hold what they both believe to be exclusive access to a single piece of data.[5]

"The lock has a timeout, which is always a good idea, otherwise a crashed client could end up holding a lock forever," distributed systems researcher Martin Kleppmann noted in his foundational 2016 analysis. "However, if the GC pause lasts longer than the lease expiry period, it may go ahead and make some unsafe change."[1]

Enforcing sequence with fencing tokens

The lock service itself is powerless to prevent this collision. By the time the delayed client wakes up and transmits its write request, the lock server is no longer involved in the transaction.[3]

The client communicates directly with the storage layer, which has no native awareness of distributed lock ownership. This architectural blind spot means that adding more robust consensus protocols to the lock server does not solve the underlying race condition.[1]

Whether the lock is managed by a single instance or a highly available cluster, the storage engine remains vulnerable to delayed packets. The protection must be enforced at the destination.[1]

To bridge this gap, engineers implement fencing tokens—a strictly monotonic sequence number generated by the lock service. Every time a client successfully acquires or renews a lock, the service returns an integer that is guaranteed to be higher than any previously issued token.[1]

The client is required to append this fencing token to every write request it sends to the shared storage system. The storage engine, in turn, maintains a high-water mark recording the largest token it has successfully processed.[5]

Generating monotonic sequences

This simple integer comparison forms an impenetrable barrier against stale operations. When the first client wakes up from its extended garbage-collection pause, it attempts to write using its original, lower token, such as 33.[1]

The storage system compares this incoming number against the higher token, such as 34, already submitted by the second client. Recognizing the sequence violation, the database outright rejects the delayed write.[1]

This mechanism shifts the ultimate responsibility for data integrity from the volatile lock lease to the durable storage layer. The lock server coordinates the sequence, but the database enforces the physical boundary.[3]

"You should implement fencing tokens," the official Redis documentation states regarding distributed locking patterns. "This is especially important for processes that can take significant time and applies to any distributed locking system."[2]

Not all distributed lock implementations natively support the generation of strictly increasing tokens. Standard Redis setups, for example, require developers to manually script an atomic increment operation alongside the lock acquisition.[2]

The limits of time-based assumptions

Dedicated consensus systems are explicitly designed to provide these monotonic guarantees out of the box. Apache ZooKeeper assigns a 64-bit sequential transaction identifier, known as a zxid, to every state change.[3]

Clients can seamlessly extract this identifier and use it as a cryptographically safe fencing token. Similarly, the etcd key-value store, which underpins Kubernetes cluster coordination, tracks every modification through a 64-bit global revision number.[4]

When a client acquires a distributed lease in etcd, the creation revision of that specific key serves as an automatically incrementing token that satisfies the fencing requirement.[4]

Hazelcast implements this pattern directly within its FencedLock API, abstracting the sequence generation away from the application logic. The framework replicates the lock state across a consensus group and automatically increments the token each time the lock transitions.[5]

"FencedLock tracks liveness of lock holders via a session mechanism that works in a unified manner," Hazelcast engineers explained in a technical breakdown of the feature. "It allows third-party systems to participate in the locking protocol."[5]

Implementing storage-side checks

The necessity of fencing tokens exposes a fundamental truth about distributed systems: wall-clock time is an illusion. Relying on a 30-second lease assumes that 30 seconds on the lock server perfectly matches 30 seconds on the client.[1]

In reality, clock drift and CPU scheduling make time highly subjective. A hypervisor managing virtual machines can freeze an entire guest operating system to migrate it to a different physical host.[1]

Network latency introduces identical hazards without requiring a process pause. A client might generate a write request well within its valid lease window, only for the packet to get trapped in a degraded network switch.[1]

In a documented incident at GitHub, network packets were delayed in transit for approximately 90 seconds before being delivered. If a system relied purely on a standard 30-second lock lease, those delayed packets would have silently overwritten newer data.[1]

For strict correctness, the storage layer must actively participate in the fencing protocol. Relational databases like PostgreSQL implement this easily by adding a last-token column to the target table, allowing the application to include a conditional clause in its update statement.[1]

If the query affects zero rows, the client immediately knows its token was rejected and its lock has expired. By removing time from the equation and relying entirely on strictly ordered sequences, fencing tokens allow distributed architectures to maintain absolute consistency.[6]