Delta-of-Deltas Timestamps and XOR Floating-Point Encoding: How Meta's Gorilla Compresses Time Series

Time-series compression only works if time is predictable. If a sensor fires at random intervals, the mathematical tricks that shrink data to a fraction of its size fall apart. But in modern infrastructure, telemetry is ruthlessly periodic.[3]

A server logs its CPU usage exactly every 60 seconds, and a network switch reports its throughput on a rigid cadence. That predictability is what Meta exploited in 2015 to build Gorilla, an in-memory time-series database. The system squeezed 16-byte data points down to an average of 1.37 bytes.[1]

The 12-fold compression ratio allowed Meta to keep a massive 26-hour window of operational data entirely in memory. While Gorilla itself remained an internal tool, the algorithms it introduced became the blueprint for the entire database industry.[1][3]

The 16-Terabyte Memory Problem

By the spring of 2015, Meta's monitoring systems were generating more than two billion unique time series. The infrastructure added about 12 million new data points every second, totaling over one trillion points per day.[1]

Each data point consisted of a 64-bit timestamp and a 64-bit floating-point value, consuming 16 bytes. Storing a 26-hour rolling window of that data in memory would require 16 terabytes of RAM, which was too resource-intensive for a practical deployment.[1]

The engineering team needed a way to compress the data without sacrificing the ability to query it in tens of milliseconds. They recognized that operational telemetry has a unique shape that general-purpose compression algorithms like Gzip fail to exploit.[1]

"Our insight is that users of monitoring systems do not place much emphasis on individual data points but rather on aggregate analysis," the engineering team wrote in their 2015 VLDB paper.[1]

Because recent data points are far more valuable for diagnosing an ongoing outage, the system was optimized for high-speed ingestion and fast reads over the most recent window. To make it fit, they split the compression into two distinct streams.[1]

Gorilla was designed as a write-through cache sitting in front of a slower, disk-based HBase cluster. It buffered the incoming stream, compressed it on the fly, and served the high-speed queries directly from RAM.[1]

The architecture prioritized availability over strict consistency. If a Gorilla node crashed before flushing its compressed buffers to the persistent storage tier, a few seconds of telemetry would be permanently lost.[1]

The engineering team accepted that trade-off because missing a few seconds of CPU metrics is harmless, whereas an unresponsive monitoring dashboard during a severe outage is catastrophic.[1][3]

Storing the Gap Between Gaps

A standard timestamp is a 64-bit integer representing the number of milliseconds since the Unix epoch. Storing that full integer for every single reading wastes an enormous amount of space, because the numbers are highly repetitive.

The first optimization is delta encoding, which stores the difference between the current timestamp and the previous one. If a feed samples every 60 seconds, the database simply writes a delta of 60,000 milliseconds instead of a massive epoch integer.[2]

Gorilla took this a step further by calculating the delta of the deltas. On a steady cadence, the gap between readings does not change, meaning the difference between the current delta and the previous delta is exactly zero.[1][2]

When the delta-of-deltas is zero, Gorilla encodes the entire timestamp as a single zero bit. In Meta's production environment, about 96 percent of all timestamps compressed down to that single bit, effectively eliminating time from the storage footprint.[1]

When the predictability breaks slightly, the algorithm uses a variable-length encoding scheme. If the delta-of-deltas falls between negative 63 and 64 milliseconds, Gorilla writes a two-bit control sequence followed by a 7-bit value.[1][2]

Larger deviations trigger wider buckets. A gap between negative 255 and 256 milliseconds requires a three-bit prefix and 9 bits, while a gap up to 2,048 milliseconds takes 12 bits. Anything larger forces the system to write the full 32-bit integer.[1][2]

Because the vast majority of telemetry data arrives on time, those larger buckets are rarely used. The system effectively turns time into a structural assumption, only spending disk space when the schedule is violated.[1][3]

Canceling Out Floating-Point Noise

Compressing the actual measurements required a different approach, because floating-point numbers do not compress well with standard delta math. Instead, Gorilla uses an exclusive OR bitwise operation to compare the current value with the previous one.[1][2]

Infrastructure metrics change slowly. If a server's memory utilization moves from 73.2 percent to 73.4 percent, the binary representations of those two floating-point numbers are almost identical.[2]

The XOR encoding relies on the IEEE 754 standard for floating-point numbers, where the sign, exponent, and mantissa are packed into 64 bits. Because adjacent telemetry values share the same exponent and upper mantissa, the XOR operation zeros them out.[1][3]

When the database XORs those two similar numbers, the identical bits cancel out, leaving a result filled with leading and trailing zeros. Gorilla simply counts the zeros, discards them, and stores only the meaningful bits in the middle.[1][2]

When a value does change, Gorilla writes a control bit, followed by 5 bits to declare the number of leading zeros, and 6 bits to declare the length of the meaningful data block.[1]

The system then writes only those meaningful bits to the block. If the next value in the series has the exact same number of leading and trailing zeros, the algorithm skips the length declarations entirely and just writes a control bit followed by the data.[1]

If the value has not changed at all, the XOR result is entirely zero, which the system encodes with a single bit. Across Meta's vast telemetry landscape, this XOR technique reduced the 64-bit float to an average of just over one byte.[1]

When Predictability Breaks Down

The Gorilla algorithms are highly specialized, and their efficiency degrades rapidly when the underlying data behaves erratically. The system relies entirely on the assumption that timestamps arrive on schedule and values drift slowly.[3]

If a network delay causes a timestamp to jitter by a few milliseconds, the delta-of-deltas is no longer zero. The database must fall back to the variable-length encoding buckets, spending extra bits to store the unexpected gap.[1][3]

Similarly, if a metric spikes wildly from one reading to the next, the XOR operation fails to produce a long run of zeros. In those cases, the database is forced to store the full 64-bit value, losing the compression benefit entirely.[1][2]

The system also requires data to arrive in chronological order. Because each compressed point is mathematically derived from the one immediately preceding it, inserting a late-arriving data point into the middle of a compressed block requires rewriting the entire sequence.[1][3]

The compression operates strictly within a single time series. Gorilla does not attempt to find patterns across different servers or different metrics, because interleaving multiple streams would destroy the predictable deltas the algorithm relies on.[1]

This makes the system vulnerable to high cardinality. If an engineering team tags their metrics with unbounded labels like unique user IDs or request hashes, the database must create millions of separate compressed blocks, exhausting the memory overhead.[3]

The Blueprint for Modern Telemetry

Despite those constraints, the Gorilla paper fundamentally changed how the software industry handles time-series data. The 12-fold compression ratio proved that in-memory telemetry was economically viable at a massive scale.[1][3]

Today, the delta-of-deltas and XOR encoding techniques are the standard across the observability landscape. Open-source time-series databases like Prometheus, InfluxDB, and QuestDB all implemented variations of the Gorilla algorithms in their core storage engines.[2]

The open-source database QuestDB recently integrated Gorilla's delta-of-deltas encoding directly into its binary wire protocol. By compressing the timestamps before they even cross the network, the system dramatically increases ingestion throughput.

InfluxDB utilizes the same XOR floating-point compression in its storage engine, though it pairs it with different integer compression strategies like ZigZag encoding for the timestamps.

By shrinking a 16-byte data point down to a fraction of its size, these systems allow modern engineering teams to monitor millions of containers and microservices simultaneously. The mathematical tricks Meta published in 2015 remain the foundation of that visibility.[1][3]