Notes from compaction hell

LSM trees are a great default until they aren’t. We spent a quarter pushing on RedDB’s storage engine under a workload that turned out to be antagonistic to the defaults — and a quarter is roughly how long it took to learn which knobs mattered. This post is the version of those notes that we wish someone had handed us in week one.

The shape of the problem matters, so: the workload was mixed OLTP (~80% point reads, ~15% range scans, ~5% writes), values ranging from 200 bytes to 64 KiB with a long tail, and a working set roughly 3× larger than RAM. Steady-state throughput was fine. The pain was the latency tail during major compactions: P99 spikes from ~4 ms to ~180 ms, lasting 30–90 seconds, repeating every few hours.

The Grafana screenshot we kept staring at

Imagine a P99 latency panel with a flat green floor at 4 ms and four red mountains per hour reaching 180 ms. The mountains are not random — they line up exactly with lsm.compactions.in_progress going from 0 to 1, with disk.read.bytes_per_sec doubling, and with the OS page cache hit rate dropping 8 percentage points. The peaks themselves are not the worry; the duration is. A 30-second window of bad tail latency, four times an hour, means a non-trivial fraction of customer requests get caught.

The first three things we tried did nothing useful.

What did not help

Reaching for tiered compaction at the first sign of write amp. Tiered compaction trades read amp for write amp. Our workload was read-heavy. Switching strategies made the steady-state P50 worse without flattening the tail. We reverted in a week.

Tuning per-level fan-out without first looking at value sizes. We spent a sprint sweeping level_multiplier from 8 to 16 and the only meaningful signal was noise. The actual problem was that compaction was moving 16 KiB values around to keep the keys sorted, and the keys weren’t the part that hurt.

Adding more compaction threads when the bottleneck was disk, not CPU. iostat -x 1 was showing the SSD pinned at 95% utilization during compactions; CPU was sitting at 30%. Throwing more threads at it made the tail worse because the threads contended for I/O queue depth that the device couldn’t honor.

The lesson is dull but real: measure before turning knobs. We had Prometheus metrics for compaction count but not compaction byte volume, and not the breakdown between key and value bytes. Adding those was the single most valuable hour of the quarter.

What did help

1. Per-level bloom filter sizing

The default bloom configuration used 10 bits per key globally. That’s a defensible default — false positive rate around 1% — but the L0/L1 levels are on the hot path for every point read, and the cost of a false positive there is a full SSTable read on the wrong file.

[lsm.bloom]
# 10 bits/key was the global default. Hot levels benefit from more.
level_0 = 16  # FPR ~0.04%, costs ~6 bits/key of RAM per level-0 file
level_1 = 14  # FPR ~0.16%
level_2 = 12  # FPR ~0.61%
default = 10  # FPR ~1%, for cold levels where bloom RAM matters more

Effect on P99 point-read latency at steady state: 4.2 ms → 3.1 ms. Effect on RAM for bloom filters: +18%. Worth it.

2. Separating large values from the key-ordered storage

The biggest single win. Anything over 8 KiB now gets written to a blob area and the SSTable only stores a pointer. Compaction stops moving cold image bytes around to keep numeric IDs in sorted order. Background write volume during major compaction dropped 4.7×.

The configuration:

[lsm.blob]
enabled = true
min_value_size_bytes = 8192
gc_threshold_ratio = 0.5  # rewrite blob file when 50% of its bytes are dead

Effect on P99 during compaction: 180 ms → 42 ms. This is the change that actually made the on-call complaints stop. If you remember nothing else from this post: blob separation for any workload with a meaningful tail of large values.

3. Backpressure on the write path

A slow client is uglier than a buggy one. When the LSM gets behind — when L0 has more files than level0_slowdown_writes_trigger — we now intentionally inject delay into write acknowledgements rather than letting L0 grow unbounded. Letting L0 grow unbounded leads to a cliff: eventually compaction can’t keep up at all, reads start touching dozens of files, and the engine becomes unrecoverable without operator intervention.

[lsm.write_pressure]
level0_slowdown_writes_trigger = 20  # start adding delay
level0_stop_writes_trigger = 36      # hard stop, return WriteThrottled
soft_pending_compaction_bytes_limit = "64GiB"
hard_pending_compaction_bytes_limit = "256GiB"

The client sees a WriteThrottled error sooner under stress, which is a clear signal callers can react to (retry with backoff, queue, shed load). The alternative — silent unbounded growth until the engine collapses — is much worse for everyone.

4. Compaction priority by score, not by level

The default scheduler prefers compacting whichever level is most over-target. That’s fine in isolation but it ignores the fact that some files have many more delete tombstones than others, and compacting those is a much better use of I/O budget than evenly trimming each level. We added a tombstone-density signal to the compaction priority score:

fn compaction_priority(file: &SSTable) -> f64 {
    let size_pressure = file.size_bytes as f64 / file.target_size_bytes as f64;
    let tombstone_factor = 1.0 + 3.0 * file.tombstone_ratio.min(1.0);
    let age_factor = 1.0 + 0.5 * (file.age_seconds() / 86_400.0).min(2.0);
    size_pressure * tombstone_factor * age_factor
}

Read amp on tables with heavy churn dropped noticeably. The exact gain depends on your write pattern, but for our workload — where a few hot tables had ~40% tombstone density — point reads on those tables went from touching 4–6 SSTables to typically 1–2.

Concrete numbers, before and after

The table below is steady-state during business hours, averaged across our three largest production clusters, with the same workload and the same hardware.

Metric	Before	After	Notes
Read P50	1.8 ms	1.4 ms	Mostly the blob change keeping the page cache useful
Read P99 (no compaction)	4.2 ms	3.1 ms	Per-level bloom sizing
Read P99 (during major compaction)	180 ms	42 ms	Blob separation is the bulk of this
Write P99	6.0 ms	5.4 ms	Mostly unchanged; backpressure rarely triggers
Bytes compacted per hour (peak)	24 GiB	5.1 GiB	Blob bytes don’t move with keys anymore
RAM for bloom filters	1.1 GiB	1.3 GiB	The tax for the bloom change
WriteThrottled errors per day	0 (engine fell over instead)	~12	A clean error is better than a cliff

The P99-during-compaction row is the one we care about most. 42 ms is not great, but it does not generate customer-visible incidents. 180 ms did.

Tiered compaction: revisited, and still wrong for us

After the other changes landed, we tried tiered again. With blob separation pulling the large-value cost out of compaction, the write-amp argument for tiered is much weaker. With per-level bloom sizing, the read-amp penalty of tiered is more visible because the rest of the system is now faster. Tiered lost worse the second time than the first.

If your workload is write-dominated with small values, the conclusion is probably opposite. We didn’t have that workload, so we don’t have credible numbers for it.

What we still don’t love

The latency tail during major compactions is honest, not pretty. 42 ms P99 is a number we can live with operationally — it doesn’t trip SLOs — but it’s a number, not a zero. The next move on our list is a “compaction budget” that limits how much I/O bandwidth compaction is allowed to consume during business hours and lets it catch up overnight. That’s a different post; we haven’t shipped it yet.

The other thing we haven’t solved: snapshot isolation under heavy compaction load is fine for correctness but causes a small extra read amp because old SSTable generations have to stick around for readers. We’ve measured it at ~3% of total read I/O. It’s the kind of thing you only see if you go looking.

The one knob to turn first

If you remember one configuration change from this post, make it blob separation. Every workload we have ever profiled benefits, and the ones with even a small fraction of large values benefit dramatically. The other three changes are worth doing, but they are increments. Blob separation is the cliff.

The order we wish we had done it in: blob separation, observability for compaction byte volume by file, per-level bloom sizing, backpressure thresholds, score-based priority. We did it roughly backward, which is why it took a quarter instead of a sprint.