Prometheus TSDB Compaction Failed: The 64 GiB Index Ceiling and What It Says About Your Cardinality

Tue, 21 Jul 2026 09:00:00 -0500

Prometheus compaction had been running fine for months. Then this showed up in the logs, and kept showing up every couple of hours:

text
level=ERROR source=db.go:1219 msg="compaction failed" component=tsdb err="compact [/var/lib/prometheus/metrics2/01KJQ22FD0C2VFGQC2T8QP70P2 /var/lib/prometheus/metrics2/01KK8B5EK0CPXWDGPRD3H89ZQP /var/lib/prometheus/metrics2/01KKSP6NZ0EYNDGJFJTD3D6N6F]: populate block: add series: write series data: \"/var/lib/prometheus/metrics2/01KKSSF1T0T0T3AN0PJ00WVQGJ.tmp-for-creation/index\" exceeding max size of 64GiB

add padding: \"/var/lib/prometheus/metrics2/01KKSSF1T0T0T3AN0PJ00WVQGJ.tmp-for-creation/index\" exceeding max size of 64GiB"

Searching that string gets you almost nothing: a handful of GitHub issues, no runbook, no explanation. So here is the explanation. Short version: this is a hard limit in the TSDB index file format, not a resource problem. You cannot tune it, add RAM to it, or upgrade past it. It is real and it needs work, but it is not a data-loss event, and the most likely root cause is cardinality. Specifically, the number of distinct series that existed anywhere inside the merged block’s time range, which is a very different number from your current head cardinality.

TSDB on Support Tools

Prometheus TSDB Compaction Failed: The 64 GiB Index Ceiling and What It Says About Your Cardinality