Anurag’s Read

TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads.

Shows two generations of a hot/cold storage design with the reasons the first one was replaced. Strong on cost trade-offs, Parquet layout choices and safe automation of data lifecycle steps.
Score
8.8
Verdict
Must read
Blog and date
Netflix,
Original length and level
13 min, advanced
Topics
storage, cost optimization, cassandra, system design
Read the original

Context

The TimeSeries Abstraction stores temporal event data for Netflix teams, and some datasets hold many petabytes. Hot Cassandra storage gives low-millisecond reads, but it costs a lot. Old data gets few reads, and dense nodes add operational work.

Architecture

The control plane splits each dataset into time slices and records them in a metadata table. A slice becomes immutable after its write boundary passes. The first design streamed events through Kafka and Flink into Parquet files on S3.

The final design runs ROCS compaction on an immutable slice, then backs it up to S3. After safety checks, a backup memtable serves reads from that backup.

Mechanisms

M1

Streaming cold tier

Flink tasks wrote Parquet files, with one task group for each hash bucket of time series IDs. Row groups were 4 to 8 MB, and events were sorted by time inside each group. EVCache held Bloom filters and file metadata, so the planner skipped S3 reads.

M2

ReadOnlyCompactionStrategy

LCS and TWCS can leave SSTables that overlap in token space. One partition then spans several files, and each read fans out. ROCS compacts each immutable slice once and aligns the SSTables to token ranges.

M3

Backup memtable

Cassandra 4.1 supports pluggable memtables. The team set a custom type, backupmemtable, on each cold table with a timestamp parameter. The timestamp marks when the backup fully covers the table. Each S3 snapshot has a manifest that lists every SSTable and its component files.

The node downloads all components except the large Data file. It then plans reads locally and fetches Data ranges from S3.

M4

Initialization and gates

Initialization runs once for each table on a separate thread pool. It retries with exponential backoff and checks file sizes against the manifest. A .timestamp marker file restarts the work if the backup timestamp changes. Reads move to S3 only if the sstable_tasks virtual table shows no active compaction and complete_backups shows every node uploaded.

Design decisions

No.ChoseInstead ofWhy
D1Read the existing disaster recovery backupsKafka and Flink streaming of data to S3Streaming cost a lot and widened on-call work. The backups already exist, so there is no new transfer cost.
D2Small Parquet row groupsThe recommended Parquet row group sizeReads are point and range lookups, so small groups reduce the bytes read.
D3A separate Cassandra cluster for cold readsCold reads on the hot clusterS3 reads use much more CPU, and a spike could hurt hot traffic.

Key numbers

  • Cassandra cold storage holds over 15 PB of compressed data.
  • Cold reads reach thousands each second in each region.
  • The p90 latency improved about 30% over the earlier cold storage, and p99 stays under about 500 ms.
  • The daily compaction job took about 12 hours on the largest datasets.

Failures and limits

  • The post names no production incident.
  • Larger compression chunks reduce S3 calls, but they over-fetch on small reads.
  • The node keeps only index and metadata files, so each read still goes to S3 for data.

Lessons you can use

  • Check whether the data already exists in another form before you build a new copy.
  • Isolate a costly new read path from the hot path.

Glossary

SSTableAn SSTable is an immutable file where Cassandra stores sorted data on disk.
MemtableA memtable is the in-memory structure that Cassandra writes to before it flushes to disk.
Row groupA row group is a block of rows in a Parquet file that a reader can fetch alone.