Anurag’s Read

The foundations of Canva's continuous data platform with Snowpipe Streaming.

A cost-driven migration with clear numbers, a rejected option, and frank notes on workarounds such as one client per shard. Useful for anyone who runs a high-volume data pipeline.
Score
8.6
Verdict
Must read
Blog and date
Canva,
Original length and level
11 min, advanced
Topics
data platform, cost optimization, streaming, migrations
Read the original

Context

Canva grew from about 60 million to over 200 million monthly users in five years. The data platform handled about 25 billion records each day. Producers wrote to Kinesis Data Streams (KDS), and AWS Data Firehose landed files in S3 for Snowflake to load.

Firehose bills each record in 5 KB steps. It grew to nearly 50% of platform cost. Using KDS as the Firehose source added about 30% on top.

Architecture

Producers still write records to KDS. A Kinesis Client Library (KCL) application reads the shards. Each shard maps to one Snowpipe Streaming channel, and each channel belongs to a client.

The client buffers rows and flushes them to S3 in an intermediate format. Snowflake then migrates this data into its optimized table format. A RecordProcessor polls its channel for the latest committed offset token and reports it to the KCL as the checkpoint.

Mechanisms

M1

Channel and offset token

A channel is a logical stream inside a client. It holds an offset token, which records the highest committed position. The insert call is asynchronous, so the processor cannot assume a row is safe after the call returns. The processor polls the committed offset token and passes it to the KCL as the checkpoint.

M2

Shard stealing and invalidation

The KCL balances load by letting new workers steal shard leases. The handoff is not coordinated, so two processors can read one shard for a short time. This can move an offset in an unsafe way, and Snowflake then invalidates the channel. Channels share compacted blobs on a client, so one invalid channel invalidates every channel on that client.

KCL record processors only move forward, so they cannot rewind to the last committed token.

M3

One client per shard

The team gave each shard its own client. A stolen shard then invalidates only its own channel. The cost is a higher per-client fee, more threads with duplicated I/O, and smaller batches that compress worse.

M4

Unhealthy processor workaround

The KCL cleans up stolen channels but cannot restart other invalid channels. A processor that finds its channel invalid marks itself unhealthy and drops its lease through the LeaseCoordinator. Another worker then takes the shard. The team calls this a hack.

Design decisions

No.ChoseInstead ofWhy
D1Switch Firehose from a KDS source to Direct PUTClient-side aggregation and compression before FirehoseDirect PUT saved about 30% with no new format. Aggregation would force every downstream reader to know the format and decompress it.
D2Snowpipe Streaming into SnowflakeFirehose with the Snowflake destinationThe Firehose Snowflake destination costs about three times the S3 destination. Snowpipe Streaming also gives a write-once path.
D3Use the KCL to read KinesisA custom connector on the Kinesis APIsThe KCL was faster to deliver. The team accepted its limits.

Key numbers

  • Firehose reached nearly 50% of platform cost.
  • KDS as a Firehose source added about 30%, and Direct PUT removed it.
  • Snowpipe Streaming charges 0.01 credits per client per hour.
  • The system has run over 6 months and moved over 20.35 PiB.
  • Time to query is about 10 minutes.
  • Cloud spend fell by 45%.

Failures and limits

  • Shard stealing invalidated channels, and one bad channel took down all channels on its client.
  • The KCL cannot rewind a processor, so channels invalidated for other reasons stayed in a zombie state.
  • The lease-drop workaround is fragile and not recommended in general.
  • One client per shard raises cost and lowers compression.

Lessons you can use

  • Read the billing unit of each managed service, because a 5 KB minimum can dominate the bill for small records.
  • Test the failure behavior of a client library under rebalancing before you depend on it.
  • Choose the faster integration first and write down the gaps it leaves.

Glossary

Offset tokenA value the producer sets on a channel to mark the last row Snowflake has committed.
Shard stealingThe KCL practice of letting a new worker take a shard lease from another worker.
Direct PUTA Firehose mode where producers send records straight to Firehose with no Kinesis stream between.