Anurag’s Read

S3 Files and the changing face of S3.

The team shows how a design effort failed when it tried to blend two models, and how an explicit boundary solved the problem. The post lists the hard parts, such as atomic renames and permission checks, and the real numbers of the final system.
Score
8.7
Verdict
Must read
Blog and date
AWS,
Original length and level
25 min, advanced
Topics
storage, system design, file systems, consistency
Read the original

Context

Customers keep data in S3 but run tools that expect a filesystem. A genomics lab at UBC ran GATK4 on a shared NFS filer. It wanted S3 plus serverless compute that scales to zero.

Teams copied data between S3 and a POSIX filesystem by hand, which the author calls data friction. The same pattern appears in media, ML, chip design, and science. S3 already added S3 Tables for structured data and S3 Vectors for embeddings.

S3 Files adds a filesystem view of a bucket.

Architecture

A customer mounts an S3 bucket or prefix on an EC2 instance, container, or Lambda function. Amazon EFS provides the file layer and holds changes. The file layer and S3 stay separate, and a sync layer links them.

On first access to a directory, the system imports its metadata from S3 and builds the namespace in the background. File writes collect in the EFS layer and commit to S3 as one PUT about every 60 seconds. Changes made through the S3 API flow back into the file view.

Large sequential reads bypass NFS and use parallel S3 GET requests.

Mechanisms

M1

Stage and commit

The team found that the file and object boundary is a real semantic boundary. Files allow in-place edits, fine-grained concurrent visibility, and mmap. Objects are immutable, use whole-object PUTs, and signal change by events with at-least-once delivery. The team borrowed the git model.

Changes build up in the file layer and push to S3 as a unit. The boundary is explicit and can change later.

M2

Import and hydration

The mount imports metadata per directory on first access. Files under 128 KB load their data during import. Larger files load data lazily on read. A mount can start with millions of objects.

File data not read for 30 days leaves the file view, though it stays in S3. This keeps cost close to the working set.

M3

Conflict handling

S3 is the source of truth when both sides change the same file. The file version moves to a lost+found directory. A CloudWatch metric records the event. The file side uses NFS close-to-open consistency, and the object side keeps strong consistency with atomic PUTs.

Each side keeps its native semantics.

M4

Permissions and naming

File permissions live on inodes and are checked at open. S3 uses IAM policy on key prefixes. The design stores permissions on the mount and enforces them in the file layer. IAM remains a backstop at the S3 layer.

Object keys that cannot be valid POSIX names stay out of the file view. The system emits an event for each one.

M5

Read bypass

For high-throughput sequential reads, the client routes reads to parallel S3 GET requests instead of NFS. The author reports about 3 GB/s per client. Throughput scales to terabits per second across many clients.

Design decisions

No.ChoseInstead ofWhy
D1Separate file and object layers with a sync boundaryOne unified file and object system, code-named EFS3Every unified design forced one side into a lowest-common-denominator compromise.
D2Permissions on the mount, IAM as backstopPOSIX and IAM permissions on every objectThe two models differ too much to co-represent on one object.
D3Skip non-portable names and emit an eventFailing or blocking those names, or forcing a common naming structureEither option would break assumptions that existing apps on one side rely on.
D4Multiphase access with a sync layerFull convergence of consistency rulesMost real workloads run in phases, and convergence would break existing S3 applications.

Key numbers

  • S3 sends over 300 billion event notifications per day.
  • S3 holds exabytes of Parquet and serves over 25 million requests per second to it.
  • S3 Tables has over 2 million tables.
  • Commits to S3 happen about every 60 seconds.
  • Files under 128 KB hydrate on import, and data evicts after 30 days without access.
  • A warning appears above 50 million objects per mount.
  • Read bypass reaches about 3 GB/s per client.

Failures and limits

  • S3 has no rename, so a directory rename copies and deletes every object under the prefix.
  • The fixed 60-second commit window offers no control at launch.
  • Concurrent edits resolve in favor of S3 and move the file version to lost+found.
  • Some keys never appear as files.
  • The team dropped the idea of objects that act as both files and directories after about 20 minutes.
  • The customer beta lasted about nine months.

Lessons you can use

  • Keep two different data models separate and make the boundary explicit.
  • Give each side its native consistency instead of one weaker shared rule.
  • Pick a clear source of truth for conflicts and record the loser.
  • Make mount scope small, because rename cost and object count grow with prefix size.
  • Use a lazy load for large data and an eager load for small data.

Glossary

HydrationLoading file data from S3 into the file layer.
Close-to-open consistencyAn NFS rule where a client sees the latest writes when it opens a file after another client closes it.
mmapA system call that maps a file into process memory for direct reads and writes.
lost+foundA directory that holds file versions set aside after a conflict.