Anurag’s Read

From SSH to REST: A Security-Driven Modernization of Slack's EMR Data Pipelines.

A detailed migration story with rejected alternatives, a clever use of YARN Distributed Shell, and honest notes on hidden limits that SSH had hidden. The rollout and tracking approach applies to any large migration.
Score
8.6
Verdict
Must read
Blog and date
Slack,
Original length and level
15 min, advanced
Topics
migrations, data engineering, security, airflow
Read the original

Context

By 2024, Slack's data platform ran more than 700 Airflow jobs that reached AWS EMR clusters over SSH. An SSH operator opened a connection to the EMR master node and ran a command such as spark-submit. This widened the attack surface, forced key rotation, and made audit hard.

Jobs shared master node resources, and Kubernetes pod restarts broke sessions and left zombie jobs. SSH also blocked Spark on Kubernetes, EMR on EKS, and the Whitecastle move of AWS infrastructure into child accounts.

Architecture

Airflow operators now call Quarry, Slack's REST job gateway, over HTTP. Quarry authenticates callers with service-to-service tokens instead of SSH keys. Quarry submits work to YARN, Trino, or Snowflake and records job state on the server.

Spark jobs go through the Livy REST API, and Hive jobs go through HiveServer2. Shell-style jobs, including MapReduce and more than 300 CLI jobs, use YARN Distributed Shell. For these, the client uploads a script to S3, and Quarry sends a JSON application spec to the YARN ResourceManager.

YARN starts a container, downloads the script, and runs it.

Mechanisms

M1

REST job lifecycle

A POST submits a job and returns a job ID. A GET reads the status, and a DELETE cancels the job. The server holds the job state, so a client crash or restart does not kill or orphan the job. YARN, Trino, and Snowflake already expose HTTP APIs.

The YARN API only covers Hadoop-native jobs, so it could not run commands like aws s3 sync or hadoop distcp.

M2

YARN Distributed Shell

Distributed Shell is an existing YARN application that runs a shell script inside a container. The spec sets the application type to MAPREDUCE. The container command starts the Distributed Shell ApplicationMaster class. Three environment variables give the S3 location, the length, and the timestamp of the script.

YARN then adds memory and vCore limits, container isolation, retries, clean cancellation, and logs in the YARN UI.

M3

Quarry responsibilities

Quarry handles authentication, job submission, state tracking, lifecycle control, and observability. Lifecycle control covers clean cancellation and cleanup. Observability covers structured logs, metrics, and tracing. Airflow is the main client.

Teams no longer need SSH keys or special security groups to submit work.

M4

Migration tracking and deprecation

An analytics dashboard queried the Airflow database to find remaining SSH tasks. The team retired operator types one at a time, starting with CrunchExecOperator and then S3SyncOperator. Each retirement was a small project. They restricted SSHOperator to block new use.

Teams in Search Infrastructure, Data Engineering and Analytics, and ML Services migrated jobs in parallel.

M5

Regional rollout

The work was effectively eight parallel migrations. Each region needed its own Quarry configuration, endpoints, and routing rules. The team validated changes in a US pilot region first. Then they rolled out region by region and moved from Dev to GovDev and CommDev, and then to Prod.

Regions differed in network setup, data sovereignty rules, and cluster versions.

Design decisions

No.ChoseInstead ofWhy
D1Use YARN Distributed Shell for shell-style jobsA custom remote-execution wrapper serviceThe wrapper was too complex and needed custom security. Distributed Shell already existed and used standard REST and existing authentication.
D2Rely on protocols already in YARNRemote execution frameworks such as Ansible or SaltThey add new dependencies to maintain.
D3Reuse the existing YARN application typeA new YARN job type built from scratchIt needed too much build and upkeep work.
D4Migrate one operator type at a timeEach operator became a small project, which lowered risk.

Key numbers

  • More than 700 jobs across 7 operator types.
  • 8 data regions and 5 teams took part.
  • The migration took 3 quarters from pilot to 100%.
  • More than 300 jobs were CLI based and used Distributed Shell.
  • The 80% milestone came during the OKR-driven phase.
  • The authors report two years in production since the change.

Failures and limits

  • Jobs that ran under SSH failed YARN virtual memory checks, because SSH had bypassed YARN limits.
  • The team set yarn.nodemanager.vmem-check-enabled to false on all clusters, since physical memory limits suffice.
  • A search task moved to a staging analytics cluster timed out when it connected to sts.amazonaws.com on port 443.
  • That cluster sat in a stricter network segment, and the dependency was hidden from the job configuration.
  • The team moved search tasks to a dev ETL cluster with correct routing and scaled it up.
  • The post gives no before and after rates for job failures or latency.

Lessons you can use

  • Build migration-progress monitoring before you migrate.
  • Test resource limits against the old SSH baseline during the pilot.
  • Map network topology and account boundaries early, because they hid problems until late.
  • Test in every environment tier, since account-boundary issues appeared only in some.

Glossary

EMRAmazon Elastic MapReduce, a managed service for Hadoop and Spark clusters.
YARNThe Hadoop resource manager that starts and limits containers for jobs.
LivyA REST service that submits Spark jobs.
Zombie jobA job that keeps running after its client connection has dropped.
Airflow operatorA reusable task type in an Airflow workflow.