M1REST job lifecycle
A POST submits a job and returns a job ID. A GET reads the status, and a DELETE cancels the job. The server holds the job state, so a client crash or restart does not kill or orphan the job. YARN, Trino, and Snowflake already expose HTTP APIs.
The YARN API only covers Hadoop-native jobs, so it could not run commands like aws s3 sync or hadoop distcp.
M2YARN Distributed Shell
Distributed Shell is an existing YARN application that runs a shell script inside a container. The spec sets the application type to MAPREDUCE. The container command starts the Distributed Shell ApplicationMaster class. Three environment variables give the S3 location, the length, and the timestamp of the script.
YARN then adds memory and vCore limits, container isolation, retries, clean cancellation, and logs in the YARN UI.
M3Quarry responsibilities
Quarry handles authentication, job submission, state tracking, lifecycle control, and observability. Lifecycle control covers clean cancellation and cleanup. Observability covers structured logs, metrics, and tracing. Airflow is the main client.
Teams no longer need SSH keys or special security groups to submit work.
M4Migration tracking and deprecation
An analytics dashboard queried the Airflow database to find remaining SSH tasks. The team retired operator types one at a time, starting with CrunchExecOperator and then S3SyncOperator. Each retirement was a small project. They restricted SSHOperator to block new use.
Teams in Search Infrastructure, Data Engineering and Analytics, and ML Services migrated jobs in parallel.
M5Regional rollout
The work was effectively eight parallel migrations. Each region needed its own Quarry configuration, endpoints, and routing rules. The team validated changes in a US pilot region first. Then they rolled out region by region and moved from Dev to GovDev and CommDev, and then to Prod.
Regions differed in network setup, data sovereignty rules, and cluster versions.