EMR on EKS

Overview

EMR on EKS lets you run Apache Spark jobs with the EMR runtime on an Amazon EKS cluster, which is a managed Kubernetes service. Instead of creating a separate EMR cluster of virtual machines for analytics, you submit jobs to a Kubernetes namespace. The jobs run as pods, next to other applications that share the same worker nodes.

The main reason to choose it is consolidation. If a company already operates Kubernetes, analytics can share the same compute pool, the same deployment tools, the same monitoring and the same security model as services and machine learning workloads. For teams that prepare and score tables for models, including the tabular foundation models described on /model/research, it offers a way to run large Spark jobs on the same platform that hosts model serving.

What it is

Amazon EMR normally provisions its own clusters. EMR on EKS changes the unit of deployment. There is no EMR cluster to create. The service acts as a job submission layer that talks to an EKS cluster you already have.

An analogy: a large shared office building. Classic EMR is like renting a private floor for each project. EMR on EKS is like booking meeting rooms in a building that other teams also use. You get what you need for the time you need it, and the building is kept busy.

The main pieces

• EKS cluster: the Kubernetes control plane and worker nodes, managed by you.
• Namespace: a logical partition inside the Kubernetes cluster where jobs run.
• Virtual cluster: an EMR object that maps to one namespace. It is the target you submit jobs to.
• Job run: one submitted application, such as a Spark job.
• Job execution role: the identity that the job uses to reach data and services.
• EMR release: the version of the EMR runtime container image for Spark.
• Pod templates: files that customize how driver and executor pods are created.
• Spark operator or submitter: an alternative way to submit Spark jobs through Kubernetes tools.

What you get compared with open-source Spark on Kubernetes

You could run plain Spark on Kubernetes yourself. EMR on EKS adds:

• The EMR optimized Spark runtime, packaged as container images.
• A managed job API with job states, logs and retries.
• Integration with the Glue catalog and with Lake Formation for governed tables.
• Integration with role based access to data through service accounts.
• Managed endpoints for interactive notebooks in some setups.

What you still manage

You operate the EKS cluster: node groups, scaling, networking, upgrades and add-ons. Kubernetes skills are needed.

Where it came from

Spark added native Kubernetes support as an alternative to YARN. In the beginning it was experimental. It matured over several releases and became a common way to run Spark in cloud native shops.

Many organizations moved their services to Kubernetes and wanted one platform for everything. Running a separate Hadoop style cluster for analytics felt like a second infrastructure to maintain. At the same time, teams did not want to give up the EMR runtime features and the managed job experience.

EMR on EKS was introduced as a bridge: keep the EMR runtime and APIs, but let the compute live on Kubernetes. It stands next to the cluster based form and the EMR Serverless form as a third way to run EMR workloads.

How it works

Step 1: you prepare an EKS cluster

You need an EKS cluster with worker nodes, either managed node groups, self-managed nodes or serverless pod capacity. You also need networking that can reach storage and the catalog, plus add-ons such as a cluster autoscaler or a node provisioning tool.

Analogy: an office building with elevators and power already working.

Step 2: you create a namespace and set up access

A namespace isolates the jobs of a team or environment. The service needs permission to create pods in that namespace. A role mapping on the Kubernetes side allows the EMR service identity to act there.

Step 3: you register a virtual cluster

A virtual cluster links to the namespace. It holds no machines. It is a pointer that tells EMR where jobs should run. You can have several virtual clusters on one EKS cluster, for example one per team or one per stage.

Step 4: you define job execution roles

Each job assumes a role that grants only the access it needs, such as reading specific S3 prefixes and the catalog. This uses service account to role mapping, so the pods receive temporary credentials without static keys.

Step 5: you submit a job run

You call the API with the virtual cluster, the release label, the execution role, the job driver details (script and arguments), and configuration overrides. The service creates a Spark driver pod. The driver asks Kubernetes for executor pods.

Step 6: Kubernetes schedules the pods

The Kubernetes scheduler places pods on nodes with enough CPU and memory. If there is no room, pods wait as pending. An autoscaler can add nodes in response, which takes some time.

Analogy: guests arrive at the building, and the front desk finds free rooms, or opens a new floor if all are full.

Step 7: the job runs

Executors read data from S3, process partitions and write results. Shuffle data goes to local disks or other configured storage. Logs go to the log destination you set, such as S3 or a monitoring stream.

Step 8: the job ends and pods go away

When the job finishes, executor pods are removed and the driver ends. Capacity returns to the pool. The job run record remains with its state and timing.

Step 9: scale in

If you use an autoscaler, empty nodes are removed after a delay. This is how analytics and other workloads share and release capacity.

Key concepts and terms

A worked example

A team runs a daily Spark job that builds model-ready shards from curated tables.

Prepare a namespace and virtual cluster

kubectl create namespace analytics-jobs

aws emr-containers create-virtual-cluster \
  --name example-analytics-vc \
  --container-provider '{
    "id": "example-eks-cluster",
    "type": "EKS",
    "info": {"eksInfo": {"namespace": "analytics-jobs"}}
  }'

The command output includes a virtual cluster ID that you use in later calls. Before this step, the service identity must be mapped to a role in the cluster so it can create pods in the namespace.

A pod template for executors

# executor-template.yaml
apiVersion: v1
kind: Pod
spec:
  nodeSelector:
    workload: analytics
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "analytics"
      effect: "NoSchedule"
  containers:
    - name: spark-kubernetes-executor
      resources:
        requests:
          cpu: "3"
          memory: "12Gi"
        limits:
          memory: "12Gi"

This template places executors on nodes labeled for analytics and reserves memory. Store it in object storage so the job can reference it.

Submit the job

aws emr-containers start-job-run \
  --virtual-cluster-id vc-example-0001 \
  --name daily-shards \
  --execution-role-arn arn:aws:iam::000000000000:role/example-emr-eks-job-role \
  --release-label emr-7.0.0-latest \
  --job-driver '{
    "sparkSubmitJobDriver": {
      "entryPoint": "s3://example-code/build_shards.py",
      "entryPointArguments": ["2026-01-15"],
      "sparkSubmitParameters": "--conf spark.executor.instances=10 --conf spark.executor.memory=10g --conf spark.executor.cores=3"
    }
  }' \
  --configuration-overrides '{
    "applicationConfiguration": [{
      "classification": "spark-defaults",
      "properties": {
        "spark.kubernetes.executor.podTemplateFile": "s3://example-code/executor-template.yaml",
        "spark.dynamicAllocation.enabled": "true",
        "spark.dynamicAllocation.shuffleTracking.enabled": "true"
      }
    }],
    "monitoringConfiguration": {
      "s3MonitoringConfiguration": {"logUri": "s3://example-emr-logs/"}
    }
  }'

The release label and sizes are examples. Dynamic allocation with shuffle tracking lets Spark add and remove executors without an external shuffle service, which is helpful on Kubernetes.

The job script

# build_shards.py
import sys
from pyspark.sql import SparkSession, functions as F

day = sys.argv[1]
spark = SparkSession.builder.appName(f"shards-{day}").getOrCreate()

df = spark.read.parquet("s3://example-lake-curated/customer_daily/").filter(F.col("feature_date") == day)

num_cols = [c for c, t in df.dtypes if t in ("double", "bigint", "int", "float")]
stats = df.select([F.avg(F.col(c).isNull().cast("int")).alias(c) for c in num_cols]).first().asDict()
keep = [c for c in num_cols if stats[c] < 0.5]

(
    df.select("customer_id", *keep)
    .repartition(16)
    .write.mode("overwrite")
    .parquet(f"s3://example-lake-model/shards/day={day}/")
)
spark.stop()

This script keeps numeric columns with less than half of their values missing and writes 16 shards. The null share check is a simple example of a profile based decision.

Check the job run

aws emr-containers describe-job-run \
  --virtual-cluster-id vc-example-0001 \
  --id 000000000example

Fields such as state, failure reason and timestamps help in monitoring. If the job fails, inspect the driver logs first.

A numeric walk-through of sharing capacity

Suppose the Kubernetes cluster has 20 nodes with 16 cores each, 320 cores total. Services use around 120 cores during the day and fewer at night. At night, a Spark job asks for 10 executors with 3 cores each, which is 30 cores, plus a driver. The scheduler finds room on existing nodes, so no extra nodes are needed. A larger monthly job asks for 100 executors, 300 cores. That would not fit alongside daytime services, so it is scheduled at night, or the autoscaler adds nodes. The key point is that capacity bought for services is reused for analytics when idle.

A second example: namespace quota and limit range

Isolation on a shared cluster starts with quotas. A quota caps the total resources that jobs in a namespace can request, so a runaway job cannot take the whole cluster.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: analytics-quota
  namespace: analytics-jobs
spec:
  hard:
    requests.cpu: "200"
    requests.memory: 800Gi
    limits.memory: 900Gi
    pods: "400"

A limit range sets defaults for pods that do not declare their own values.

apiVersion: v1
kind: LimitRange
metadata:
  name: analytics-defaults
  namespace: analytics-jobs
spec:
  limits:
    - type: Container
      default:
        memory: 4Gi
      defaultRequest:
        cpu: "1"
        memory: 4Gi

With these two objects in place, a job that asks for too much simply waits or is rejected, and services in other namespaces keep working.

A second example: separating driver and executor placement

Drivers must survive for the whole job, while executors can come and go. Use two pod templates to express this.

# driver-template.yaml
apiVersion: v1
kind: Pod
spec:
  nodeSelector:
    capacity-type: on-demand
  containers:
    - name: spark-kubernetes-driver
      resources:
        requests:
          cpu: "2"
          memory: 8Gi

# executor-spot-template.yaml
apiVersion: v1
kind: Pod
spec:
  nodeSelector:
    capacity-type: spot

If a spot node disappears, only the executors on it are lost. Spark recomputes their tasks on other executors. If the driver were on that node, the application would end.

Reading pod states during a run

Most failed jobs on Kubernetes can be understood from three views: the job run state in the EMR API, the pod events in Kubernetes, and the Spark driver log. Look at them in that order.

Image and version management

The runtime is delivered as container images. Treat them like any other image.

• Pin a release label and test upgrades in a separate virtual cluster.
• If you add libraries, build a custom image on top of the base image and scan it.
• Keep images in a registry close to the cluster so pulls are fast.
• Use a small, stable set of images instead of one per team.

Custom images are handy for Python libraries that a model scoring job needs. Keep the layer with those libraries small so pods start quickly.

Interactive work

Data scientists often want notebooks that run on the same cluster. Managed endpoints can provide interactive sessions that attach to a virtual cluster. Keep these in a separate namespace with tight quotas, and set idle timeouts so that forgotten sessions release capacity.

Strengths and limitations

Strengths

• One platform for services, machine learning serving and analytics.
• Reuses existing Kubernetes capacity, tooling and monitoring.
• EMR optimized Spark runtime and a managed job API.
• Namespace and virtual cluster isolation for teams and environments.
• Fast job start compared with creating a new cluster of virtual machines, when nodes are already available.
• Fine control through pod templates, node selectors and tolerations.
• Role mapping for pods avoids static credentials.
• Works with the Glue catalog and with Lake Formation governed tables.
• Supports multiple Spark versions side by side through release labels.

Limitations

• You must operate EKS: upgrades, networking, add-ons and node scaling.
• Kubernetes skills are required, and debugging pods differs from debugging YARN.
• Scheduling delays occur when new nodes must be added.
• Shuffle heavy jobs need careful local disk and memory planning.
• Noisy neighbors: analytics pods can disturb services if requests and limits are not set well.
• Not every EMR cluster feature exists in the same form, since there is no HDFS or YARN.
• It mainly targets Spark and some related workloads, not every Hadoop ecosystem tool.
• The extra layers can make cost accounting harder without good tagging.

How it compares with alternatives

The decision often comes down to who runs what. If a platform team already runs Kubernetes and wants one pool, EMR on EKS fits. If nobody wants to run infrastructure, a serverless form fits better. If the team needs special Hadoop ecosystem tools, cluster based EMR may be needed.

Role in tabular foundation models

A tabular foundation model learns from raw tables with rows and columns, mixed types, missing values and often no helpful column names. See /model/architecture for its structure and /model/benchmarks for evaluation. EMR on EKS is relevant because the data work around such a model is large, and Kubernetes is also where model serving and training jobs often live.

One cluster for data prep and model serving

The model may be served by pods on the same Kubernetes cluster. Spark jobs that score a full table can call the model service over the cluster network, which has low latency and no public internet hop. Resource quotas per namespace keep batch scoring from starving the serving pods. This is a concrete advantage over a separate analytics cluster in another network.

Building training corpora

Jobs can read many tables from S3, normalize types and nulls, drop unusable tables, de-duplicate rows and write Parquet shards, as described for Amazon EMR. The same code runs on Kubernetes. A namespace for corpus building with its own quota keeps heavy runs isolated.

Synthetic table generation

Pre-training on synthetic tables needs a huge number of generated tables, each created from a random process with its own seed. Each table is independent, so generation parallelizes well as many small Spark tasks or as many independent pods. Kubernetes handles this style naturally, and spot nodes lower cost because lost tasks can simply be rerun with the same seed.

Row and column structure, mixed types, missing values

Jobs preserve column order and types, and write explicit nulls. They can compute per-column profiles, such as null share, distinct count and type guess, and write them as side tables. These profiles feed decisions about imputation and about which columns to include in a shard.

In-context learning episodes

A model that uses in-context learning is trained on episodes, each a small table split into context rows and query rows. Spark on Kubernetes can build millions of these episodes in parallel and write them as grouped files. Keep episodes grouped into fewer large files so training reads are efficient.

Imputation and anomaly detection at scale

For batch imputation, a job scans partitions, finds missing cells, sends batches to the model service and writes filled values along with a flag that marks them as imputed. For anomaly detection, it writes a score per row. Output partitions can be registered in the AWS Glue Data Catalog so other tools find them.

Continual learning

When new tables arrive, a scheduled job extracts, profiles and shards them. A workflow tool such as Apache Airflow can trigger job runs through the API, then a training job on the same cluster. The cluster can scale up for the nightly run and down by morning. Record the job run IDs and data snapshots to know which data each model version saw.

Sector and semantic classification

Column feature extraction, such as value pattern counts and length statistics, is a natural Spark job. The same cluster can host the classifier service. Results are written back as catalog properties, and AWS Lake Formation tags can then drive access rules.

Governance and isolation

Per-team namespaces, execution roles and Lake Formation integration let you give a model pipeline narrow access to only approved tables. Network policies in Kubernetes can limit which pods may call the model service or reach data stores. Add data lineage records for traceability.

Practical guidance and best practices

Cluster and node setup

• Create dedicated node groups for analytics with labels and taints, so Spark pods land on suitable nodes.
• Use instance types with enough memory per core for shuffle heavy work.
• Use spot nodes for executors and on-demand nodes for drivers, since a lost driver ends the job.
• Use an autoscaler that reacts to pending pods, and set sensible maximum node counts.
• Keep the Kubernetes version and add-ons current and tested with your EMR release.

Job configuration

• Set CPU and memory requests equal to or close to limits to avoid eviction surprises.
• Account for memory overhead beyond the executor heap.
• Use dynamic allocation with shuffle tracking, or a remote shuffle option, so executors can scale.
• Choose executor sizes that pack well on nodes. Odd sizes waste cores.
• Place driver pods on stable nodes.
• Use pod templates for node selection, tolerations and sidecar settings.

Storage and shuffle

• Use fast local storage for shuffle on nodes.
• Watch for pods evicted due to disk pressure.
• Keep durable data in S3, not in pod volumes.
• Compact small files after large writes.

Security

• Use dedicated execution roles per job type with least privilege.
• Map roles to Kubernetes service accounts, never embed keys.
• Use namespace level resource quotas and limit ranges.
• Apply network policies to restrict traffic.
• Encrypt data at rest and in transit, and use private endpoints where available.

Observability

• Send Spark and pod logs to durable storage.
• Collect pod metrics together with Spark metrics, so you can see CPU, memory and shuffle for each job.
• Tag job runs with team and project for cost reports.
• Alert on long pending times, which indicate lack of capacity.

For model pipelines

• Run batch scoring in its own namespace with a quota, so it cannot starve the serving pods.
• Use a concurrency limit when calling model endpoints from executors.
• Write results to versioned prefixes, then promote.
• Keep the job code, config and input snapshot recorded for every run.

Common mistakes

1. No resource requests: pods are scheduled badly and disturb neighbors.
2. Running drivers on spot nodes: a reclaimed node ends the whole application.
3. Ignoring memory overhead: executors get killed for exceeding limits even though heap looks fine.
4. No autoscaler tuning: pods wait long for nodes, or nodes are over-provisioned.
5. Sharing one namespace for everything: quotas, roles and logs become tangled.
6. Static credentials in images or configs: use role mapping instead.
7. Treating pods as durable: local volumes vanish when pods end.
8. Too many small executors: scheduler pressure and overhead rise.
9. Skipping network policy: any pod can call any service.
10. Version mismatch: add-ons and Kubernetes versions that the runtime image does not expect cause strange failures.
11. Heavy batch jobs during peak service hours: user facing latency suffers.
12. No cost tags: shared clusters make it hard to see who spent what.

Frequently asked questions

What is EMR on EKS?

It is a way to run Spark jobs with the EMR runtime on an Amazon EKS Kubernetes cluster. You submit jobs to a virtual cluster that maps to a namespace, and the jobs run as pods on your existing worker nodes.

How is it different from regular EMR?

Regular EMR creates and manages its own clusters of virtual machines. EMR on EKS has no separate cluster. It uses your Kubernetes cluster for compute, so analytics shares infrastructure with other applications.

What is a virtual cluster?

A virtual cluster is an EMR object that points to a Kubernetes namespace. It does not own any machines. It is the target you use when you submit job runs.

Who manages the Kubernetes cluster?

You do. The managed Kubernetes service runs the control plane, but you manage node groups, scaling, networking, upgrades and add-ons. EMR on EKS handles the job submission and runtime, not the cluster itself.

Can jobs share a cluster with web services?

Yes, and that is the main idea. To do it safely, set resource requests and limits, use namespaces with quotas, use node selectors or taints where needed, and schedule heavy batch jobs when services are quiet.

How do jobs get permissions to S3?

Each job uses an execution role mapped to a Kubernetes service account. The pods receive temporary credentials for that role, so there are no static keys in code or images.

Can I use spot capacity?

Yes. Executors are a good fit for spot nodes because Spark can recompute lost tasks. Drivers should run on on-demand nodes, because losing a driver ends the application.

Does it support the Glue Data Catalog and Lake Formation?

Yes. Spark jobs can use the Glue Data Catalog as the metastore, and governed tables can be read under Lake Formation rules with supported configurations. Check the current documentation for your release.

When should I choose EMR Serverless instead?

Choose the serverless form when you do not want to operate infrastructure and your jobs are intermittent. Choose EMR on EKS when you already run Kubernetes and want to share capacity and tooling.

How does it help with tabular model workloads?

It lets data preparation, batch scoring and model serving live on one platform. Jobs can build training shards, create in-context episodes, impute missing values and score anomalies, while the model service runs nearby with isolation through namespaces and quotas.

Glossary

• Kubernetes: an open-source system for running containers across machines.
• EKS: the managed Kubernetes service on AWS.
• Pod: the smallest unit that Kubernetes schedules.
• Namespace: a logical division inside a Kubernetes cluster.
• Virtual cluster: the EMR pointer to a namespace.
• Driver: the Spark process that plans and coordinates work.
• Executor: a Spark process that runs tasks.
• Pod template: a file that customizes pod specs.
• Service account: an identity for pods inside Kubernetes.
• Execution role: the cloud role a job uses.
• Taint and toleration: a pair of rules that keep or allow pods on certain nodes.
• Node selector: a label rule for placing pods.
• Autoscaler: a component that adds or removes nodes.
• Dynamic allocation: Spark feature that varies executor count.
• Shuffle tracking: Spark feature that tracks shuffle data so executors can be released.
• Resource quota: a limit on resources used in a namespace.
• Spot capacity: discounted capacity that may be reclaimed.
• Release label: the version of the EMR runtime.
• Episode: a small table split into context and query rows for training.
• Tabular foundation model: a model pre-trained on many tables to work on new ones.

Summary

EMR on EKS runs Spark jobs with the EMR runtime on a Kubernetes cluster you already operate. A virtual cluster maps to a namespace, job runs become driver and executor pods, and role mapping gives them access to data. It shines when a platform team wants one pool of capacity for services, machine learning and analytics, and it asks for Kubernetes skills and careful resource settings in return.

For tabular foundation models, it is a practical home for the data side of the work and for the serving side too. Spark jobs build corpora and episodes, profile columns, impute missing cells and score anomalies, while the model service runs nearby with quotas and network policies. Versioned outputs, orchestration and catalog registration support continual learning and traceability. Read next about Amazon EMR, EMR Serverless, Kubernetes and Apache Spark to compare the options.

Related

It belongs to AWS data analytics in my stack, under Data engineering and lakehouse. Others in the same group: