Rebuilt the service comparison for batch ETL, buffered delivery, and stateful streams; corrected current names, runtime constraints, retries, and recovery guidance.
AWS data transformation is not one service choice. A pipeline may need a durable stream, a compute engine, and a delivery target, each with a different job. For batch work, AWS Glue and Amazon EMR run data-processing frameworks, while AWS Batch queues and schedules containerized jobs. For streaming, Amazon Kinesis Data Streams and Amazon MSK retain and distribute events; Lambda handles bounded event-driven code, AWS Glue Streaming and Amazon Managed Service for Apache Flink run stream-processing jobs, and Amazon Data Firehose buffers and delivers data.
Choose by the work you need to perform: file-oriented ETL, custom distributed compute, simple per-event changes, buffered delivery, or continuous stateful logic such as event-time windows. Latency, ordering, replay, failure handling, and cost follow from that choice.
Quick comparison: what each service does
Service
Primary role
Choose it when
Main trade-off
AWS Batch
Queues and schedules containerized jobs on managed compute
You have independent, resource-heavy jobs or existing containerized tools
You package and operate the job; queueing and capacity startup are not a real-time response path
AWS Glue
Managed data integration and ETL, including batch and Spark streaming jobs
You need repeatable transformations across supported data stores or streams
Streaming mode determines supported sources and operations; running jobs incur ongoing compute charges
Amazon EMR
Managed environments for frameworks such as Apache Spark and Hadoop
You need framework compatibility, runtime control, or broader cluster options
More control brings more configuration and operational responsibility
AWS Lambda
Runs event-driven code and bounded stream batches; Kinesis mappings also support tumbling windows
A small transformation can finish within the function and source limits
Retry, concurrency, ordering, and execution-time limits shape the design
Amazon Kinesis Data Streams
Retains an ordered stream for independent consumers and replay
You need a reusable stream with consumer-controlled processing
You design partition keys, capacity, consumers, and retention for the workload
Amazon Data Firehose
Buffers, optionally transforms, and delivers streaming data to destinations
You want managed ingestion and delivery without running a consumer application
Buffering and destination retries affect freshness; processing control is limited
You need the Flink programming model and direct control over stateful event-time processing
You design the Flink application, state, checkpoints, watermarks, and sink behavior
Amazon MSK
Managed Apache Kafka clusters for Kafka producers and consumers
Kafka protocol, clients, topics, or existing Kafka architecture are requirements
Provisioned clusters require capacity and Kafka operations planning; serverless has its own constraints and pricing
These roles can be combined. A stream is not a transformation engine, and a delivery service is not a general-purpose stream processor. For example, a Kinesis stream can retain events while a Flink application calculates rolling aggregates and a Firehose stream delivers a copy to Amazon S3.
AWS Batch is a job scheduler and managed compute service, not an ETL engine. You submit a job definition that describes a container, its command, resource needs, and permissions. Jobs wait in a queue until the scheduler can place them in an associated compute environment. That makes Batch useful for independent transformations that already run as containers, including custom binaries, scientific software, and long-running or parallel work.
How to use it for transformation
Store inputs and outputs in a system such as Amazon S3, then let the container read, transform, and write them. AWS Batch can manage compute capacity with Amazon EC2, AWS Fargate, or Amazon ECS Managed Instances, depending on the environment and job requirements. It can also use Spot capacity where interruption is acceptable. The job image and transformation logic remain yours to build and maintain.
Choose Batch when you need queueing, dependencies, resource-aware placement, or a familiar container runtime. For a conventional Spark ETL job where you want AWS to provide the data-processing runtime, start by comparing AWS Glue or Amazon EMR instead.
Queueing, retries, and performance
Batch is designed to make queued work run when suitable compute is available. Queue time and capacity startup make it a poor fit for a response that must happen as each event arrives. Bundle work when setup overhead is material, but measure the job and input pattern instead of assuming a universal minimum job duration.
Retry behavior can be configured for a job. A retry repeats the work, so write transformations to tolerate a job running again: use deterministic output paths, conditional writes, a run identifier, or a deliberate overwrite or upsert strategy. Spot interruptions and application failures need different handling; a retry policy cannot make a non-idempotent write safe.
Compute and operating trade-offs
Managed compute environments reduce the work of provisioning capacity, but you still choose resource limits, instance families or Fargate sizing, queue priorities, and any interruption tolerance. Watch queue wait time as well as job duration, failure reason, vCPU and memory use, and the cost of idle or overprovisioned capacity. See AWS's documentation on AWS Batch components and automated job retries.
Lambda is useful when a transformation is short, event-driven, and can run within the function's configured limits. It can validate a message, normalize a record, add a small enrichment, or route a batch. Lambda is compute: it does not retain a stream for consumers to replay.
Event source mappings and batches
For Kinesis Data Streams, DynamoDB Streams, and Kafka sources such as Amazon MSK, a Lambda event source mapping polls the source and invokes the function with batches. Batch size and batching-window settings affect both efficiency and how long records wait before invocation. The source mapping, not the function alone, governs polling and delivery behavior; see AWS's guides for Kinesis and MSK event sources.
Use this pattern for simple transformations that do not require substantial shared state across events. Lambda invocations are stateless by default, though Kinesis event source mappings support bounded tumbling-window aggregation with state passed between invocations. These windows use Kinesis arrival timestamps and do not provide general event-time processing. For broader stateful processing, compare Glue Streaming's Spark micro-batch mode with Managed Service for Apache Flink; the right fit depends on sources, latency, programming model, and recovery behavior.
Execution limits and scaling
Lambda scales with incoming work, but source pollers, function concurrency, downstream capacity, and account quotas still matter. The standard Lambda execution mode supports timeouts up to 15 minutes. Lambda Managed Instances supports timeouts up to 90 minutes for supported asynchronous and event-source mapping invocations, but a source can impose a lower ceiling. Check the current timeout guidance for the execution mode and source you plan to use before choosing Lambda for a long transformation. Increasing concurrency can also overload a database or API that the function calls.
Batching can reduce invocation overhead, but it is not a fixed latency guarantee. If a zero or short batching window is important, verify the exact source mapping settings and observe the end-to-end delay. Cold starts, polling, queueing, downstream calls, and retries all contribute to what a consumer experiences.
Retries, ordering, and duplicate effects
For stream event sources, a failed batch can be retried, and the default retry behavior can hold up later records on the affected shard or partition. Configure retry count and record age for the recovery time you can accept, enable partial batch responses where supported, and send discarded records to an on-failure destination when you need to investigate or replay them. AWS documents these failure and destination controls for Kinesis mappings and Kafka retry controls.
Assume a record may be processed again after a timeout or retry. Make side effects idempotent, for example by recording a stable event ID with the result, and monitor iterator age or consumer lag. Lambda's timeout limit is documented in the Lambda function timeout guide.
Kinesis Data Streams is a durable, ordered stream that separates event producers from consumers. It is useful when several applications need to read the same event flow independently, when consumers may fall behind temporarily, or when you need to replay retained records. Kinesis stores data for a configured retention period; it does not transform the records by itself.
Partition keys, ordering, and capacity
A partition key maps related records to the same shard, where consumers read records in the sequence assigned by the service. That sequence does not necessarily match the order in which producers created events: PutRecords can partially fail and allow later records to be accepted first. If strict write order for one key matters, serialize writes and use PutRecord with SequenceNumberForOrdering; see AWS's PutRecords ordering notes and PutRecord ordering parameter. Choose keys that match the ordering requirement while spreading load; a hot key can concentrate traffic and constrain throughput. Capacity can be provisioned or on-demand, and the correct choice depends on the traffic pattern and consumer design.
Each consumer has its own progress and may read at a different pace. You can attach Lambda for straightforward event processing, Glue for streaming ETL, or Managed Service for Apache Flink for custom stateful logic. Consumer lag, throttling, shard distribution, and retention headroom are useful operational signals. Do not use a single published latency or per-shard figure as a workload guarantee; limits depend on capacity mode, read method, record size, and consumer count.
Give consumers dedicated throughput
Standard Kinesis consumers share a shard's read throughput. Enhanced fan-out gives each registered consumer a dedicated read path and pushes records instead of requiring that consumer to poll. It can help when multiple applications read the same stream and shared throughput is a bottleneck, but it does not guarantee end-to-end latency. Include its additional consumer and data retrieval charges in the comparison; see AWS's enhanced fan-out guide.
Replay and recovery
Kinesis Data Streams retains records for 24 hours by default; you can extend the period to as long as 365 days, with additional charges above the default period. Retention lets a consumer resume from an earlier position while records remain available, but it is a recovery window rather than a permanent archive. See the current retention limits and behavior. If records must be kept longer, write them to durable storage such as S3 as a separate pipeline. A consumer checkpoint tracks progress, but it does not make writes to an external database exactly once; the sink still needs idempotent or transactional behavior.
When to choose a stream
Use Kinesis Data Streams when you need an AWS-native retained stream with multiple consumer applications and control over their processing. If your producers or consumers depend on Kafka APIs, topics, and ecosystem tooling, compare Amazon MSK. If you only need to ingest and deliver records to supported destinations, Amazon Data Firehose may need less application code.
Amazon Data Firehose, formerly Amazon Kinesis Data Firehose (the service rename), is a managed delivery service. Producers can write directly to it, or it can read from sources such as Kinesis Data Streams and Amazon MSK. Firehose buffers records and delivers them to configured destinations, including S3, Redshift, OpenSearch, and supported third-party endpoints. Depending on the destination and configuration, it can also convert formats, partition S3 output, and invoke Lambda to transform buffered batches.
This buffering is useful for landing data efficiently, especially in object storage, but it means Firehose is not a per-event processing engine. Buffer size and interval are delivery hints, and actual freshness also depends on throughput and destination behavior. Choose it when managed ingestion and delivery matter more than custom consumer control or stateful event-time logic. The Firehose guide describes its sources, destinations, and buffering.
Transform and handle delivery failures
A Lambda transformation attached to Firehose receives a batch and returns transformed records; use it for bounded record cleanup or format changes. If the transformation invocation fails, Firehose retries it according to its configured policy. Records that cannot be transformed or delivered need an explicit recovery path: enable the appropriate S3 backup or error output, monitor delivery and transformation errors, and test what happens when the destination is unavailable. See AWS's guidance for transforming source data and handling transformation failures.
Firehose does not replace a retained stream when consumers need to manage offsets, replay records independently, or maintain application state. Keep the original in a stream or archive it to S3 if those recovery options are requirements.
Amazon MSK runs Apache Kafka for applications that use Kafka's producer, consumer, topic, and client interfaces. It is a fit when compatibility with Kafka applications or the Kafka ecosystem is a real requirement. Kafka topics retain records according to their configuration, and consumers track their own positions; retention and replay still depend on the topic policy and available storage.
MSK offers provisioned and serverless cluster types. Serverless reduces broker-capacity management, while provisioned clusters give you more direct control over the cluster shape. Both still require decisions about partitions, networking, access, schemas, retention, monitoring, and consumer behavior. MSK is not automatically the best choice for every high-volume stream; use Kinesis Data Streams when its AWS-native model fits better and you do not need Kafka compatibility.
MSK can feed Lambda, Firehose, Glue streaming jobs, or Managed Service for Apache Flink. MSK Connect provides managed Kafka Connect connectors, but connector availability and behavior depend on the specific connector. See the MSK developer guide and MSK Serverless documentation.
AWS Glue is a strong starting point for managed transformations across data stores. Glue ETL jobs commonly use Apache Spark or Python shell runtimes, and Glue Studio provides visual authoring for supported jobs. AWS Glue for Ray is closed to new customers as of April 30, 2026; existing customers can continue to use it, as described in AWS's Glue for Ray lifecycle notice. The Data Catalog stores metadata about tables and schemas. A crawler can discover metadata, but it does not process the underlying rows and is optional when a job already knows its schema and source path.
Use Glue for repeatable cleansing, joins, format conversion, and loading curated data into destinations such as S3 or Redshift. Glue Streaming is built on Spark Structured Streaming and can read from Kinesis Data Streams and Kafka sources, including MSK. Its default micro-batch mode supports stateful operations such as aggregations and event-time windows and uses checkpoints to track stream progress. AWS Glue 6.0 also offers an opt-in real-time mode, but that mode is limited to stateless Kafka transformations written in Scala with a fixed worker count; it does not support Kinesis, stateful windows, or auto-scaling. Provide enough workers to cover the source partitions because AWS warns that unassigned partitions are not processed. Check the current Glue streaming execution modes for the exact requirements. A continuously running job has ongoing compute cost even when traffic is light.
Amazon EMR provides managed environments for open-source data frameworks such as Apache Spark and Hadoop. Choose EMR when you need a particular framework version, custom libraries or configuration, an existing Spark or Hadoop application, or deployment options such as EMR on EC2, EMR on EKS, or EMR Serverless. It can read and write data in S3 and other supported stores.
Glue and EMR can both run Spark workloads. Glue is usually simpler when the task is a managed ETL job and its supported runtime meets your needs. EMR gives you more choice over the framework environment and job deployment, with additional decisions about runtime, capacity, and operations. The EMR service guide covers its frameworks and architecture.
Amazon Managed Service for Apache Flink, formerly Amazon Kinesis Data Analytics for Apache Flink (renamed by AWS), runs continuously on streaming sources such as Kinesis Data Streams and Amazon MSK. Use it when the transformation depends on state across events: keyed counts, rolling aggregates, joins, sessions, or event-time windows. Glue Streaming can also run stateful Spark transformations in micro-batches. Flink is a better fit when the application needs the Flink programming model and fine-grained control over continuously maintained state, event-time windows, and recovery; compare source and sink support, latency needs, and team familiarity before choosing.
Event time, windows, and late records
Processing time is when the application sees a record. Event time is when the event occurred according to its timestamp. If events can arrive late or out of order, a watermark strategy tells Flink how far event time has advanced and when a window can be emitted. The allowed lateness and handling for very late records are application choices; they affect both result completeness and how quickly results become final. A timestamp from the source alone does not define a correct watermark strategy.
Checkpoints, recovery, and sink semantics
Flink checkpoints capture application state so the job can recover after a failure. Snapshots can also preserve state for planned changes. These mechanisms protect the processing state; they do not by themselves make every external side effect exactly once. End-to-end guarantees depend on the source replay behavior, checkpoint and restore configuration, and whether the sink supports transactions or idempotent writes. Treat retries and recovery as possible duplicate attempts unless the complete source-to-sink path provides the semantics you require.
Files or tables on a schedule: use Glue for managed ETL, EMR for more framework control, or Batch when the transformation is an arbitrary containerized program.
A short reaction to each event: use Lambda when a bounded function and its source mapping meet the required throughput and retry behavior.
Several independent consumers or replay: put events in Kinesis Data Streams or MSK, selected by whether the AWS-native stream model or Kafka compatibility fits the producers and consumers.
Managed delivery into a data lake or analytics destination: use Data Firehose when buffering is acceptable and its destinations or transformation hooks cover the pipeline.
State across events and event-time results: use Glue Streaming when its Spark mode, supported sources, and recovery model fit the workload; choose Managed Service for Apache Flink when its application model and APIs fit better.
Compare whole-pipeline cost
Compare the actual billing units for the selected services and Region: job runtime and compute, continuously running streaming capacity, stream throughput and retention, function duration and requests, destination writes, storage, and data transfer. A low price for one service does not account for the consumers, retries, staging, or monitoring around it. Check each service's current pricing page with a representative traffic pattern before estimating total cost: AWS Batch, Glue, EMR, Lambda, Kinesis Data Streams, Data Firehose, MSK, and Managed Service for Apache Flink.
Design for retries and replay
Most data pipelines retry work somewhere: a job scheduler can rerun a container, a Lambda event source can retry a failed batch, and a stream processor can restore from a checkpoint and read again. Retrying protects against transient failure, but it can repeat writes. Choose a stable event or job key, make sink operations idempotent where possible, and preserve failed or rejected records in a place you can inspect and replay.
Also distinguish delivery buffering from stream retention. Firehose buffers records before sending them to a destination; Kinesis Data Streams and Kafka topics retain records for consumers to read according to their retention settings. A system that only delivers to a destination may not provide the independent consumer offsets or replay window your recovery plan needs.
Conclusion
Start by deciding whether the work is a scheduled data set transformation or a continuous event flow. For batch ETL, compare Glue, EMR, and Batch by runtime and control needs. For streams, choose a source such as Kinesis Data Streams or MSK only when you need its retention, replay, or consumer model; use Lambda for simple bounded handlers, Data Firehose for buffered delivery, Glue Streaming for Spark-based stream ETL, and Managed Service for Apache Flink when its stateful application model fits the workload.
Then test the pipeline's real failure paths: late and duplicated events, throttled destinations, retries, checkpoint recovery, and replay after an outage. Make the output safe to write more than once, and size retention and buffering for the recovery time your team actually needs.
FAQs
How do I choose between AWS Batch and AWS Lambda for data transformation?
Use Lambda for bounded event-driven code that fits its execution and source limits. Use AWS Batch when you need queueing and managed compute for containerized jobs that may run longer, need more resources, or should run when capacity is available. Both can retry work, so make output writes safe to repeat.
How do Kinesis Data Streams and Amazon Data Firehose differ?
Kinesis Data Streams retains events for consumer applications to read independently and replay within the retention period. Amazon Data Firehose buffers records and delivers them to configured destinations, with optional transformations. Choose Streams for consumer control and replay; choose Firehose when managed delivery and buffering meet the need.
When should I use Amazon MSK or Managed Service for Apache Flink?
Use MSK when your producers or consumers require Apache Kafka compatibility and topic-based Kafka behavior. Use Managed Service for Apache Flink when its continuous application model and APIs suit your stateful processing needs; Glue Streaming also supports stateful Spark micro-batch work. They can be used together: MSK can supply events and Flink can transform them.
