AWS Lambda now supports queues for Apache Kafka in self-managed Kafka event sources

AWS Lambda now supports queues for Apache Kafka in self-managed Kafka event source mappings (ESM), for Kafka clusters that customers run themselves and for managed offerings such as Confluent Cloud. Customers can now scale processing for Kafka workloads beyond the partition count of a topic, increasing throughput during traffic spikes and improving processing resilience.

Customers building event-driven Kafka workloads for use cases such as image and video encoding, fraud detection, notification delivery, and telemetry ingestion treat each event as an independent unit of work, where processing speed matters more than strict ordering. Previously, Kafka ESMs used the Kafka consumer group protocol, which assigns each partition to exactly one consumer within a consumer group. Customers needing higher parallelism had to over-partition topics or move traffic to a separate queuing system, adding infrastructure to operate and producer code to maintain. Queue support for self-managed Kafka ESMs is built on Kafka share groups, which are generally available in Apache Kafka 4.2. This launch enables multiple event pollers to consume concurrently from the same partition, with each event poller supporting up to 10 concurrent invocations. This decouples parallelism from partition count with no changes to producers, topic configuration, or function code. As a result, customers get higher throughput during traffic spikes and more resilient processing, because a slow or failed event no longer holds up other events in the same partition. Queue mode does not preserve partition ordering and is intended for workloads that do not require it.

This capability requires a Kafka cluster with share groups enabled, such as Apache Kafka 4.2 or later, and is available in all AWS commercial Regions where self-managed Kafka ESM is available. To get started, create a self-managed Kafka ESM with ConsumptionMode set to Queue through the AWS Management Console, Lambda API, AWS CLI, AWS CloudFormation, AWS SAM, or AWS CDK. Standard Lambda pricing applies. Customers who configure provisioned mode for their self-managed Kafka ESM also pay for Event Poller Unit (EPU) usage. To learn more, see the AWS Lambda Developer Guide and AWS Lambda pricing.