AutoMQ

non-cncf

First Research Pass

GitHub Repository Official Website

Content Lifecycle & Evolution State

1

Initial Seed

Metadata Loaded

2

First Research

Automated Gathering

3

Tech Writing

Content Structured

4

Approval

Editorial Verified

First Research Pass Completed: Initial facts, key features, and documentation links have been gathered. Detailed technical writing and synthesis are currently in queue.

Overview

AutoMQ is a cloud-native, diskless alternative to Apache Kafka® on S3 or any S3-compatible storage. It is designed to offer 10x cost savings, scale in seconds, and eliminate cross-AZ traffic costs while maintaining 100% Kafka compatibility.

Key Features

Stateless broker architecture leveraging object storage (S3) instead of local disk storage.
100% Apache Kafka® compatible (retention of Kafka API, protocol, and ecosystem).
Autoscale in seconds without data rebalancing.
Zero cross-AZ traffic costs due to its storage-compute separation and proxy layers.
Built-in auto-balancer for automatic partition scheduling.
Table Topic feature for automatic Kafka topic to Iceberg table conversion (Zero-ETL).

Use Cases

Cloud-native data streaming, event-driven architectures, replacing traditional stateful Kafka clusters to reduce operational costs, real-time analytics, scaling for unpredictable traffic spikes, and integrating streams into data lakes.

Getting Started

Run AutoMQ locally using Docker (`curl -O https://raw.githubusercontent.com/AutoMQ/automq/refs/tags/1.5.5/docker/docker-compose.yaml && docker compose -f docker-compose.yaml up -d`) or deploy on Kubernetes for production via helm charts.

Recent Updates

Introduced Table Topic, a new feature combining stream and table functionalities to unify streaming and data analysis (supports Apache Iceberg and integrates with catalog services like AWS Glue and S3 tables).

Did You Know?

AutoMQ replaces Kafka’s classic shared-nothing architecture with a shared storage architecture using a custom S3 Storage Adapter. According to their performance benchmarks, AutoMQ can speed up partition reassignment by 300x compared to Apache Kafka, cutting an operation that traditionally took 12 minutes down to just 2.2 seconds. It also boasts a 200x efficiency improvement during catch-up reads. While AutoMQ emphasizes cost-savings and cloud scalability, evaluators often compare it with more established Kafka distributions and managed offerings.