Apache Kafka Embraces Cloud-Native Architecture with Tiered Storage

Apache Kafka is undergoing a fundamental architectural shift away from local disk dependency toward a cloud-native, tiered storage model. This evolution aims to decouple compute from storage, enabling significant cost reductions and improved scalability for modern enterprise workloads.

The transition marks the end of an era where Kafka relied strictly on bare-metal optimizations. By leveraging object storage and separating state from processing, organizations can now achieve greater flexibility without sacrificing performance.

Key Facts

• Architectural Shift: Kafka is moving from a shared-nothing, local-disk design to a tiered storage architecture that utilizes remote object storage.
• Cost Efficiency: Enterprises report up to 90% reduction in infrastructure costs by eliminating the need for high-performance local SSDs on every broker node.
• Discover Financial Case: Discover Financial Services migrated its credit card settlement environment to this new model, handling millions of daily transactions via Amazon EMR.
• Latency Maintenance: Despite using slower remote storage for older data, single-digit millisecond latency is preserved for recent hot data through intelligent caching mechanisms.
• KIP-421 & KIP-570: These key Kafka Improvement Proposals formalize the design changes, allowing brokers to offload log segments to cloud storage automatically.
• Operational Simplicity: The new model simplifies cluster management by removing the complexity of balancing disk space across heterogeneous hardware configurations.

The End of Bare-Metal Optimization

For over a decade, Apache Kafka’s legendary performance stemmed from a specific design choice. It wrote sequential append-only logs directly to local disks. This approach minimized latency by avoiding network overhead during writes. It also leveraged the operating system’s page cache for rapid reads. However, this model was built for a world where servers had fast, local solid-state drives (SSDs). It assumed a static hardware environment where capacity and performance were tightly coupled. Migrating this architecture "as-is" to the cloud exposed severe financial inefficiencies. Cloud providers charge a premium for high-IOPS block storage. Maintaining large clusters of expensive NVMe drives solely for Kafka’s log retention became unsustainable. The rigid coupling of compute and storage meant that scaling one often required scaling the other unnecessarily. This led to wasted resources and inflated operational expenditures for companies running large-scale event streaming platforms. The industry needed a solution that could leverage the elasticity of the cloud while retaining Kafka’s core strengths. The answer lay in rethinking how data persistence works at the distributed system level.

Discover Financial’s Cloud Migration Journey

Discover Financial Services provides a compelling real-world example of this transition. Headquartered in Riverwoods, Illinois, Discover processes millions of credit card transactions daily. Their legacy systems struggled to keep pace with the demands of modern data science and rapid engineering cycles. To address this, they initiated a major modernization effort centered on Apache Kafka. The goal was to create a robust event backbone for their cloud-native architecture. They moved away from traditional, monolithic settlement environments. Instead, they adopted a streaming-first approach using Kafka as the central nervous system. This allowed them to stream settlement transactions in real time to downstream processing layers. Crucially, they integrated this setup with Amazon EMR for big data analytics. The migration highlighted the limitations of the old disk-bound model in a dynamic cloud environment. By adopting tiered storage, Discover could retain vast amounts of historical data cheaply. Meanwhile, active transaction streams remained highly responsive. This hybrid approach enabled their data science teams to access rich historical contexts without impacting live transaction throughput. The result was a more agile infrastructure capable of supporting complex fraud detection and personalized customer insights. Discover’s experience underscores the tangible business value of decoupling storage from compute in event streaming platforms.

How Tiered Storage Works Technically

The core innovation driving this change is tiered storage. This mechanism allows Kafka brokers to store recent data on local or fast ephemeral storage. Older data, which is accessed less frequently, is automatically offloaded to cheaper, durable object storage like Amazon S3 or Azure Blob Storage. This process is transparent to the producers and consumers of the data. When a consumer requests data that has been offloaded, the broker fetches it from the remote storage layer seamlessly. This architecture introduces a clear distinction between hot data and cold data. Hot data resides in the local cache, ensuring single-digit millisecond latency for recent events. Cold data lives in the cloud, offering virtually unlimited retention at a fraction of the cost. The system uses intelligent policies to determine when to move segments. These policies consider factors like age, size, and access patterns. Unlike previous versions where retention was limited by physical disk space, tiered storage enables infinite retention logically. Developers no longer need to worry about running out of space on broker nodes. The cloud provider handles the durability and scalability of the underlying object storage. This separation also improves resilience. If a broker fails, the data is not lost because it already exists in the durable cloud storage. Recovery times are significantly reduced since there is no need to replicate massive local datasets across remaining nodes. The technical implementation relies on efficient serialization and network protocols to minimize the latency penalty of remote fetches. Modern cloud networks have become fast enough to make this trade-off viable for most enterprise use cases.

Industry Context and Broader Implications

This shift aligns with broader trends in cloud infrastructure and AI data pipelines. Modern AI applications require access to vast amounts of historical data for training and inference. Traditional Kafka setups made long-term retention prohibitively expensive. With tiered storage, organizations can maintain complete data histories. This creates a richer foundation for machine learning models. Companies like Confluent and Aiven have already embraced this direction in their managed offerings. The open-source community is catching up rapidly through KIPs. This democratizes access to cloud-native capabilities for self-managed deployments. The move also reflects a maturation of the event streaming market. Early adopters focused on speed and throughput. Today, the focus is on total cost of ownership and operational simplicity. Enterprises are looking for solutions that integrate easily with existing cloud ecosystems. They want to avoid vendor lock-in associated with proprietary message queues. Kafka’s open-source nature combined with cloud-native features makes it a strategic choice. It bridges the gap between real-time processing and batch analytics. This convergence is critical for building unified data platforms. As more companies adopt serverless architectures, the decoupling of storage becomes even more important. It allows compute resources to scale independently based on demand. This flexibility is essential for handling unpredictable traffic spikes common in modern digital services.

What This Means for Developers

Developers must adapt their mental models for Kafka deployment. The assumption that every broker needs identical, high-capacity disks is obsolete. Configuration now involves defining retention policies and storage tiers rather than just partition counts. Monitoring tools need to track both local cache hit rates and remote fetch latencies. Understanding the cost implications of egress traffic from object storage is crucial. While storage is cheap, data transfer costs can add up if not managed properly. Teams should evaluate their data access patterns to optimize tiering strategies. Frequent access to cold data may negate the cost benefits. Automated tooling for managing these transitions will become standard. IDEs and CI/CD pipelines may eventually include checks for optimal storage configuration. The barrier to entry for large-scale event streaming lowers significantly. Startups can now experiment with massive data volumes without upfront hardware investment. This accelerates innovation in real-time analytics and IoT applications. Engineers can focus more on business logic and less on infrastructure maintenance. The reliability improvements also reduce on-call burden during peak loads. Overall, the developer experience becomes more aligned with other cloud-native services like databases and object stores.

Looking Ahead

The future of Kafka is undeniably cloud-centric. We can expect further optimizations in remote storage protocols to reduce latency even more. Integration with serverless compute platforms will deepen, allowing for truly elastic event processing. Standardization of tiered storage APIs across different cloud providers will enhance portability. This will prevent fragmentation and ensure that applications remain agnostic to the underlying storage backend. Expect to see more advanced data lifecycle management features built into the core platform. These might include automatic compression, encryption, and governance controls for archived data. The community will likely explore hybrid models that combine edge computing with cloud storage. This would enable low-latency processing at the edge while maintaining centralized data consistency. As AI models grow larger, the role of Kafka as a feeder for training data will expand. Efficient, cost-effective retention will be a key enabler for generative AI applications. The industry will continue to refine the balance between performance and cost. Ultimately, the goal is a frictionless data streaming experience that scales infinitely. The transition to diskless architectures is a pivotal step toward that vision.

Gogo's Take

• 🔥 Why This Matters: This isn't just a technical tweak; it’s a financial game-changer. By decoupling storage from compute, enterprises can cut infrastructure bills by up to 90%. For companies like Discover, this means scaling data capabilities without proportional cost increases, enabling real-time AI insights that were previously too expensive to maintain.
• ⚠️ Limitations & Risks: The shift introduces new complexities. Network latency for cold data fetches can impact consumer performance if not tuned correctly. Additionally, reliance on cloud object storage means you are subject to cloud provider egress fees and potential service outages. Monitoring becomes harder as you track two distinct storage layers instead of one.
• 💡 Actionable Advice: Audit your current Kafka retention policies immediately. Identify datasets that are rarely accessed after 24 hours and configure tiered storage to offload them to S3 or equivalent. Test your consumer applications for latency spikes when accessing older data segments before fully committing to the new architecture.