Resilient Hybrid Kafka: Automating On-Prem Deployments with Confluent Cluster Linking

When working in highly regulated industries like healthcare, ensuring continuous data processing and retention is not merely nice to have, it’s a legal requirement. Apache Kafka is the ideal solution for highly resilient, real-time, massively scaled, event streaming, but even it can be vulnerable to outages when you have only a single cluster. This talk explores an innovative solution Improving helped deploy at a Fortune 100 client that leveraged powerful Confluent tools like Confluent for Kubernetes (CFK) to create repeatable, automated deployments of Kafka clusters across 30 on-prem data centers and Cluster Linking to replicate critical event data to a central Confluent Cloud cluster. We will delve into the technical details on how this architecture allowed for local event processing even if connectivity to Confluent Cloud failed, with byte-for-byte replication that would catch up as soon as connections were restored. Learn how this system enabled compliance with stringent regulations and centralized data across the enterprise to allow for aggregation and analytics, all while providing the resiliency needed to prevent costly system downtime.