An Enterprise Eventing Platform Using Kafka

The Customer

This publicly traded Fortune 25 company is the largest Medicaid and Medicare managed care provider in the country focused on transforming the health of the communities it serves.

The Project

An Enterprise Eventing Platform using Kafka

Overview

This healthcare organization has 26.6 million members and over 72,500 employees across 50 states. Through mergers and acquisitions, their aggressive growth strategy has increased revenue by 13% year-over-year to $126 billion in 2021.

The Challenge

This company’s ambitious growth mindset created data integration and data migration challenges. Departments could not discover or share valuable data and often needed permission from database owners, which slowed down business processes. They needed to remove data silos that hindered data discoverability between teams, and they needed to move from batch processing to real-time stream processing to improve data accuracy and accelerate business processes.

They identified Apache Kafka as a solution but lacked the expertise for an enterprise-wide implementation that ensured SOC compliance. Once the Kafka implementation was complete, they needed help migrating and transforming massive amounts of data from multiple newly acquired systems into a standard format. The unprecedented scale of this project included billions and billions of data points from over ten different systems, making each decision crucial to the overall long-term success of the technology implementation.

The Solution

Phase 1: Kafka Implementation

For this project, we refactored monolithic systems into distributed micro-services, developing an enterprise eventing platform to accommodate the transformation to event-driven architecture. Our applications provided a representational state transfer (REST) and remote procedure call (gRPC) interface to allow development teams to publish and consume events. The platform included:

• Structured schema evolutions with Confluent Schema Registry and AVRO

• Configurable (self-service) event-routing with K-Tables

• Dynamic event-aggregation with Kafka Streams

• Distributed event tracing with Jaeger

• Event querying against a MongoDB event-store hydrated by Kafka Connect

• Self-Service Portal UI and API to allow teams to onboard and maintain their configurations

Phase 2: Data Normalization as a Service

Once we implemented Kafka, the platform around Kafka, and trained the development team on the software, we began the normalization of billions of pieces of data across over ten systems. The primary tech stack included Java/Spring Boot apps running in Kubernetes talking to Apache Kafka and databases such as Mongo, Neo4j, and Postgres. In recent months, they’ve adopted Quarkus as an alternative to Boot. A frontend VueJS app was built from scratch to dynamically manage the mappings between data formats. We developed Gitlab CI/CD pipelines for deployments and built out monitoring and metrics through Kibana/Grafana. We ran this project across the entire stack, and we trained their staff on the technologies.