The Customer
Thrivent is a client-owned financial services organization that provides a range of financial solutions, including banking, investments, insurance, and generosity programs.
The Project
Thrivent Enterprise Integration Datastore (TEID)
Overview
Improving partnered with Thrivent to develop the Thrivent Enterprise Integration Datastore (TEID). This project aimed to consolidate and streamline data from various systems of records, enhancing the accessibility and usability of data for front-end systems, thus improving user experience on their website and mobile applications. The scope of the project included data ingestion, organization, and real-time updates, ultimately modernizing Thrivent's data infrastructure and operations.
The Business Problem
Thrivent faced significant challenges with their existing data management systems, which were slow, disjointed, and relied heavily on multiple third-party data hosts. The siloed nature of data storage across different systems and departments resulted in inefficiencies and slow data access, impacting their ability to provide a responsive and cohesive user experience on their digital platforms. Additionally, maintaining and integrating legacy systems like SIFT and the mainframe added complexity and cost, hindering their modernization efforts.
The Solution
Improving's approach involved creating a unified data integration platform using Confluent's Kafka for real-time data streaming and DynamoDB for fast data storage and retrieval. We collaborated with various teams at Thrivent, including those handling front-end systems, APIs, and data records, to design and implement a robust data pipeline. This pipeline aggregated and normalized data from multiple sources, ensuring real-time data synchronization and seamless integration with new systems like Salesforce. We also leveraged GraphQL for efficient data querying, enhancing the usability of the datastore for front-end developers.
Technologies Used
Kafka: For real-time data streaming and integration.
DynamoDB: For fast and efficient data storage and retrieval.
GraphQL: For flexible and efficient data querying.
AWS: Used for S3 buckets and other cloud services.
Kubernetes and OpenShift: For container orchestration and management.
Hashicorp Vault: For secure management of secrets and access control.
Terraform: For infrastructure as code and consistent state management.
The Business Benefits
Improved Data Accessibility: Centralized data store enabled quick and easy access to comprehensive data sets.
Enhanced User Experience: Faster, more responsive web and mobile applications.
Operational Efficiency: Streamlined data processes and reduced redundancy across departments.
Cost Savings: Potential reduction in API usage fees from third-party data providers.
Modernized Infrastructure: Transitioned from legacy systems to modern, scalable solutions.
Increased Trust in Data: Reliable and real-time data updates allowed for more confident business decisions.
Partnerships
The project involved collaboration with several technology partners. Key partnerships included Confluent for Kafka implementation and support, Apollo for GraphQL development, and Datadog and Elastic for observability and logging. These partnerships provided essential tools and expertise that contributed to the successful implementation of the TEID project.
Lessons Learned
Data Governance: Implementing a robust schema registry was crucial for managing data integrity and consistency.
Real-Time Processing: Real-time data updates significantly improved operational efficiency but required careful handling of batch processes.
System Integration: Bridging communication gaps between teams was essential for seamless integration and project success.
Technology Choices: While DynamoDB provided excellent response times, it posed challenges for data publishing and required additional tooling for observability.
Legacy System Migration: Gradual migration from legacy systems to modern platforms is a complex and ongoing process.
Scalability: Ensuring the solution could handle large data volumes and maintain performance was critical for long-term success.
Conclusion
The Thrivent Enterprise Integration Datastore project exemplifies Improving's capability to deliver innovative, scalable, and efficient data solutions. By leveraging modern technologies and fostering strong partnerships, we helped Thrivent achieve significant improvements in data accessibility, user experience, and operational efficiency. Our collaborative and adaptive approach ensured the successful implementation of a complex data integration project, setting the stage for future advancements and continued modernization efforts.
