Built a Centralized Big Data Platform for an Australian Healthcare Giant

Each hospital facility had its own IT department. This decentralized data system had become a cornerstone problem for the client, causing ripple effects in different aspects of their business. Lack of integrated data was causing many logistical issues, like duplication of efforts and more, leading to delayed care and fragmented overall patient experience.

Business teams were also struggling to increase the revenue without an integrated view of a patient’s journey. With so many gaps in data, many times their marketing teams were left to act on the basis of guesswork than actual insights derived from data. Accurately matching the products and services with the current needs of the patients was needed urgently to mark a huge business win.

Building the next-generation cloud-based centralized data platform seemed like a huge step, but it was causing an unbearable loss to the client and could no longer be put on hold.

The client approached Symv to build this centralized platform while carefully migrating their existing patient data. They were ready to upgrade their IT infrastructure completely and be the data-first healthcare company.

Symv's expertise and experience with handling projects related to ETL pipelines, big data storage, and processing put the client’s doubts at ease pretty quickly. Cost optimization, timely delivery, and innovation were at the heart of this collaboration.

The plan implemented by Symvoulos' engineers was a scalable, well-rounded solution to any existing or future data needs of the client and included the following:

• Architecting an all-new IT infrastructure to manage data for all hospitals in one centralized location by one team. S3 buckets were used for organized data storage.
• Configuring a CI/CD workflow and tooling for automated deployments.
• Setting up a security and governance control framework required by the Australian Healthcare industry.
• Using Airflow for building a fault-tolerant ETL pipeline for monitoring, logging, and alerting. EMR and Redshift were used for analysis.
• Offering support for ETL transformations.

Besides building the platform, Symvoulos' team also offered necessary training to the client’s team on the new data management best practices. This cross-functional team played an important role in delivering a cloud-based data platform with continuous deployment and testing.

Once the project was complete, the client’s operations team took over to deliver simple work requests like one-off extracts or even some complicated projects. The solution was made to be self-serve so their business teams can derive insights and reporting functionality without any dependency.

The project entailed a lot of successful deliverables. The first one was automating the deployment and configuration of AWS Landing Zone implementation that seamlessly brought together different services used by the current system from multiple direct connections. Next, we ran a production-quality Airflow cluster to manage ETL and ingestion of incoming data automatically.

Automating the complete SDLC with a Jenkin-based, fully-automated CI/CD pipeline was done to enable deploying Airflow DAGs including Testing and Linting. The team created and managed images with Packer for services like Jenkins and Airflow.

In addition, Terraform deployments included switching the complete infrastructure from landing Zone to Servers. We set up AWS Redshift/GlueMaking data sources and custom EMR clusters which were available on request. Automating application log shipments to Sumologic was also implemented.