The Problem

FinCrime data was scattered around different teams at the bank in many different formats, resulting in inaccurate reporting.

The Brief

Create a unified data hub with automatic ingestion for all FinCrime data that can serve accurate, up-to-date data to Tableau for report generation.

The Solution

We set up a system of automated StreamSets pipelines that ingest data directly from source systems such as Oracle and MongoDB into a Snowflake data hub. Data Quality checks, error handling, and monitoring are performed on the pipelines, and Tableau reports connect directly to Snowflake for final reporting.

Tech We Used

Scala, Spark
Amazon S3