Portfolio
Data Warehouse & Analytics Platform
Insurance Analytics / Data Engineering
100x
Query performance improvement vs PostgreSQL
4
CSV export pipelines for Rankomat
1
Analytics for Insly Broker Poland
Overview
Led development of a high-performance analytics data warehouse at Insly, replacing slow PostgreSQL-based reporting with a ClickHouse-powered platform. The system handles multi-tenant analytics for Insly Broker Poland and integrates with Rankomat via structured CSV exports, enabling real-time dashboards and large-scale data analysis.
Business Context
Insly serves insurance brokers across Central Europe with a platform for policy management and analytics. The analytics layer was built on top of the operational PostgreSQL database, causing performance issues as data volumes grew to millions of records. Brokers needed real-time dashboards and the business required integration with Rankomat, Poland's leading insurance price comparison platform.
Challenge
Insurance analytics required querying millions of policy and claims records, but PostgreSQL-based reporting was too slow for real-time dashboards and large data exports.
- PostgreSQL queries on millions of insurance records taking minutes instead of seconds
- Multi-tenant data isolation requirements across broker organizations
- Rankomat integration requiring 4 structured CSV export pipelines with strict formatting
Solution
We adopted ClickHouse as the analytical engine for its columnar storage and vectorized query execution, combined with DBT for maintainable data transformations and Python ETL pipelines for data ingestion from the operational PostgreSQL databases.
- Migrated analytical workloads from PostgreSQL to ClickHouse with columnar storage
- Built DBT-based data transformation pipelines for multi-tenant analytics
- Implemented 4 CSV export pipelines for Rankomat integration
Approach & Methodology
We started by profiling the slowest analytical queries and understanding the access patterns. ClickHouse was selected for its exceptional performance on insurance analytics workloads. We used DBT to create maintainable, version-controlled transformation logic and Python ETL pipelines for data ingestion. The migration was done incrementally to avoid disrupting existing reporting.
Implementation Details
ClickHouse Columnar Analytics
Designed ClickHouse schemas with appropriate sorting keys and materialized views for the most common analytical query patterns, achieving sub-second response times on queries that previously took minutes.
DBT Transformation Pipelines
Implemented DBT models for data transformation with incremental processing, enabling reliable and testable data pipelines from operational PostgreSQL sources to the analytical warehouse.
Rankomat CSV Export Integration
Built 4 automated CSV export pipelines for the Rankomat price comparison platform, handling data formatting, validation, and delivery scheduling on AWS EKS.
Key Decisions
- ClickHouse over BigQuery — self-hosted on AWS EKS for cost control and data residency compliance
- DBT for transformations — version-controlled, testable SQL models replace ad-hoc scripts
- MinIO/S3 for staging — intermediate storage for large dataset transfers between systems
Tech Stack
Related Services
The following services were utilized in this project to deliver successful outcomes.
Lessons Learned
- ClickHouse sort key selection is critical — wrong keys can negate columnar performance benefits
- DBT incremental models require careful design of uniqueness keys for insurance data with corrections
- Multi-tenant isolation in ClickHouse needs row-level security design from day one
Project Information
Timeline
5 months
Team
2 engineers (data + backend)
Results
100x
Query performance improvement vs PostgreSQL
4
CSV export pipelines for Rankomat
1
Analytics for Insly Broker Poland
Have a Similar Challenge?
Let's discuss how I can help your project succeed with proven architecture and AI solutions.