Biswajit Brahmma
Lead Consultant — Senior Data Engineer & Platform Architect
Senior Data Engineer with 9.5 years of total experience, including 6.5+ years building production ETL/ELT pipelines and scalable data platform infrastructure across retail, finance, and social impact platforms. Partnered with stakeholders to design Medallion architectures and CDC pipelines on Databricks (Unity Catalog), Snowflake, and AWS; implemented data governance and data quality frameworks, optimizing processing by 70% and data reliability by 85%.
Genpact
Senior Data Engineer
- Core Data Engineer and Maintainer for the Japan Data Quantum Lakehouse, architecting end-to-end Medallion pipelines and enterprise data products.
- Architected idempotent, config-driven PySpark pipeline for incremental ingestion and CDC across 28 partners, reducing execution time by 70%.
- Designed and owned DAVIS, an enterprise Python reconciliation framework for KPI validation (Demand, Inventory, Invoice) that reduced data quality issues by 85%.
- Led a zero-downtime Unity Catalog migration and enterprise-wide Bitbucket to GitHub migration with strict data governance compliance.
- Engineered automated ELT pipelines using AWS event-driven architecture (EC2, Glue, Lambda, EventBridge) and Snowflake, processing up to 20 GB daily.
- Achieved a 75% reduction in data processing time and improved storage efficiency by 60% through modern ingestion architectures.
- Implemented proactive data observability, cutting pipeline downtime by 45% and incident volume by 30%.
Knoema IT Solutions
Senior Data Engineer II
- Developed automated ETL/ELT pipelines for finance and market intelligence datasets using Python, Pandas, and SQL, unifying data from 10+ formats into Snowflake.
- Designed and built a scalable AWS S3 data lake architecture with Apache NiFi pipelines for multi-level compressed files ingestion.
- Scheduled and managed 40+ automated data pipelines across environments via Knoema Datahub, maintaining a 95% job success rate.
Collabera (Client: ZS Associates)
Senior Software Engineer (Contract)
Designed scalable ETL pipelines with Databricks, Airflow, PySpark, and Spark SQL into AWS Aurora; built interactive analytics apps with Python, Dash, Plotly, and PostgreSQL.
Dhwani Rural Information Systems
Project Manager, Software Application & MIS
Managed a 7-person engineering team to build a centralized data warehouse and automated ETL pipelines (Python, MySQL, AWS EC2), powering analytics for 100,000+ users.
Lend A Hand India
Officer Technology, Data Analysis & Project Coordination
Designed and implemented a GCP-native data platform (BigQuery, GCS, MongoDB) across 4 state education departments, enabling data-driven governance for 1,200+ teachers.
Cloud & Platforms
Data Infrastructure & Engines
Programming & Query
DevOps, AI & Architecture
Cricket Intelligence Platform
Medallion Data LakehouseEnd-to-end cricket data platform applying scalable Medallion architecture, automated data validation, and orchestration to ingest, transform, and serve ball-by-ball performance insights.
NYC Taxi Fare Prediction
ML PipelineEnd-to-end Machine Learning pipeline using Linear Regression, Ridge Regression, and XGBoost to predict NYC Taxi fares based on massive geospatial datasets.
Indian School of Development Management (ISDM)
Post Graduate Diploma in Development Leadership (PGP-DL)
Jadavpur University, Faculty of Engineering & Technology
Bachelor of Engineering in Power Engineering
Let's Build Resilient Data Systems Together
Open for Staff / Lead Data Engineering roles, architecture advisory, and technical collaboration.