Hi, my name is

Anay Vyawahare.

I build pipelines that turn raw data into something useful.

I'm a data engineer who enjoys building reliable pipelines, data infrastructure, and warehouses. I'm currently looking for full-time Data Engineering roles.

Get In Touch Download Resume

01. About Me

I'm a Data Engineer with 1.5+ years of production experience building ETL/ELT pipelines in Python and SQL on AWS. My project work extends into Apache Spark, Delta Lake lakehouse design, dimensional modeling with dbt and Snowflake, Airflow orchestration, and pipeline monitoring with Prometheus and OpenLineage.

From Jan 2025 to Sep 2026, I worked as a Data Engineer at a startup — building production ETL pipelines, leading a 3-engineer team through a service migration, and designing event-driven ingestion workflows on AWS. I like owning problems end-to-end and thinking through data architecture before writing a single line of code.

Outside of the job, I build end-to-end projects — streaming CDC pipelines, lakehouse architectures, and pipeline observability platforms — to deepen my knowledge of data engineering at scale.

02. Experience

Data Engineer @ The Lead Market

Jan 2025 – Sep 2026

  • Built and maintained production ETL pipelines in Python that ingest, validate, and transform lead and campaign data from third-party REST APIs into PostgreSQL and DynamoDB for downstream reporting.
  • Defined transformation and data quality rules with business stakeholders; ran SQL-based validation and source-to-target reconciliation to catch load discrepancies before they reached reports.
  • Designed event-driven ingestion workflows on AWS using Lambda, EventBridge, and S3 with idempotent processing and retry handling; instrumented CloudWatch metrics to surface silent pipeline failures faster.
  • Led a 3-engineer team through a service migration, architecting schema design and backfill strategy and assigning work; reconciliation between legacy and target stores confirmed zero record loss at cutover.
  • Profiled slow SQL and added indexing and partition-aware filters, cutting run time on high-volume tables.
  • Containerized services with Docker and automated checks through GitHub Actions CI/CD, reducing recurring manual verification; maintained runbooks and deployment documentation for handoff.

03. Projects

Streaming CDC Lakehouse

Change-data-capture pipeline where Debezium tails the PostgreSQL WAL into Kafka topics with Avro schemas in Confluent Schema Registry. Processed with Spark Structured Streaming using event-time windowing and exactly-once MERGE upserts into Delta Lake, with a ClickHouse serving layer for sub-second dashboard aggregates.

  • Python
  • Kafka
  • Debezium
  • Spark Structured Streaming
  • Delta Lake
  • ClickHouse
  • PostgreSQL

ShopFlow: Batch ETL E-Commerce Data Platform

Batch ETL pipeline moving transactional data from PostgreSQL into a partitioned S3 lake with PySpark schema enforcement and dedup. Modeled a Snowflake star schema in dbt with SCD Type 2 snapshots, orchestrated in Airflow with Terraform-provisioned infrastructure and dbt-test gated publication.

  • Python
  • Spark
  • PostgreSQL
  • AWS S3
  • Airflow
  • Snowflake
  • dbt
  • Terraform
  • Docker

Pipeline Observability Platform

Collected OpenLineage events from Airflow DAGs and Spark jobs into Marquez for column-level impact analysis. Built freshness, volume, and schema-drift detectors with SLA-scoped Slack alerting, plus Prometheus exporters and Grafana dashboards tying failures to schema changes.

  • Python
  • Airflow
  • OpenLineage
  • Marquez
  • Prometheus
  • Grafana
  • PostgreSQL

04. Skills

Languages

  • Python
  • SQL
  • Bash

Data Engineering

  • Apache Spark
  • Airflow
  • dbt
  • ETL / ELT Pipelines

Data Warehousing

  • Snowflake
  • Delta Lake
  • PostgreSQL
  • DynamoDB

Cloud & DevOps

  • AWS (S3, Lambda, EventBridge)
  • Docker
  • Terraform
  • GitHub Actions