AirflowdbtPostgreSQLDocker
Drone-Trajectory Data Pipeline
Training project
A fully dockerized data-warehouse stack for traffic-trajectory data from swarm drones (pNEUMA): Airflow ingests large CSVs (~87MB each), and dbt builds tested, documented staging/production models for spatial-temporal analysis.
The problem
Traffic-trajectory data captured by swarm drones (the pNEUMA dataset) arrives as large CSV files that are awkward to store, transform, and query.
What I did
- Built a fully dockerized data-warehouse stack.
- Used Airflow to ingest the large trajectory CSVs, around 87MB each.
- Modeled the data with dbt, including tests, documentation, and staging and production schemas.
- Made the workflow reproducible for spatial-temporal analysis of vehicle paths.
Results
- Turned bulky raw CSVs into a queryable, tested warehouse.
- Documented a reproducible workflow for spatial-temporal analysis.