Two years of NYC yellow taxis
An Apache Spark + Delta Lake pipeline over every TLC yellow-taxi trip in 2023–2024. Every number on this page is read from the pipeline's own output.
Daily trips
Silver trips per pickup day. The shaded days are a gap in the published source file that the pipeline detects (under 50% of the ±7-day median).
Pickups by borough
Two years of silver trips.
Trips by hour and weekday
Busiest pickup zones
Top 12 of 263 zones.
Where trips start
All 263 taxi zones, drawn from the TLC's own boundary file. Two years of silver trips, by pickup zone.
Data quality
Quarantined rows by rule
Errors: the row moves to quarantine with its reasons.
Kept but tagged
Warnings: the row stays in silver with a flag.
Bronze rows by month
Each month reconciled exactly against the source file's own row count.
What profiling found
- Fare-total mismatch is a vendor convention, not corruption. The gap is almost always exactly $2.50 or $1.75: vendor 1 folds the congestion and airport fees into
extra. So it is a warning, not an error. - A source gap, 21–24 Sep 2023. Those days hold under half the usual trips in the published file. Gold flags them and the forecast skips them.
- Rule thresholds came from the data. Speed, duration, distance and fare limits were set after profiling the distributions, not guessed.
Demand forecast
Daily pickups, held-out quarter
Hourly per-zone predictions summed to days.
What the model leans on
Engineering
A tuning study on the real silver table, and Structured Streaming replays (from files and from Kafka) checked against the batch result.