02 · Projects / 01
GH Pulse
Problem
GitHub Archive publishes every public GitHub event as hourly gzipped JSON. In that raw form it can't answer simple questions about repos and contributors.
Data
8,784 gzipped JSON files a year, streamed with retries.
Method
httpx and tenacity download the files. DuckDB unnests them into Parquet (100M+ rows) and aggregates repo stats, contributor counts and event-type breakdowns. PostgreSQL takes upserts plus an audit-log row for every run.
Result
A live Streamlit dashboard, 15+ tests, CI on every push, and idempotent loads with an audit trail.
- 01 Extract 8,784 gzipped JSON files a year, streamed with retries
- 02 Transform 100M+ rows unnested by DuckDB into Parquet
- 03 Aggregate Metrics repo stats, contributor counts, event-type breakdowns
- 04 Load Postgres upserts plus an audit log row for every run