Cloudflare Data Platform

148,140,122 rows · 5 pipelines

Billions of rows.
Sub-second dashboards.

A showcase of Cloudflare's data pipeline architecture. Collector Workers fetch public data, Pipelines ingest into R2, Aggregator Workers run analytics, and Workers SSR delivers zero-jump dashboards from KV.

Platform Architecture

5 pipelines · 10 workers · 1 KV namespace
Data SourceCollectorCold PathAggregatorHot PathDashboardOpen-Meteo*/15 minPipelines + R2*/15 minWorkers KV/weatherGitHub + HN*/5 minPipelines + R2*/5 minWorkers KV/zeitgeistGBFS (3 cities)*/5 minPipelines + R2*/10 minWorkers KV/mobilityCoinGecko*/5 minPipelines + R2*/5 minWorkers KV/cryptoNPM RegistryhourlyPipelines + R2*/6 hrWorkers KV/frameworks
Click any pipeline row to inspect the hot/cold split and the R2 SQL query loop