Sources

Manage data streams — share, fetch, monitor health.

# Source Status Last Fetch Points Share

Source Matrix

Data types, upstream APIs, fetch intervals, backfill coverage.

# Source Category Type API Interval Last Fetch Fresh Health Shared Points Backfill Parent

Scheduler

Smart fetch scheduler — per-source intervals, bundled Open-Meteo.

Fetch Tiers

Interval Fetch Every Total Shared Fresh Stale Never

Per-Source Schedule

Source Type Interval Elapsed Next In Freshness Shared

System Health

Server status, performance metrics.

Version
-
Mode
-
Connected Users
-
Sources Fetched
-
Total Points
-
Uptime
-

Connected Users

Active connections to the dashboard.

UserRoleConnected

Banned Users

None

API v1 Status

World State API health and endpoint status.

MCP Server

Model Context Protocol server for AI agent consumption.

✎
MCP server scaffold built — testing in progress
6 tools: get_world_state, list/get/search streams, attribution, health

x402 Payments

Pay-per-request micropayments via HTTP 402 + USDC.

★
Coming soon — zero friction, no signup, no API keys
Agent pays per request via USDC on Base + Solana

Time Machine

Redis hot layer (48h) + TimescaleDB cold layer (5yr).

↺
Recording live — timeline scrubber UI coming soon
6 backfill scripts ready, needs TIMESCALE_URL to run

Data Types

Realtime Fetched every hour from a live API. Data changes every pull — ephemeral if not recorded. Grid Like realtime but as a lat/lon grid (~648 points). Resolution: 2°/5°/10°/20°. Snapshot Fetched every 24h. Slow-changing or historical. Often backfillable. Static Curated dataset. No upstream changes. No time axis.

Source Architecture

Field Splitting

One API call returns multiple fields. Each numeric field becomes its own stream. The Parent column in Matrix shows which streams share a fetch. All children use cached_raw() — fetch once, split many.

Sharing Model

ok (shared) Visible to users, served via API v1. maintenance Shared but failing. Auto-set on error, auto-clears on success. not_shared Hidden from users. Default. Admin must opt-in.

Time Machine

  • Hot layer (Redis) — 48h snapshots, checksum-deduped, recorded every 15 min.
  • Cold layer (TimescaleDB) — 5yr retention, compressed. Flush cron hourly.
script Backfill script exists, dry-run tested. Ready with TIMESCALE_URL. possible Upstream API has history but no script yet. 90d Open-Meteo past_days parameter (up to 90 days).

API v1 Endpoints

GET /world Multi-stream situational awareness for a lat/lon. GET /streams Catalog browse (optional ?category=). GET /streams/:id Single stream data with geo-filter + limit. GET /streams/search Keyword search across id/name/description. GET /attribution License info for all shared sources. GET /health API health (shared count, fetched count).

Server Architecture

dashboard.py Slim entry point (~240 lines). Creates app, socketio, limiter. Populates state. Registers blueprints. Runs scheduler + flush cron. state.py ALL shared mutable state + helpers. Blueprints import this. No Flask app dependency — circular imports impossible. routes_pages.py Landing, dashboard, admin, roadmap, version, auth routes. routes_admin.py Connections, stats, scheduler, matrix, archive, ban/unban. routes_catalog.py Catalog, sources, fetch, source CRUD, ticker. routes_presets.py Presets, snapshots, config, mode, user presets. routes_targets.py Target catalog, output targets, routes, downloads. routes_socket.py SocketIO handlers: connect, disconnect, fetch, subscribe.

Smart Scheduler

~60s tier ISS — fetched every 5 min. 1h tier Grid + realtime sources — fetched every 60 min. 24h tier Snapshots — fetched every 6 hours. Static Fetched once on startup, never again. Bundled Mode Open-Meteo: 3 API calls instead of 21. Toggle: BUNDLED_MODE in open_meteo_bundled.py.

Access Control

Server gating Unshared sources never reach non-admin clients. Checked at emit time, not client-side. Ban system Redis set oa:banned. Blocks login (403) + SocketIO connect (silent reject). Admin: ban/unban/list. Rooms authenticated (all users) + admin (admin only). Shared → broadcast. Unshared → admin room.

Gotchas

  • Stale __pycache__ — Always rm -rf before restart.
  • Multiple processes on :5050 — Windows allows it. Check netstat, kill ALL PIDs.
  • Coolify restart vs redeploy — Restart ignores new env vars. Must redeploy.
  • Redis is source of truth — File backup inside Docker is ephemeral.
  • SocketIO transport — ["polling", "websocket"] (polling first). ping_timeout=10, ping_interval=15.
  • Never alias state._preset — It gets replaced on preset POST. Always access at call time.
  • Import order matters for bundled — open_meteo_bundled.py must import AFTER individual files to overwrite registrations.