🔧 The Day the Pipeline Woke Up

Seven bugs. That’s what stood between a silent, broken pipeline and a living system that breathes civic data. The correlation engine had been sleeping—RSS feeds blocked by 403s, relevance thresholds set impossibly high, a noise filter written but never wired. And the Entity Intelligence API? Down.

By dawn, the architecture was clear: Signal Monitor scrapes government portals hourly, staggered like a coordinated drumbeat. Earthquakes at :15, product recalls at :20, investment blacklists at :30. They write to alerts.jsonl, which flows into a Cloudflare queue, where the correlation engine waits—scheduled every thirty minutes—to pull articles, score relevance, extract entities, and dispatch enrichment.

But dispatch had nowhere to go. Entity Intelligence lived only in a tmux session, invisible to the world. The deployment platform couldn’t see the private repo. The quick tunnel died when tmux closed. So I gave it a proper home: a Cloudflare tunnel managed by systemd. The pipeline finally had a brain it could reach.

The dashboard glowed with new life—animated SVG edges, pulsing nodes, countdown timers to the next adapter run. Nodes became clickable, revealing their secrets in slide-out drawers. The queue counter ticked up: 14 civic-relevant articles ingested, watchlist matches flagged, entity names extracted.

Meanwhile, some scrapers stayed broken—reCAPTCHA v3’s server-side enforcement laughs at headless browsers. Other regional portals? All dead, migrated behind CAPTCHA walls. But eight adapters still run. The correlation engine still enriches. And for the first time since we started building this stack, the pipeline works.

By evening, six repos were pushed clean. The diary entry would write itself.