Architecture

This page describes the Beacon Pulse architecture as it works today, not the older intended design.

Current Components

Component	Current responsibility
apps/pulse-public	Public Pulse UI, Pulse JSON APIs, embeds, /admin shell, docs redirects, community and source helpers
apps/pulse-ingest	Upload APIs, queue processing, replay, regeneration, quota controls, backfill tools, media analysis, scheduled repair jobs
Cloudflare D1	Export records, message hashes, daily digests, public weekly summaries, concept graphs, health snapshots, communities, sources, quota tables, media tables
Cloudflare R2	Raw exports and ZIP-backed media objects
Cloudflare Queue	beacon-pulse-uploads for ingest work and background jobs
Workers AI and Media bindings	Text summarization plus image, audio, and sampled-video analysis

End-To-End Flow

flowchart LR
  Admin[Admin user] -->|/admin| PublicWorker[Pulse public worker]
  PublicWorker -->|upload, replay, quota, media ops| IngestWorker[Pulse ingest worker]
  IngestWorker -->|store raw export| R2[(R2)]
  IngestWorker -->|create export row| D1[(D1)]
  IngestWorker -->|enqueue work| Queue[[Queue]]
  Queue --> IngestWorker
  IngestWorker -->|daily + weekly + concepts + health| D1
  PublicUsers[Public users] -->|/pulse and /pulse*.json| PublicWorker
  PublicWorker --> D1

What The Workers Actually Own

Public worker

• serves /pulse, /pulse.json, /pulse/history.json, /pulse/daily.json, /pulse/trends.json, /pulse/concepts, /pulse/concepts.json, /pulse/embed
• serves /pulse/communities.json and /pulse/sources.json
• hosts /admin
• exposes only three public-worker admin helper APIs: /admin/communities, /admin/sources, /admin/exports/assign-source
• enforces admin access on /admin*

Ingest worker

• handles uploads, file listing, delete, replay, regenerate, quota, clear, media, and backfill routes
• consumes queue work
• runs scheduled self-healing every 5 minutes
• enforces admin access on all ingest admin routes

Current Data Products

Data product	Where it lives	Current exposure
Raw exports	R2	Admin-only
Message hashes	D1	Admin-only
Daily digests	D1	Public today through /pulse/daily.json because PUBLIC_DAILY_DIGESTS=true
Weekly summaries	D1	Public
Weekly concept graphs	D1	Public
Weekly health snapshots	D1	Public as part of weekly responses
Community and source metadata	D1	Mixed: admin-managed, but communities and sources also have public listing endpoints

Operational Behaviors That Affect The Architecture

• Omitting community and source on public Pulse endpoints does not produce a neutral view. The worker defaults to ihouse-nyc and ihouse-main-whatsapp-group.
• Weekly summaries are created only when a week has at least 3 processed days or 20 total messages.
• /presign still exists, but it is a placeholder helper. The live admin upload flow uses /upload and multipart endpoints.
• Replay and regenerate-from-raw workflows depend on the raw object still being present in R2.
• Raw-export cleanup is now implemented in code. Default cleanup targets are 72 hours after success and 168 hours after failure when delete-after-processing is enabled.

Verified against beacon-platform/apps/pulse-public/src/index.ts, beacon-platform/apps/pulse-ingest/src/index.ts, and the rendered UI on April 22, 2026.