Beacon Platform Docs

Queue Architecture

3 min read

Beacon uses a single Cloudflare Queue (beacon-pulse-uploads) for all async processing. The ingest worker is both producer and consumer.

Message Types

type fieldProducerConsumer action
(no type — export event)Upload handlers, self-healing cronParse export, deduplicate, generate daily digests and weekly summaries
media_analysisenqueueMediaAnalysisJobs()Analyze one media artifact (image, audio, or video)
media_deferred_batchenqueueDeferredMediaBatches() (cron */15)Resume deferred media analysis for one export
regenerate_weekly/regenerate/weekly endpointRebuild weekly summaries for one week or range
regenerate_weekly_all/regenerate/weekly/all endpointRebuild all weekly summaries for a community
backfill_concept_pillarsWeekly summary pipeline (after saveConceptGraph)Run AI pillar classifier on a concept graph saved with keyword-only pillars

Export event payload

{
  "community_id": "ihouse-nyc",
  "objectKey": "community/ihouse-nyc/abc123.zip",
  "source": "upload",
  "source_id": "ihouse-main-whatsapp-group",
  "eventType": "created"
}

Media analysis payload

{
  "type": "media_analysis",
  "media_id": "...",
  "export_id": "...",
  "community_id": "...",
  "source_id": "..."
}

Regenerate weekly payload

{
  "type": "regenerate_weekly",
  "community_id": "ihouse-nyc",
  "source_id": "ihouse-main-whatsapp-group",
  "week_start": "2026-04-19",
  "week_end": "2026-04-25",
  "force": false,
  "compare_previous_week": false
}

Backfill concept pillars payload

{
  "type": "backfill_concept_pillars",
  "community_id": "ihouse-nyc",
  "source_id": "ihouse-main-whatsapp-group",
  "week_start": "2026-04-19"
}

Enqueued automatically by the weekly summary pipeline after saving a concept graph with keyword-only pillar assignments. The consumer loads the saved graph, runs the AI pillar classifier over all nodes, and saves the refined graph back.

Retry Policy

Cloudflare Queues retry failed messages automatically with exponential backoff. The ingest worker additionally handles specific failure modes:

  • AI quota exhaustion: Queue message is re-enqueued with a delay until midnight UTC when the daily quota resets. Delay is tracked per message to avoid retry storms.
  • Too-many-SQL-variables: Caught explicitly; logged and retried with a delay (30s × 2^attempt, capped at 2 minutes).
  • Recoverable export failures: retryRecoverableFailedExports() (cron) identifies exports with status='failed' and recoverable error messages, resets them to queued, and re-enqueues them. Max retries: AUTO_RETRY_MAX_ATTEMPTS (tracked in error_message).
  • Recoverable media failures: Same pattern via retryRecoverableFailedMedia().

DLQ Behavior

Cloudflare Queues does not have a configurable DLQ for Workers consumers. Messages that exhaust all delivery attempts are dropped. The application compensates for this via the cron self-healing job which detects stuck exports by updated_at staleness.

Stuck Export Detection

Every 5 minutes the scheduled handler queries:

SELECT * FROM exports
WHERE status IN ('processing', 'weekly_processing', 'queued')
  AND (days_processed < days_total OR weeks_processed < weeks_total OR status = 'queued')
  AND (julianday('now') - julianday(COALESCE(processed_at, uploaded_at))) * 1440 >= 10

Any export stuck for ≥ 10 minutes is re-enqueued. See export-lifecycle for full state transition detail.

Cron Self-Healing Tasks

Two separate cron triggers run on different cadences to prevent slow background work from delaying the critical retry loop:

High-priority (*/5 * * * *)

All five sub-tasks run in parallel:

TaskWhat it does
finalizeCompletedWeeklyExportsMarks weekly_processing exports completed when weekly work is already done
retryRecoverableFailedExportsResets and re-enqueues recoverable failed exports (up to 10 per tick)
retryRecoverableFailedMediaSame for failed media items
resetStuckDeferredProcessingMediaResets media stuck in deferred_processing for > 5 minutes back to pending_analysis
requeueStalePendingMediaJobsRe-enqueues stale pending_analysis media jobs older than 15 minutes

Low-priority (*/15 * * * *)

TaskWhat it does
enqueueDeferredMediaBatchesResumes up to 12 deferred exports per tick
cleanupRetainedRawExportsDeletes R2 raw exports past their retention window (batch of 75 per tick)

Concurrency

  • Daily digests: Configurable via daily_pipeline_mode and daily_concurrency in quota_settings. Default is serial; parallel mode allows up to 10 concurrent AI calls.
  • Weekly regeneration across sources: Up to 3 sources processed in parallel (SOURCE_CONCURRENCY=3). Weeks within a source are sequential to preserve compare_previous_week correctness.
  • Video frame analysis: Frames extracted and Stage 1 classified in parallel; Stage 2 (caption) sequential to enforce maxVideoFrameStage2PerExport cap.
  • Image Stage 1: Classifier and object detector run in parallel (Promise.all).

Graph View

Backlinks