Beacon Pulse runs on Cloudflare Workers (pulse-ingest + pulse-public), Cloudflare Queues, R2, and D1. This document describes the platform’s performance characteristics, known bottlenecks, and the strategies applied to address them.
System Overview
End-to-End Flow
POST /upload/multipart/complete
→ R2 PUT (raw export)
→ D1 INSERT into exports (status='queued')
→ Queue.send() to beacon-pulse-uploads
↓
Queue consumer
→ parseWhatsAppZipExportFromR2() R2 range reads
→ deduplicateMessages() D1 batch INSERTs into message_hashes
→ saveMediaArtifacts() R2 PUT per artifact + D1 per artifact
→ enqueueMediaAnalysisJobs() Queue.send() per artifact
→ runDailyDigestWorkItems() AI ×2 per day + D1 INSERT
→ regenerateWeeklySummaryForWeek() AI ×2 per week + D1 ×6–8
↓
Public API: GET /pulse/history.json
→ D1 SELECT weekly_summaries_public
→ HTTP response (Cloudflare Cache API, 5-min TTL)
Storage Layer
| Layer | Used For | Notes |
|---|---|---|
| R2 | Raw exports, media artifacts | Range-read ZIP parsing; 72h/168h cleanup |
| D1 | All state: exports, hashes, digests, quotas | Primary bottleneck; targeted by caching and batching improvements |
| Queues | Upload jobs, media jobs, regen jobs | Single queue; 6 message types |
| KV | Quota bypass flag | Optional; 60s TTL; falls back to D1 |
| Cloudflare Cache API | Public API responses | 5-min TTL on /pulse.json, /pulse/history.json, /pulse/concepts.json |
Bottlenecks Identified
N+1 Query Patterns
| Pattern | Location | Queries/Operation |
|---|---|---|
| Media content-hash dedup | saveMediaArtifacts() | 1 SELECT per artifact → replaced with single batched IN query |
/pulse/history.json media counts | public/index.ts | 1 correlated subquery per digest row → replaced with aggregate pre-fetch |
| Daily digest media summary context | Pipeline | 1 JOIN per day |
Sequential AI Calls
| Pattern | Impact |
|---|---|
| Image Stage 1: classifier then objectDetector | 2× slower than necessary |
| Video frames: extract → classify → detect → caption, frame-by-frame | ~9 serial ops per video |
regenerateWeeklySummariesForCommunity(): sources sequential | 100+ sequential operations for large communities |
Missing Caching (Before Improvements)
| What | Where | Cost |
|---|---|---|
| Quota config / bypass flag | Fresh D1 SELECT per AI call | ~50–100ms per AI call |
| Public API responses | No Cache API | Full D1 read on every request |
| Quota status in cron | Re-read per tick | 3–5 redundant queries per tick |
Missing Indexes (Before Improvements)
| Table | Missing Index | Needed For |
|---|---|---|
export_media | (export_id, community_id) composite | JOIN in /pulse/history.json |
exports | (status, community_id) | Cron stuck-export queries |
Improvements Applied
Parallelization
Image Stage 1 — classifier and objectDetector run in parallel.
Both are mandatory Stage 1 calls with no dependency on each other. Running them with Promise.all halves per-image AI turnaround time.
Video frame analysis — two-phase parallel execution.
Phase 1: all frame extraction and Stage 1 classification run in parallel (independent per frame).
Phase 2: Stage 2 caption decisions run sequentially to preserve the maxVideoFrameStage2PerExport cap without a concurrent counter race.
Weekly regeneration — sources run in parallel (SOURCE_CONCURRENCY=3). Sources within a community have no cross-source data dependencies. Up to 3 sources are regenerated concurrently. Weeks within each source remain sequential because each week may compare against the previous week’s summary.
Cron sub-tasks — all tasks run in parallel per schedule.
*/5 * * * * (high-priority): 5 tasks via Promise.all — finalizeCompletedWeeklyExports, retryRecoverableFailedExports, retryRecoverableFailedMedia, resetStuckDeferredProcessingMedia, requeueStalePendingMediaJobs.
*/15 * * * * (low-priority): 2 tasks via Promise.all — enqueueDeferredMediaBatches, cleanupRetainedRawExports.
Each task reads/writes separate row sets; no cross-task D1 conflicts.
Deferred media batch — parallel candidate loading and R2 PUTs.
getDeferredMediaReadiness and processDeferredMediaBatch load image/audio/video candidates in parallel. Within each batch, R2 PUTs and D1 status updates are collected and fired in parallel via Promise.all + db.batch() rather than in sequential loops.
Batching
Media content-hash dedup — single IN query per export.
Instead of one SELECT per artifact, all content hashes are collected before the loop and fetched in a single SELECT ... WHERE content_hash IN (?) (chunked at MEDIA_DEDUP_CHUNK_SIZE=24 to stay within D1 variable limits). For a 50-artifact export this reduces 50 queries to 3.
Deferred media status updates — db.batch() instead of sequential UPDATEs.
Status transitions (mark as deferred_processing, reset on failure, mark as pending_analysis) are batched into single db.batch() calls rather than one UPDATE per row.
Deferred batch limit increased.
enqueueDeferredMediaBatches was capped at 3 exports per cron tick. Increased to 12 to drain backlogs faster during high-ingestion periods.
Caching
Quota bypass flag — in-memory cache with 30s TTL.
The module-level _mediaConfigCache variable caches quota settings within a Worker isolate. setMediaQuotaBypass() calls invalidateMediaConfigCache() immediately on write so the new state is reflected on the next read within the same isolate without waiting for TTL expiry.
Quota bypass flag — optional KV cross-invocation cache.
When beacon_pulse_cache KV namespace is provisioned, bypass flag is also written to KV with 60s TTL. This reduces D1 reads across separate Worker invocations. Falls back gracefully to D1 when KV is absent.
Public API responses — Cloudflare Cache API.
/pulse.json, /pulse/history.json, /pulse/concepts.json, and /pulse/daily.json are cached via caches.default with a 5-minute TTL. The cache key is the full request URL. Cache is written on first miss; subsequent requests within the TTL window skip D1 entirely.
Pagination
/pulse/history.json — cursor-based pagination.
The legacy LIMIT ? OFFSET ? pattern degrades as offset grows (D1 scans all prior rows). Cursor mode uses WHERE week_start < ? ORDER BY week_start DESC LIMIT ? — O(1) per page regardless of history depth. next_cursor is returned in the response. Legacy offset mode is retained for backward compatibility. See api-contracts for full pagination spec.
Async Pillar Assignment
Concept graph pillar assignment is deferred.
Weekly summary generation previously ran AI pillar assignment synchronously as part of the pipeline, adding one AI call per week. Now, the concept graph is saved immediately using keyword-only pillar assignment (assignPillarsKeywordOnly), then a backfill_concept_pillars queue message is enqueued for async AI-quality refinement. The public API handles null pillar fields gracefully during the brief window before backfill runs.
Dual Cron Schedules
Cron split into two schedules.
All 7 self-healing tasks previously ran in a single */5 * * * * tick. Slow cleanup operations could delay stuck-export detection. Now:
*/5 * * * *— high-priority (stuck detection + 5 retry tasks)*/15 * * * *— low-priority (deferred media drain + raw export cleanup)
See queue-architecture for the full task schedule.
Indexes Added
| Migration | Index | Benefit |
|---|---|---|
fix_daily_digests_weekly_summaries_pk.sql | Full PK on daily_digests (community_id, source_id, day_date) | Prevents cross-source digest collision |
add_export_media_community_index.sql | (export_id, community_id) on export_media | Faster JOIN in /pulse/history.json |
add_exports_status_index.sql | (status, community_id) on exports | Faster cron stuck-export detection |
Parallelization Safety
| Operation | Safe to Parallelize? | Notes |
|---|---|---|
| Image Stage 1 classifier + detector | Yes | Independent; same input bytes |
| Video frame extraction | Yes | Different timestamps; no shared state |
| Video frame Stage 1 per frame | Yes | Independent; quota cap still enforced |
| Weekly regen across sources | Yes | Different source_id rows; no cross-source dependency |
| Cron sub-tasks | Yes | Each reads/writes separate row sets |
| Deferred media readiness checks | Yes | Read-only; independent per export |
| Weekly regen within one source | No | Each week may compare against previous week |
| Daily digest generation within export | Configurable | Parallelizable with quota stagger |
| Video frame Stage 2 caption decisions | No | Sequential to protect stage2Used counter |
Caching Reference
| What | Layer | TTL | Invalidation |
|---|---|---|---|
| Quota bypass flag (in-process) | Module-level variable | 30s | On every setMediaQuotaBypass() call |
| Quota bypass flag (cross-invocation) | KV (optional) | 60s | On every /quota/bypass POST |
| Public API responses | Cloudflare Cache API | 5 min | By URL key; stale on write within TTL |
| Quota config (model rates, limits) | In-memory per invocation | Invocation lifetime | Re-read next invocation |
| Community/source lists | Cloudflare Cache API | 30 min | On community/source mutation |
Known Limits and Risks
| Area | Risk | Mitigation |
|---|---|---|
| Image Stage 1 parallel | Doubled concurrent AI calls per item | Quota is per-day, not per-second; acceptable |
| Video frame parallel | Higher burst AI usage during video processing | maxVideoFrameStage2PerExport cap still applies |
| Parallel weekly regen | D1 write contention on ON CONFLICT upserts | Safe — upsert uses full PK including source_id; no cross-source conflicts |
| Cache API stale reads | Up to 5 minutes after digest write | Acceptable for community health product; TTL is documented |
| KV bypass stale | Up to 60s stale bypass state | Admin-only feature; acceptable |
| Cache API in test runtime | caches.default behaves differently in workerd test mode | Spy on cache calls; validate in staging |
| D1 variable limit | Large IN queries (media dedup, batch inserts) | Chunked at MEDIA_DEDUP_CHUNK_SIZE=24 and DAY_BATCH_SIZE=50 |