Workers and Jobs

@ai/jobs uses BullMQ with Redis (REDIS_URL) and starts workers from packages/jobs/worker.ts.

Queues

• hello-world
• transcription
• file-indexing
• log-cleanup

Start Worker Processes

Local worker dev mode:

bun dev:worker

Direct package script:

bun --filter=@ai/jobs worker:dev

Docker worker profile:

docker compose --profile worker up -d worker

Scheduled Jobs

log-cleanup repeat job is registered on worker startup:

• schedule: daily at 02:00 UTC
• default retention payload: 30 days
• existing repeatable entries are removed then re-registered to avoid duplicates

Queue Behavior

• retries are configured per queue with exponential backoff
• transcription and file-indexing workers run with concurrency 2
• log-cleanup runs serially (concurrency: 1)
• job cleanup is enabled (removeOnComplete / removeOnFail thresholds)

Operational Checks

docker compose exec redis redis-cli ping
docker compose logs -f worker
docker compose logs -f app

Inspect BullMQ keys:

docker compose exec redis redis-cli
# KEYS bull:*

Failure Patterns to Watch

• worker starts but no throughput: Redis connectivity or missing worker process
• repeated transcription failures: AI provider keys/quotas or converter/downstream file issues
• file indexing failures: extraction service failures, storage download issues, or memory service connectivity
• log cleanup not running: worker not running or schedule registration not completed at startup

Safe Operations Checklist

• REDIS_URL resolves from worker runtime context
• Worker process is running continuously in target environment
• Logs are monitored for failed jobs and worker errors
• Optional features are disabled when dependent services are absent