BullMQ Job Retry Strategies: From Automatic Retries to Dead Letter Queues

Job failures are inevitable in distributed systems. A network blip takes down your upstream API for three seconds. A database deadlock stalls a transaction. A memory spike causes a worker to crash mid-process. How you handle these failures determines whether your system is fragile or resilient.

BullMQ — the most popular Redis-backed job queue for Node.js — gives you a rich toolkit for handling failures. In this post, you'll learn:

• How automatic retries work with the attempts option
• Fixed, exponential, and custom BullMQ backoff strategies with real code
• Stalled jobs — what causes them and how to prevent them
• Manual retry APIs and the UnrecoverableError pattern
• How to build a dead letter queue pattern with BullMQ
• How QueueHub helps you monitor, debug, and retry failed jobs at scale

Whether you're new to background job processing or you're tuning an existing production queue, this guide covers everything you need to build a resilient job queue retry strategy.

---

Understanding Job Failures in BullMQ

Before diving into retry configurations, it helps to understand how and why jobs fail in BullMQ.

How Jobs Fail

There are three main paths to failure:

1. Processor throws an error — The most common case. Your worker function throws an Error object (or a subclass like UnrecoverableError).
2. Job stalls — The worker grabs a lock on a job but stops processing without completing or failing it. BullMQ's stall detection system eventually intervenes.
3. Force-failed via API — You explicitly move a job to the failed state using the SDK.

The Job Lifecycle (Simplified)

waiting → active → completed
              ↓
           failed (or → delayed → waiting, if retrying)

Default Behavior

By default, BullMQ does not retry failed jobs. The attempts option defaults to 1 — one initial try and done. Failed jobs land in the "failed" set and stay there until they're cleaned up by removeOnFail settings or manually removed.

// A job that fails once and stays failed
await queue.add('email', { to: 'user@example.com' });
// attempts defaults to 1 — no retry

---

Automatic Retries: The attempts Option

The simplest way to introduce BullMQ job retries is the attempts option. It tells BullMQ how many times to try processing a job before considering it permanently failed.

Basic Retry Configuration

await queue.add('send-email', { to: 'user@example.com' }, {
  attempts: 3,  // 1 initial try + 2 retries
});

With attempts: 3, BullMQ will retry the job up to two more times after the initial failure. Each retry re-enters the waiting state and gets picked up by an available worker.

How BullMQ Tracks Attempts

BullMQ maintains two counters on every job:

• job.attemptsMade — Incremented on each failure (except for RateLimitError, DelayedError, and WaitingChildrenError)
• job.attemptsStarted — Incremented every time the job moves to the active state

Retried jobs re-enter the waiting queue respecting their priority, so higher-priority jobs still get processed first.

Queue-Level Defaults

You don't need to set retry options on every individual job. Configure them once on the queue:

const queue = new Queue('orders', {
  defaultJobOptions: {
    attempts: 3,
    backoff: { type: 'exponential', delay: 1000 },
  },
});

Now every job added to orders automatically gets 3 attempts with exponential backoff. Individual jobs can still override these defaults.

Retry Exhaustion

When all attempts are exhausted, the job lands in the failed set permanently. The queue emits a failed event you can listen to, and the job is available for manual inspection or dead letter processing.

---

Backoff Strategies

Retrying immediately after a failure is rarely the right choice. If a database is overwhelmed, retrying instantly makes things worse. BullMQ backoff strategies introduce a delay between retries, giving your system time to recover.

Fixed Backoff

Fixed backoff uses a constant delay between each retry attempt. It's ideal for rate-limited APIs or predictable recovery scenarios.

await queue.add('api-call', { url: 'https://api.example.com/data' }, {
  attempts: 5,
  backoff: { type: 'fixed', delay: 2000 },
});
// Retries at: 2s, 4s, 6s, 8s after each failure

Each retry waits exactly delay milliseconds from the moment the previous attempt failed.

Exponential Backoff

Exponential backoff is the gold standard for transient failures. The delay doubles with each retry: delay = 2^(attemptsMade - 1) * baseDelay.

await queue.add('webhook', { payload: { event: 'order.created' } }, {
  attempts: 5,
  backoff: { type: 'exponential', delay: 1000 },
});
// Retries at: 1s, 2s, 4s, 8s, 16s after each failure

This strategy is excellent for cascading failures, server overload situations, and any scenario where a quick recovery is unlikely.

Adding Jitter

Without jitter, all retrying jobs resume at the same interval, creating a thundering herd when the system recovers. Jitter spreads retries across a time window.

await queue.add('notification-batch', { ids: [1, 2, 3, 4, 5] }, {
  attempts: 8,
  backoff: { type: 'exponential', delay: 1000, jitter: 0.5 },
});

The jitter value ranges from 0 (no randomness) to 1 (full randomization):

• Fixed with jitter: random delay between delay and delay * (1 + jitter)
• Exponential with jitter: random delay between the calculated value and half of it

No Backoff (Immediate Retry)

If you omit the backoff option entirely, retries happen immediately with no delay. This is risky — it can overwhelm downstream services and mask the underlying problem. Always pair attempts > 1 with a backoff strategy.

Custom Backoff Strategies

For advanced use cases, BullMQ lets you define a custom backoff function on the worker. This is the most flexible queue retry mechanism BullMQ offers.

const worker = new Worker('orders', async job => {
  // process order
}, {
  settings: {
    backoffStrategy: (attemptsMade: number, type: string, err: Error, job: Job) => {
      // Error-aware backoff
      if (err.message.includes('rate_limit')) {
        return 60_000; // Wait a full minute
      }
      return attemptsMade * 5_000; // Linear: 5s, 10s, 15s...
    },
  },
});

await queue.add('process-order', { orderId: 123 }, {
  attempts: 3,
  backoff: { type: 'custom' },
});

The backoffStrategy function receives the attempt count, the backoff type string (from the job options), the error that caused the failure, and the job itself. You can implement any logic you need.

Multiple Custom Strategies

You can even support multiple named strategies within a single worker:

settings: {
  backoffStrategy: (attemptsMade, type, err, job) => {
    switch (type) {
      case 'linear': return attemptsMade * 2000;
      case 'aggressive': return Math.min(attemptsMade * 1000, 10_000);
      default: return 1000;
    }
  },
}

Key behaviors:

• Return 0 → retry immediately (job moves to the end of the waiting list)
• Return -1 → fail the job immediately with no more retries

---

Stalled Jobs: Detection and Handling

Stalled jobs are one of the most misunderstood aspects of BullMQ. A stall happens when a worker holds a lock on a job but stops renewing it — the job is "stuck" in the active state without making progress.

What Is a Stalled Job?

When a worker starts processing a job, it acquires a lock (default TTL: 30 seconds). The worker must periodically renew this lock by calling job.updateProgress() or through the heartbeat mechanism. If the lock expires without renewal, BullMQ's stall detection system moves the job back to the waiting queue so another worker can pick it up.

Importantly, there is no "stalled" state in BullMQ — only a stalled event. The job itself transitions from active back to waiting (or to failed if it exceeds maxStalledCount).

Common Causes

• Worker crash — process.exit(), uncaught exception, OOM killer
• CPU-bound operations — Heavy computation that blocks the event loop longer than lockDuration
• Infinite loops or deadlocks — Synchronous blocking code that never yields
• Network partitions — Worker loses connectivity to Redis

Configuration Parameters

const worker = new Worker('paint', processor, {
  connection,
  lockDuration: 30_000,       // Lock TTL (default: 30s)
  stalledInterval: 15_000,    // How often to check for stalls
  maxStalledCount: 1,         // Max stalls before failing (default: 1)
});

How Stall Detection Works

1. Worker grabs a lock when it starts processing a job
2. Worker must renew the lock before lockDuration expires
3. If the lock expires, another worker (or the queue) detects the stall
4. The job moves back to the waiting queue
5. If maxStalledCount is exceeded, the job is permanently failed

Preventing Stalls

• Keep the event loop responsive — avoid synchronous CPU-heavy work in the processor
• Set timeouts on all external calls (HTTP requests, database queries)
• Break long operations into smaller, awaitable chunks
• Use sandboxed processors for risky or CPU-intensive code
• Set lockDuration higher than your worst-case processing time

Sandboxed Processors

For CPU-heavy jobs, BullMQ's sandboxed processors run the job in a separate process:

// main.ts
const worker = new Worker('paint', './painter.js', { connection });

// painter.ts — runs in a separate process
export default async (job) => {
  // CPU-heavy work won't stall because it's isolated
  const result = performExpensiveRender(job.data.scene);
  return result;
};

Listening for Stalled Events

worker.on('stalled', (jobId: string) => {
  console.warn(`Job ${jobId} stalled — moving back to waiting`);
  // Alert your monitoring system
});

Track stall events in your monitoring system — a high stall rate indicates worker health problems or misconfigured lock durations.

---

Manual Job Retries with job.retry()

Automatic retries handle transient failures, but some scenarios require manual intervention. BullMQ's job.retry() method lets you retry jobs on demand.

When to Use Manual Retries

• After fixing the root cause of a failure (e.g., a misconfigured API key)
• Re-processing completed jobs after a system bug was patched
• Workflow recovery after a deployment rollback

Basic Usage

import { Queue, Job } from 'bullmq';

const queue = new Queue('my-queue');
const job = await Job.fromId(queue, 'job-id');

// Retry a failed job
await job.retry();

// Retry a completed job
await job.retry('completed');

The first argument specifies which state the job should be retried from — 'failed' (default) or 'completed'.

Resetting Attempt Counters

When you manually retry, BullMQ preserves the attemptsMade counter by default. This means if a job exhausted its 5 attempts and you retry it manually, it might not retry again on failure. Use resetAttemptsMade to give the job a clean slate:

// Reset attemptsMade so the job gets a full retry allowance
await job.retry('failed', { resetAttemptsMade: true });

// Reset both counters
await job.retry('failed', {
  resetAttemptsMade: true,
  resetAttemptsStarted: true,
});

What Happens When You Retry

1. Job moves from the completed/failed set back to the waiting queue
2. failedReason, finishedOn, processedOn, and returnvalue are cleared
3. A waiting event is emitted
4. Parent dependencies are restored (for flow jobs)

Error Handling

try {
  await job.retry('failed');
} catch (error) {
  // -1: job doesn't exist
  // -3: job not in expected state
  console.error('Retry failed:', error.message);
}

---

Stopping Retries: The UnrecoverableError Pattern

Not all failures should be retried. Some errors are permanent — no amount of retries will fix them. For these cases, BullMQ provides the UnrecoverableError class.

When to Stop Retrying

• Invalid input data — Malformed JSON, missing required fields, nonexistent user IDs
• Authentication failures — Expired API keys, revoked tokens
• Business logic violations — Trying to process an already-cancelled order
• Validation errors — Data that fails schema validation

Using UnrecoverableError

import { Worker, UnrecoverableError } from 'bullmq';

const worker = new Worker('orders', async job => {
  const { userId } = job.data;
  const user = await db.findUser(userId);

  if (!user) {
    // No point retrying — user doesn't exist
    throw new UnrecoverableError('User not found');
  }

  await processOrder(user);
});

Behavior

When you throw UnrecoverableError:

• The job goes directly to the failed set — no retries are attempted
• It overrides any attempts setting on the job
• It's the foundation for building a dead letter queue pattern

Rate-Limit-Aware Retry Exhaustion

Combine RateLimitError and UnrecoverableError for intelligent retry management:

if (job.attemptsStarted >= job.opts.attempts) {
  throw new UnrecoverableError('Max attempts reached');
}
throw new RateLimitError();

This pattern sends a rate-limit signal until max attempts are reached, then permanently fails the job.

---

Dead Letter Queue Pattern with BullMQ

A dead letter queue (DLQ) is a separate queue that stores permanently failed jobs for later inspection and replay. It's a standard pattern in message queuing systems, and BullMQ makes it straightforward to implement.

Concept

Instead of letting permanently failed jobs clutter your main queue (and occupy Redis memory), you move them to a dedicated DLQ queue. This keeps your main queue clean and focused on active work, while the DLQ serves as an audit trail for failures.

Implementation

const dlq = new Queue('orders-dlq');

const worker = new Worker('orders', async job => {
  // process...
}, {
  connection,
});

worker.on('failed', async (job, err) => {
  if (job.attemptsMade >= job.opts.attempts) {
    // Copy to DLQ for later inspection
    await dlq.add(job.name, {
      ...job.data,
      _originalJobId: job.id,
      _failedReason: err.message,
      _failedAt: new Date().toISOString(),
    });
  }
});

The DLQ stores metadata about the original job — the _originalJobId, _failedReason, and _failedAt timestamp — so you can trace failures back to their source.

Retention with removeOnFail

Control how long failed jobs stay in your main queue with removeOnFail:

await queue.add('process-payment', { amount: 100 }, {
  attempts: 3,
  backoff: { type: 'exponential', delay: 2000 },
  removeOnFail: {
    age: 7 * 24 * 3600,  // Keep failed jobs for 7 days
    count: 10_000,        // Max 10K failed jobs
  },
});

This keeps your Redis memory usage predictable while your DLQ retains the permanent record.

---

Monitoring Retries with QueueHub

Configuring retries in code is only half the battle. To operate a system in production, you need real-time visibility into what's happening with your jobs.

Why Monitoring Matters

• Retry loops can hide systemic issues — A job that retries 20 times before succeeding might mask an upstream service degradation
• Stalled jobs indicate worker health problems — A high stall rate might mean your workers are under-provisioned
• Failed jobs need human attention — Even with automatic retries, permanently failed jobs require investigation

QueueHub Features for Retry Management

QueueHub (queuehub.tech) is a modern queue monitoring dashboard built specifically for BullMQ and BeeQueue. Here's how it helps you manage BullMQ job retries:

• Real-time dashboard — Live throughput, failure rate, and retry count charts give you instant visibility into queue health
• Job detail view — See attemptsMade, failedReason, stack traces, and full job data at a glance
• Bulk retry actions — Select multiple failed jobs and retry them in one click with optional attempt counter reset
• Live worker view — See which workers are active, stalled, or disconnected in real time
• Failed jobs filter — Search and filter by error type, job name, and time range to quickly find problematic jobs
• Multi-backend support — Works with BullMQ and BeeQueue across local Redis, TLS connections, Valkey, and AWS ElastiCache
• Agent tunneling — Connect to private Redis instances securely without opening firewall ports

Workflow: From Monitoring to Action

1. Spot a spike in failed jobs on the QueueHub dashboard
2. Inspect the job detail panel — see the failedReason is "Connection refused" on the upstream API
3. Fix the upstream service or update the endpoint URL
4. Bulk-select the failed jobs and click "Retry" with resetAttemptsMade enabled
5. Watch the success rate recover in real-time on the dashboard

---

Best Practices Summary

Area	Recommendation
Attempts	Start with 3–5 attempts; higher for idempotent jobs
Backoff	Use exponential backoff with jitter for most cases
Lock duration	Set to maxExpectedProcessingTime × 2
Stalled count	Keep at 1 (default); increase only if genuinely needed
Dead letter queue	Implement for permanently failed jobs
Monitoring	Use QueueHub for real-time visibility
Unrecoverable errors	Throw UnrecoverableError for non-retryable failures
Retention	Set removeOnFail with age/count limits

Anti-Patterns to Avoid

• ❌ Setting attempts: Infinity — your system will retry forever, masking issues and consuming resources
• ❌ Disabling backoff for retries — creates thundering herd problems on recovery
• ❌ Ignoring stalled events — stalls are early warnings of worker health problems
• ❌ Not resetting attemptsMade when manually retrying old jobs — the job will fail again without a full retry allowance
• ❌ Keeping unlimited failed jobs in Redis — unbounded memory growth leads to OOM issues

---

Conclusion

BullMQ provides one of the richest retry toolkits of any Node.js job queue. From the simple attempts option to custom backoff strategies, from stall detection to manual retry APIs, you have everything you need to build a resilient background job system.

The right BullMQ retry strategy depends on your job types:

• Network calls → exponential backoff with jitter
• Data validation → fail fast with UnrecoverableError
• Rate-limited APIs → fixed backoff or custom rate-limit-aware strategies
• CPU-heavy processing → sandboxed processors with appropriate lockDuration

Stalled jobs are normal but must be monitored — they signal worker health issues or lock configuration problems. And when automatic retries aren't enough, job.retry() and the dead letter queue pattern give you manual control.

Tools like QueueHub close the loop: configure retries in code, monitor them in a real-time dashboard, and take action with one click. Start monitoring your BullMQ queues today at queuehub.tech — it's free to get started.