Mastering BullMQ Rate Limiting: Global, Group, and Dynamic Throttling

TL;DR: BullMQ's rate limiter controls how many jobs are processed (not added) per time window, enforced at the Worker level using Redis. This guide covers the global rate limiter with limiter.max/limiter.duration, dynamic rate limiting via worker.rateLimit(), and BullMQ Pro's group rate limiter for multi-tenant fairness. We'll also explore smoothing strategies, concurrency tuning, and monitoring with Queue Hub.

1. Introduction

Rate limiting is one of the most critical yet often overlooked features in job queue systems. Without it, a burst of jobs can overwhelm your downstream services — whether that's a third-party API with hard rate caps, a database that can only handle so many connections, or a worker process with finite CPU and memory.

BullMQ, the popular Redis-backed job queue for Node.js, offers a surprisingly flexible rate limiting system. Unlike some queue frameworks that only limit how fast jobs are enqueued, BullMQ limits how fast jobs are processed. This distinction matters because it means you can enqueue jobs freely and let the worker naturally pace their execution.

There are three tiers of rate limiting in BullMQ:

Global Rate Limiter (OSS) — A single throughput cap enforced across all workers sharing a queue name
Manual / Dynamic Rate Limiting (OSS) — Programmatically pause processing when you detect backpressure (e.g., a 429 response)
Group Rate Limiter (BullMQ Pro) — Per-group rate limit counters so noisy tenants don't starve others

This makes BullMQ well-suited for everything from protecting external API endpoints (Stripe, SendGrid, Twilio) to ensuring fair scheduling in multi-tenant SaaS platforms.

If you're managing BullMQ queues in production, tools like Queue Hub provide real-time dashboards for monitoring throughput, waiting job counts, and rate limit bottlenecks — we'll cover that in Section 8.

2. Understanding the BullMQ Rate Limiter

2.1 How It Works

BullMQ's rate limiter is built on Redis atomic counters. When you configure a worker with a limiter option, every time the worker completes a job, it increments a Redis counter scoped to that queue name. If the counter exceeds limiter.max within the limiter.duration window, the worker pauses fetching new jobs until the window resets.

Key behaviors to understand:

Rate-limited jobs stay in the waiting state — they are not moved to delayed or failed. The worker simply won't pick them up until the rate limit window clears.
The rate limit is global per queue — all workers sharing the same queue name share a single Redis counter. Adding more workers does not bypass the limit.
Rate limiting is enforced at the Worker, not the Queue — the Queue object exists to add jobs and inspect state; the Worker enforces processing constraints.
Internally, BullMQ tracks rate limit state using Redis keys like {queue}:limiter.

2.2 Configuration — `limiter.max` and `limiter.duration`

The two core configuration parameters live on the Worker options:

import { Worker } from 'bullmq';

const worker = new Worker(
  'api-queue',
  async (job) => {
    await callExternalApi(job.data);
  },
  {
    limiter: {
      max: 100,      // Process at most 100 jobs
      duration: 1000, // Per 1000 ms (1 second)
    },
    concurrency: 10,
  }
);

limiter.max (number): Maximum number of jobs to process within the time window
limiter.duration (number): Time window in milliseconds
concurrency (optional, default 1): Works with the rate limiter — a single worker can process jobs in parallel, but total throughput across all workers is still capped by max/duration

2.3 Per-Second, Per-Minute, Per-Hour Patterns

You can express any time-based rate by adjusting duration:

// 100 jobs per second
{ max: 100, duration: 1000 }

// 1,000 jobs per minute
{ max: 1000, duration: 60000 }

// 10,000 jobs per hour
{ max: 10000, duration: 3600000 }

Choose the window that matches your downstream service's throttling behavior. Many APIs rate-limit per second, but some (like certain CRM platforms) use per-minute or per-hour windows.

2.4 Important Behavioral Details

QueueScheduler is not required from BullMQ v2.0+. Earlier versions needed it for rate limiting to function; modern BullMQ handles it internally.
Rate-limited jobs remain waiting — they are picked up the moment the rate limit window resets. No polling or manual intervention needed.
maximumRateLimitDelay (Worker option, default 30000ms): The maximum time the worker will idle while rate-limited before fetching again. If you need tighter pacing, lower this value.
Rate limiting is per-queue, not per-worker — adding more workers scales concurrency but not throughput past the limit.

3. Smooth vs. Burst Rate Limiting

3.1 The Burst Problem

With { max: 100, duration: 1000 }, BullMQ may process all 100 jobs in a burst at the very start of each second. If your downstream service expects evenly-spaced requests or has its own sliding-window rate limiter, bursts can trigger 429 errors, connection pool exhaustion, or database lock contention.

Consider an external API that limits to 10 requests per second with a 100ms minimum gap. Processing all 10 jobs in the first 100ms of each second will likely get you throttled, even though mathematically you're "under the limit."

3.2 Smoothing with Small Windows

The simplest smoothing trick is to use tiny windows with max: 1:

// Smooth: 1 job every ~3.33ms → 300 jobs/second evenly spaced
const worker = new Worker('smooth-queue', processor, {
  limiter: {
    max: 1,
    duration: 3.33,
  },
});

By processing at most one job per tiny window, BullMQ naturally spaces job execution evenly. The trade-off is slightly higher Redis overhead from more frequent counter checks, but for most workloads this is negligible.

For example, to pace requests at 10 per second with even spacing:

// 1 job every 100ms → exactly 10 jobs/second, perfectly spaced
{ max: 1, duration: 100 }

3.3 Burst-Capable Pattern

Sometimes bursts are desirable — for example, when a downstream service has a burst allowance (Stripe allows short bursts above the sustained limit). In that case, keep larger windows:

// Allow bursts of 200 in the first second
const worker = new Worker('bursty-queue', processor, {
  limiter: {
    max: 200,
    duration: 1000,
  },
});
// Pair with a separate cooldown mechanism at the application layer

The right choice depends on your downstream constraints. When in doubt, start smooth and only increase window size if you observe the downstream service can handle bursts.

4. Real-World Use Cases

4.1 Preventing Worker Overload

CPU-intensive workers can exhaust system memory if too many jobs run concurrently. Rate limiting gives you a second dial beyond concurrency:

const worker = new Worker('render-queue', async (job) => {
  await renderVideoFrame(job.data);
}, {
  concurrency: 5,
  limiter: { max: 20, duration: 1000 },
});

In this example, up to 5 jobs run in parallel, but the total throughput is capped at 20 jobs per second. This prevents the worker from accepting more work than it can handle, even during a sudden job burst.

4.2 Controlling External API Call Rates

A common pattern: one queue per external service, each with its own limiter:

const workers = {
  stripe:   new Worker('api-stripe',   processApiCall, { limiter: { max: 100, duration: 1000 } }),
  sendgrid: new Worker('api-sendgrid', processApiCall, { limiter: { max: 50,  duration: 1000 } }),
  twilio:   new Worker('api-twilio',   processApiCall, { limiter: { max: 10,  duration: 1000 } }),
};

This keeps rate limits isolated per service. A burst of Twilio jobs won't consume Stripe's budget.

4.3 Fair Scheduling Across Tenants (Multi-Tenant)

Consider a SaaS platform processing background jobs for 1,000 customers. Without rate limiting, one customer who enqueues 10,000 jobs can starve all others. BullMQ Pro's group rate limiter (Section 6) solves this elegantly by giving each tenant its own rate limit counter.

5. Manual / Dynamic Rate Limiting

5.1 Reacting to 429 Responses

Many APIs return 429 Too Many Requests with a Retry-After header. BullMQ's worker.rateLimit() lets you dynamically pause the queue in response:

import { Worker } from 'bullmq';

const worker = new Worker(
  'api-queue',
  async (job) => {
    const response = await callExternalApi(job.data);
    if (response.status === 429) {
      const retryAfterMs = response.headers['retry-after'] * 1000;
      await worker.rateLimit(retryAfterMs);
      // CRITICAL: Re-enqueue the job by throwing RateLimitError
      throw Worker.RateLimitError();
    }
    return response.data;
  },
  {
    limiter: { max: 100, duration: 1000 }, // Must be set even for dynamic limiting
    concurrency: 25,
  },
);

5.2 The `Worker.RateLimitError()` Contract

Throwing Worker.RateLimitError() tells BullMQ that the job was not truly "failed" — it should return to the waiting state and be retried naturally. Without this, BullMQ would mark the job as failed and move it to the failed set, which is almost certainly not what you want.

Always pair worker.rateLimit(duration) with throw Worker.RateLimitError():

worker.rateLimit(ms) tells the queue to pause for ms milliseconds
throw Worker.RateLimitError() tells BullMQ to return the job to waiting (not failed)
After the pause, the job will be picked up and retried

5.3 Inspecting Rate Limit State

Proactively checking rate limit state helps avoid surprises:

import { Queue } from 'bullmq';

const queue = new Queue('api-queue', { connection });

// Check if queue will be rate limited for a given number of jobs
const ttl = await queue.getRateLimitTtl(100);
if (ttl > 0) {
  console.log(`Queue is rate limited — TTL remaining: ${ttl}ms`);
}

// Reset the rate limit counter
await queue.removeRateLimitKey();
console.log('Rate limit counter reset');

getRateLimitTtl(limit) returns how many milliseconds until the queue can process limit more jobs. A return value of 0 means no rate limit is currently active for that many jobs.

removeRateLimitKey() is useful during testing, maintenance windows, or when you need to manually reset the counter after changing limiter configuration.

6. Group Rate Limiter (BullMQ Pro)

Note: Group rate limiting is a BullMQ Pro feature (@taskforcesh/bullmq-pro). It was part of open-source BullMQ v2.x via groupKey but was removed in v3.0+. If you need per-group rate limiting, you need the Pro package.

6.1 Why Group Rate Limiting?

Global rate limits apply across all jobs equally. In a multi-tenant system, this means a single noisy tenant can consume the entire rate limit budget, starving every other tenant.

Group rate limits solve this by giving each group (user, customer, organization, API key, endpoint) its own independent counter. Here's the key benefit: non-rate-limited groups continue processing normally while other groups are paused.

6.2 Configuration

import { WorkerPro } from '@taskforcesh/bullmq-pro';

const worker = new WorkerPro(
  'multi-tenant-queue',
  async (job) => {
    await processTenantJob(job);
  },
  {
    concurrency: 50,
    group: {
      limit: {
        max: 50,      // Per-group: 50 jobs
        duration: 1000, // Per second
      },
    },
  },
);

6.3 Adding Jobs with Groups

import { QueuePro } from '@taskforcesh/bullmq-pro';

const queue = new QueuePro('multi-tenant-queue', { connection });

// Each job is tagged with a group ID
await queue.add('email-send', { to: 'user@a.com' }, { group: { id: 'tenant-a' } });
await queue.add('email-send', { to: 'user@b.com' }, { group: { id: 'tenant-b' } });
await queue.add('email-send', { to: 'user@c.com' }, { group: { id: 'tenant-c' } });

The group.id uniquely identifies the group. Each group gets its own independent rate limit counter in Redis.

6.4 Manual Group Rate Limiting (Dynamic)

Just like the global dynamic rate limiter, groups support dynamic rate limiting:

import { WorkerPro } from '@taskforcesh/bullmq-pro';

const worker = new WorkerPro(
  'multi-tenant-queue',
  async (job) => {
    const groupId = job.opts.group.id;
    const result = await callExternalApi(job.data);

    if (result.status === 429) {
      // Rate limit only THIS group, not all groups
      await worker.rateLimitGroup(job, result.retryAfterMs);
      throw Worker.RateLimitError();
    }

    return result;
  },
  { group: { limit: { max: 50, duration: 1000 } } },
);

This is powerful: a 429 from an API called by one tenant pauses only that tenant's group, while other tenants continue processing normally.

6.5 Local Group Rate Limits (BullMQ Pro)

You can set specific rate limits per group, overriding the default group.limit:

// Set a per-group override
await queue.rateLimitGroup('premium-tenant', { max: 200, duration: 1000 });
await queue.rateLimitGroup('free-tenant',     { max: 10,  duration: 1000 });

This is ideal for tiered service plans. Premium customers get higher throughput, free-tier customers are throttled more aggressively — all within the same queue.

7. Combining Rate Limiting with Concurrency

7.1 The Concurrency-Rate Limit Relationship

concurrency controls how many jobs a worker processes in parallel. limiter controls how many jobs are processed per unit time. They work together:

High concurrency + no limiter = maximum throughput, risk of overload
High concurrency + rate limiter = parallel but throttled, ideal for I/O-bound tasks
Low concurrency + rate limiter = safe but potentially underutilized

7.2 Calculating Optimal Concurrency

To sustain a given rate limit, you need enough concurrency to keep the pipeline full:

function calculateOptimalConcurrency(
  rateLimit: number,     // e.g., 100
  avgJobDurationMs: number // e.g., 200
): number {
  // Minimum concurrency to sustain the rate limit
  const minConcurrency = Math.ceil(rateLimit * (avgJobDurationMs / 1000));
  // Add 20% buffer
  return Math.ceil(minConcurrency * 1.2);
}

// Example: 100 req/s, each job takes 200ms → ~24 concurrent
const concurrency = calculateOptimalConcurrency(100, 200);

The formula: if each job takes 200ms and you need 100 jobs/second, each worker processes ~5 jobs/second (1000ms / 200ms). So you need at least 20 concurrent slots, plus some buffer for variance.

7.3 I/O-Bound vs. CPU-Bound Considerations

Job Type	Strategy	Rationale
I/O-bound (API calls, DB queries)	1 worker, high concurrency	Async I/O is efficient per process; overhead is minimal
CPU-bound (image processing, video transcoding)	Multiple workers, low concurrency	Leverage multiple CPU cores; avoid GIL-like contention

For I/O-bound workloads, a single worker with high concurrency and a rate limiter is often optimal. For CPU-bound work, run multiple workers (processes) with low per-worker concurrency.

8. Monitoring Rate Limits with Queue Hub

8.1 What Queue Hub Shows You

Configuring rate limits is only half the battle. You also need visibility into how they're performing. Queue Hub provides:

Throughput charts: Jobs processed per second/minute — spot exactly when rate limiting is active and how close you are to the cap
Queue metrics: Waiting job counts that spike when rate limiting kicks in (a sudden jump in waiting jobs with stable worker count is the classic symptom)
Worker dashboard: Live view of connected workers, their concurrency, and processing state
Job detail view: Inspect individual jobs that were rate-limited, including how long they spent in the waiting state
Multi-backend support: Works with both BullMQ and BeeQueue

8.2 Detecting Rate Limit Issues

Here are the common signals that your rate limiting configuration needs attention:

Sudden increase in waiting job count with stable worker count — rate limit is being hit. Your limiter.max may be too low or downstream services are slower than expected.
Jobs stuck in "waiting" longer than expected — limiter.max may be too low or duration too large for the incoming job volume.
Uneven group processing (BullMQ Pro) — check per-group rate limit counters. A single tenant may be consuming all available worker capacity.
High concurrency but low actual throughput — the rate limiter is capping throughput; consider whether smoothing would improve downstream behavior.

8.3 Tuning Rate Limits with Observability

A typical Queue Hub dashboard view for a rate-limited queue:

┌─────────────────────────────────────────────────┐
│              Queue Hub Dashboard             │
├─────────────────────────────────────────────────┤
│  Queue: api-queue                                │
│  Throughput: 95 jobs/sec (rate limit: 100/sec)   │
│  Waiting: 1,203 jobs  ●●●●●●●●●○  94% utilized  │
│  Workers: 4 active / 2 idle                      │
│  Rate limit hit: 12 times in last 5 min          │
└─────────────────────────────────────────────────┘

Seeing 94% utilization with minimal rate limit hits means your configuration is well-tuned. If you see frequent rate limit hits with low utilization, your concurrency may be too low to sustain the rate limit (see Section 7.2).

9. Best Practices and Common Pitfalls

9.1 Do's

✅ Always set limiter on the Worker, not the Queue. The Queue doesn't enforce rate limits.
✅ Pair worker.rateLimit() with throw Worker.RateLimitError() — without it, the job goes to failed.
✅ Use separate queues for services with different rate limits (Stripe vs. Twilio vs. SendGrid).
✅ For smooth request pacing, use { max: 1, duration: interval } where interval = 1000 / rate.
✅ Monitor rate limit TTL with getRateLimitTtl() before adding large batches.
✅ Test your rate limiting configuration with load testing before deploying to production.

9.2 Don'ts

❌ Don't assume rate limiting prevents job addition — it limits processing, not enqueuing. Your queue may grow unboundedly if producers outpace the rate limit.
❌ Don't forget limiter.max when using dynamic rate limiting — BullMQ uses it as a fallback to determine if rate limit validation should run.
❌ Don't use groupKey in Queue options with BullMQ v3.0+ — it was removed from the open-source package.
❌ Don't set concurrency lower than your expected throughput without accounting for job duration (see Section 7.2).
❌ Don't ignore waiting job growth — a growing waiting queue behind a rate limit can signal that producers need backpressure too.

9.3 Migration Notes

BullMQ Version	Rate Limiting Support
v1.x	Initial rate limiter with `QueueScheduler` requirement
v2.x	`QueueScheduler` no longer needed; `groupKey` support in OSS
v3.x	`groupKey` removed from OSS; group rate limit → BullMQ Pro only
v5.x (current)	Stable `limiter.max`/`limiter.duration`; `maximumRateLimitDelay` option added

10. Conclusion

BullMQ's rate limiter is a lightweight, Redis-backed mechanism for throttling job processing that punches above its weight. With just two parameters — limiter.max and limiter.duration — you can protect downstream services, prevent worker overload, and ensure predictable processing behavior.

We covered three tiers of control:

Global rate limiter (OSS) — Simple, effective per-queue throttling
Dynamic rate limiting (OSS) — React to backpressure programmatically with worker.rateLimit()
Group rate limiter (Pro) — Per-group fairness for multi-tenant workloads

The right configuration depends on your workload: use small windows (max: 1) for smooth pacing, larger windows for burst-capable APIs, and group limits for multi-tenant fairness. Always monitor your rate limits in production — tools like Queue Hub make this visibility straightforward.

Now that you understand BullMQ rate limiting inside and out, it's time to put it into practice. Configure your workers, add monitoring, and iterate based on real data.

Try QueueHub for Enterprise-Grade Job Queue Monitoring

Tired of piecing together rate limit monitoring from Redis log tails and ad-hoc scripts? QueueHub provides a complete visual dashboard for BullMQ that shows you exactly what your rate limits are doing — real-time throughput charts, waiting queue trends, worker health, and rate limit hit counters — all in one place.

Try QueueHub today and get full observability into your BullMQ queues in minutes. Your downstream services will thank you.

Appendix: Quick Reference

RateLimiterOptions Interface (BullMQ v5.x)

interface RateLimiterOptions {
  max: number;      // Max jobs to process in the time period
  duration: number; // Time period in milliseconds
}

Group Rate Limit Options (BullMQ Pro)

interface GroupRateLimitOptions {
  max: number;      // Max jobs per group per time period
  duration: number; // Time period in milliseconds
}

Key API Methods

Method	Scope	Description
`worker.rateLimit(ms)`	Queue (global)	Dynamically pause the queue for `ms` milliseconds
`worker.rateLimitGroup(job, ms)`	Group (Pro)	Dynamically pause the job's group for `ms` milliseconds
`queue.getRateLimitTtl(maxJobs)`	Queue (global)	Returns TTL in ms for rate limit to allow `maxJobs` more
`queue.getGroupRateLimitTtl(groupId, maxJobs)`	Group (Pro)	Returns TTL in ms for group rate limit
`queue.removeRateLimitKey()`	Queue (global)	Resets the rate limit counter to zero