Tracing Every Tick: How to Instrument BullMQ Workers with OpenTelemetry

Your p50 queue latency is green. Your active-job count is steady. Your failure rate dashboard is flatlining at zero. Then a single email-rendering job holds a Redis connection for 14 seconds, backs up the retry queue, and cascades into a 503 storm — but your aggregate dashboards only registered "p99 blip." This is the blind spot that dashboards alone cannot fix, and it's the exact problem distributed tracing was built to solve.

When you're operating BullMQ at scale, knowing that something is wrong is table stakes. Knowing which specific job failed, why it failed, and where in its lifecycle the problem occurred — that's the difference between a 2-minute incident response and a 2-hour fire drill. In this post, we'll walk through how OpenTelemetry (OTel) gives you per-job visibility into your BullMQ workers, from enqueue to completion, and how tracing complements the dashboards and metrics you're already using.

The Two Observability Layers: Dashboards vs. Traces

Metrics and traces are complementary, not competing. Dashboards answer "what is happening?"; traces answer "why did it happen to this specific job?"

Layer	What it answers	Granularity	Tools
Metrics / Dashboards	Queue depth, throughput, error rate, p50/p99 latency	Aggregate (all jobs)	Prometheus, Grafana
Distributed Traces	Why Job #abc123 took 14 s, which sub-operation failed, what Redis command was slow	Per-job, per-operation	Jaeger, Tempo, Honeycomb

A team monitoring only dashboard aggregates is blind to individual-job pathologies. The p50 might be 200ms, but if one job in a thousand takes 30 seconds, you need to know which job and why — and that's where tracing enters the picture.

OpenTelemetry in Five Minutes (for Queue Devs)

If you're new to OpenTelemetry, here's the minimal mental model you need:

• Span: A unit of work with a name, start/end time, status, and attributes. Every operation inside your job processor can be a span.
• Trace: A tree of spans sharing a traceId, showing the full path of a job from creation to completion.
• Span kind: PRODUCER (when adding a job), CONSUMER (when a worker processes it), INTERNAL (Redis operations, completion/failure).
• Context propagation: The mechanism that carries traceId and spanId across process boundaries — in BullMQ's case, through Redis job data.
• Span links: For async/decoupled systems, links connect two spans that are causally related but don't share a parent-child execution context.

Here's the conceptual span tree for a typical BullMQ job:

[HTTP Request] ──addJob()──> PRODUCER span
                                  │
                            (context flows through Redis)
                                  │
                            CONSUMER span (worker)
                                  ├── validate-user span (child)
                                  ├── process-payment span (child)
                                  └── send-email span (child)
                                        └── Redis HSET span (auto-traced)

BullMQ's Native OpenTelemetry Story

BullMQ 5.x+ ships with first-class OpenTelemetry support through the bullmq-otel package. The ecosystem also offers a community auto-instrumentation package if you prefer the "set it and forget it" approach.

The bullmq-otel Package (Official, Built-in)

Install it as a separate package:

npm install bullmq-otel

Then create a BullMQOtel instance and pass it as the telemetry option to both your Queue and Worker:

import { Queue, Worker } from 'bullmq';
import { BullMQOtel } from 'bullmq-otel';

const telemetry = new BullMQOtel({
  tracerName: 'my-app',
  version: '1.0.0',
  enableMetrics: false,  // set true for OTel metrics
});

const queue = new Queue('email', {
  connection: { host: 'localhost', port: 6379 },
  telemetry,
});

const worker = new Worker(
  'email',
  async (job) => {
    const result = await renderAndSend(job.data);
    return result;
  },
  {
    connection: { host: 'localhost', port: 6379 },
    telemetry,
  },
);

What this gives you automatically:

• email.add (PRODUCER) when adding a job — shows the enqueue operation with a timestamp and duration
• email.getNextJob (INTERNAL) for the Redis dequeue operation — can reveal if Redis is slow to respond
• email.renderAndSend (CONSUMER) wrapping your processor function — this is the span that tells you how long your actual business logic took
• email.complete / email.fail / email.retry / email.delay (INTERNAL) for lifecycle transitions
• Rich span attributes: bullmq.queue.name, bullmq.job.id, bullmq.job.name, bullmq.job.attempts.made, bullmq.job.failed.reason

The @appsignal/opentelemetry-instrumentation-bullmq Package (Community Auto-Instrumentation)

If you prefer the auto-instrumentation approach that wires into the OpenTelemetry SDK's registerInstrumentations() pattern:

import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import { registerInstrumentations } from '@opentelemetry/instrumentation';
import { BullMQInstrumentation } from '@appsignal/opentelemetry-instrumentation-bullmq';

const provider = new NodeTracerProvider();
provider.register();

registerInstrumentations({
  instrumentations: [
    new BullMQInstrumentation({
      // Whether to link producer & consumer spans into the same trace
      useProducerSpanAsConsumerParent: true,
      // Emit individual create spans for bulk operations
      emitCreateSpansForBulk: true,
    }),
  ],
});

The useProducerSpanAsConsumerParent option is worth highlighting. When true, the consumer span becomes a direct child of the producer span in the same trace. When false (the default), producer and consumer are in separate traces, linked together via span links. We'll explore why that distinction matters next.

The Span Link Pattern: Why Async Jobs Need Links, Not Parents

Many developers new to OpenTelemetry instinctively try to make the consumer span a direct child of the producer span. This seems intuitive — after all, the consumer would not have run without the producer adding the job. But for asynchronous queues, direct parent-child relationships are technically inaccurate:

• The producer's execution has typically ended by the time the consumer runs
• Producer and consumer often run in different processes (or even different machines)
• OpenTelemetry's messaging semantic conventions expect a span link, not a parent-child edge, because the causal relationship crosses a temporal boundary

Here's how to configure @appsignal/opentelemetry-instrumentation-bullmq to use span links:

registerInstrumentations({
  instrumentations: [
    new BullMQInstrumentation({
      useProducerSpanAsConsumerParent: false,  // default — uses span links
    }),
  ],
});

With this setting, your trace backend shows two linked traces:

Trace A (API request)           Trace B (worker processing)
└── orders.add (PRODUCER)  ──── LINK ────> └── orders.process-order (CONSUMER)
                                               ├── validate-user
                                               ├── process-payment
                                               └── send-confirmation

Mode	Trace view	When to use
Links (default)	Two traces, linked	High-throughput, decoupled services; proper OTel semantics
Parent-child	One trace, full tree	Debugging a local dev environment; low-volume job systems

For production systems, stick with span links. You'll get cleaner trace data and your observability backend won't try to render impossible parent-child trees across process boundaries.

Correlating Traces with Redis Command Latency

Here's where the magic happens. When you pair BullMQ instrumentation with @opentelemetry/instrumentation-ioredis, each Redis operation BullMQ performs internally — LPUSH, BRPOP, HSET, XADD, SADD — becomes a child span under the BullMQ operation span.

This means a slow email.getNextJob span is immediately explained by a slow BRPOP sub-span. An email.complete span that takes 300ms is explained by a slow HSET in Redis. You can see at a glance whether latency lives in Redis or in your application code.

Here's what a trace with Redis instrumentation looks like in Jaeger or Tempo:

Trace: a1b2c3d4
├── orders.add (PRODUCER)              12ms
│   ├── LPUSH orders:wait             1.2ms
│   └── HSET orders:id:123            0.8ms
│
├── orders.process-order (CONSUMER)   4.2s  <-- total job time
│   ├── orders.getNextJob (INTERNAL)  320ms
│   │   └── BRPOP orders:wait         310ms  <-- slow Redis operation
│   ├── validate-user                  45ms
│   │   └── SQL SELECT                 40ms
│   ├── process-payment              3.8s   <-- root cause identified
│   │   └── HTTP POST /charge        3.7s   <-- external API slow
│   │       └── DNS lookup            1.2s
│   └── send-confirmation             25ms
│       └── SMTP send                 20ms

This is the single most valuable outcome of tracing your BullMQ workers: you can immediately distinguish between "Redis is slow" and "my code is slow" and "my external dependency is slow" — without guessing.

To set it up, configure the full Node.js SDK with both instrumentations:

import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { IORedisInstrumentation } from '@opentelemetry/instrumentation-ioredis';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({ url: 'http://localhost:4318/v1/traces' }),
  instrumentations: [
    new IORedisInstrumentation({
      // Sanitize Redis statements to avoid leaking keys in traces
      dbStatementSerializer: (cmdName, cmdArgs) =>
        `${cmdName} ${cmdArgs.slice(0, 2).join(' ')}`,
    }),
    getNodeAutoInstrumentations().filter(
      (inst) => inst.instrumentationName !== 'ioredis',
    ),
  ],
});
sdk.start();

The dbStatementSerializer option is important — it controls how much of your Redis command arguments are recorded in spans. In production, you'll want to sanitize this to avoid leaking keys or data into your traces.

Manual Instrumentation: Full Control When You Need It

Sometimes you need finer-grained spans than the automatic instrumentation provides. Perhaps you want to trace specific domain operations inside your job handler, or you're on an older BullMQ version without native OTel support, or you need to pass context across non-HTTP boundaries using propagation.inject/extract.

Enqueue Side: Inject Context Into Job Data

import { trace, context, propagation, SpanKind, SpanStatusCode } from '@opentelemetry/api';
import { Queue } from 'bullmq';

const tracer = trace.getTracer('order-service');

async function enqueueOrderJob(orderId: string) {
  return tracer.startActiveSpan(
    'orders.add',
    { kind: SpanKind.PRODUCER },
    async (span) => {
      span.setAttribute('order.id', orderId);

      // Serialize current trace context into the job payload
      const carrier: Record<string, string> = {};
      propagation.inject(context.active(), carrier);

      const job = await orderQueue.add('process-order', {
        orderId,
        otelTraceContext: carrier,
      });

      span.setAttribute('bullmq.job.id', job.id!);
      span.setStatus({ code: SpanStatusCode.OK });
      span.end();
      return job;
    },
  );
}

Worker Side: Extract Context and Create Child Spans

import { trace, context, propagation, SpanKind, SpanStatusCode } from '@opentelemetry/api';
import { Worker } from 'bullmq';

const tracer = trace.getTracer('order-worker');

const worker = new Worker(
  'orders',
  async (job) => {
    // Extract the parent trace context from the job payload
    const parentContext = job.data.otelTraceContext
      ? propagation.extract(ROOT_CONTEXT, job.data.otelTraceContext)
      : ROOT_CONTEXT;

    // Run the entire processor inside the extracted context
    return context.with(parentContext, () =>
      tracer.startActiveSpan(
        'orders.process-order',
        { kind: SpanKind.CONSUMER },
        async (span) => {
          try {
            span.setAttribute('bullmq.job.id', job.id!);
            span.setAttribute('bullmq.job.attempts.made', job.attemptsMade);

            // Child span: validate user
            const user = await tracer.startActiveSpan(
              'validate-user',
              async (subSpan) => {
                const u = await db.findUser(job.data.orderId);
                subSpan.end();
                return u;
              },
            );

            // Child span: process payment
            const payment = await tracer.startActiveSpan(
              'process-payment',
              async (subSpan) => {
                const p = await paymentGateway.charge(user);
                subSpan.setAttribute('payment.id', p.id);
                subSpan.end();
                return p;
              },
            );

            // Child span: send confirmation
            await tracer.startActiveSpan('send-confirmation', async (span) => {
              await emailService.send(user.email, 'Order confirmed');
              span.end();
            });

            span.setStatus({ code: SpanStatusCode.OK });
            return { userId: user.id, paymentId: payment.id };
          } catch (err) {
            span.recordException(err);
            span.setStatus({
              code: SpanStatusCode.ERROR,
              message: (err as Error).message,
            });
            throw err;
          } finally {
            span.end();
          }
        },
      ),
    );
  },
  { connection: { host: 'localhost', port: 6379 } },
);

Manual instrumentation gives you three things that auto-instrumentation doesn't:

1. Domain-aware sub-spans — validate-user, process-payment, send-confirmation are meaningful to your team and map to real business operations
2. Custom attributes — order IDs, user tiers, payment gateway names — whatever helps during RCA
3. Structured error recording — span.recordException(err) captures stack traces and error metadata in a format your trace backend can query

Log Correlation: Bridging Traces and Application Logs

Traces are powerful, but they don't replace logs — they enrich them. By injecting trace_id and span_id into your structured logs, you create a bridge between two observability pillars.

Here's how to do it with pino and OpenTelemetry:

import pino from 'pino';
import { trace } from '@opentelemetry/api';

const logger = pino({
  mixin() {
    const span = trace.getActiveSpan();
    if (!span) return {};
    const spanContext = span.spanContext();
    return {
      trace_id: spanContext.traceId,
      span_id: spanContext.spanId,
    };
  },
});

const worker = new Worker('email', async (job) => {
  logger.info({ jobId: job.id }, 'Starting email job');
  // ... processing ...
  logger.info({ jobId: job.id }, 'Email sent');
});

Now in your log backend (Grafana Loki, Elasticsearch, or any system that supports log correlation):

• Click a trace ID in a log line to jump directly to the full trace waterfall in Jaeger or Tempo
• Filter all logs by trace_id to see every log entry across every service for a specific job
• See the exact log lines emitted at each stage of a job's span tree

This is especially powerful during incident response: you start with a dashboard alert, drill into the trace for the failing job, then filter logs by that trace's ID to see the raw application-level events leading up to the failure.

What Distributed Tracing Catches That Dashboards Miss

Here's a concrete look at five scenarios where traces reveal what dashboards conceal:

Scenario	Dashboard Sees	Trace Sees
Slow job	p99 latency increased from 200ms to 2s	process-payment sub-span took 3.7s due to a payment gateway timeout; validate-user was fine
Stalled job	Active count drops, waiting count spikes	email.getNextJob span shows BRPOP blocking for 30s — Redis connection pool exhausted
Job fails silently	Failure rate stays flat	Worker throws on moveToCompleted due to a Redis HSET timeout — job completed in the application but BullMQ records it as failed
Thundering herd	Queue depth spikes, then recovers	All 500 jobs share the same traceId prefix — a single addBulk call that should have been rate-limited
Retry loops	Job fails, retries, fails — looks like one failure	Trace shows each retry attempt: first fails on DB timeout, second on idempotency check, third on rate limit — different root causes each time

The pattern is clear: your p50 is green, but one job is red. Dashboards average; traces individualize.

Production Considerations

Before you roll out OTel tracing to production, consider these three factors:

Sampling

Full tracing on every job is expensive and unnecessary. Use a TraceIdRatioBasedSampler to sample a percentage:

import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import { TraceIdRatioBasedSampler } from '@opentelemetry/sdk-trace-base';

const provider = new NodeTracerProvider({
  sampler: new TraceIdRatioBasedSampler(0.1),  // 10% sampling
});

• Head-based sampling (sample at enqueue time) is simple but may drop traces for rarely-occurring failures
• Tail-based sampling (keep all traces with errors) requires a collector like Grafana Tempo and gives you full coverage of failures at a fraction of the cost

For most BullMQ setups, a 10% head-based sampler with tail-based error retention is the sweet spot.

Performance Overhead

• BullMQ's OTel instrumentation adds less than 1ms per job (per the BullMQ docs)
• Span export is batched and non-blocking — the exporter queue runs on its own timer
• IORedis instrumentation adds one span per Redis command. Use dbStatementSerializer to cap payload size and avoid leaking keys

Export Configuration

• Use BatchSpanProcessor (the default in the Node.js SDK) to batch exports
• Tune scheduledDelayMillis and maxExportBatchSize for your job throughput
• OTLP/gRPC (port 4317) is more efficient than OTLP/HTTP (port 4318) for high-volume exports

OpenTelemetry + Queue Hub: Two Sides of Observability

Traces tell you what broke and why. A queue management tool lets you do something about it. They are not competing — they sit at different points in the incident response cycle.

Capability	OpenTelemetry (Tracing)	Queue Hub (Management)
Tells you	Which job is slow, and why (via sub-spans)	What's in the queue right now; retry/promote/remove
Best for	Root cause analysis, performance debugging	Daily operations: pause a queue, retry a batch, inspect job data
View	Flame graphs, span waterfalls, trace timelines	Table of jobs by status, job detail, worker live view
Action	Export to Jaeger/Tempo/Honeycomb — read-only	Write operations: retry, remove, promote, update

Here's the practical "detect → diagnose → act" workflow:

1. Detect: Your dashboard shows queue email has a 50% failure rate (metrics)
2. Diagnose: Open Jaeger or Tempo, find a failing email job, see the send-confirmation span fails with "SMTP connection refused" (tracing)
3. Act: Open Queue Hub, pause the email queue, inspect the job data for the failing batch, remove or retry the affected jobs after fixing the SMTP server

A production setup without both is like having a diagnostic scanner but no steering wheel.

Putting It All Together: Complete Example

Here's the full end-to-end setup. First, the instrumentation bootstrap that must be imported before anything else:

// instrumentation.ts — must be the first import in your application
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { IORedisInstrumentation } from '@opentelemetry/instrumentation-ioredis';
import { HttpInstrumentation } from '@opentelemetry/instrumentation-http';
import { resourceFromAttributes } from '@opentelemetry/resources';
import {
  ATTR_SERVICE_NAME,
  ATTR_DEPLOYMENT_ENVIRONMENT_NAME,
} from '@opentelemetry/semantic-conventions';

const sdk = new NodeSDK({
  resource: resourceFromAttributes({
    [ATTR_SERVICE_NAME]: 'order-processing',
    [ATTR_DEPLOYMENT_ENVIRONMENT_NAME]: process.env.NODE_ENV ?? 'development',
  }),
  traceExporter: new OTLPTraceExporter({
    url: 'http://localhost:4318/v1/traces',
  }),
  instrumentations: [
    new HttpInstrumentation(),
    new IORedisInstrumentation({
      dbStatementSerializer: (cmd, args) =>
        `${cmd} ${args.slice(0, 2).join(' ')}`,
    }),
  ],
});
sdk.start();

Then the producer that adds jobs:

// producer.ts
import { Queue } from 'bullmq';
import { BullMQOtel } from 'bullmq-otel';
import { trace, SpanKind, SpanStatusCode } from '@opentelemetry/api';

const telemetry = new BullMQOtel({ tracerName: 'order-service', version: '1.0.0' });
const orderQueue = new Queue('orders', {
  connection: { host: process.env.REDIS_HOST!, port: 6379 },
  telemetry,
});

const tracer = trace.getTracer('order-service');

export async function submitOrder(orderId: string, userId: string) {
  return tracer.startActiveSpan(
    'orders.submit',
    { kind: SpanKind.PRODUCER },
    async (span) => {
      span.setAttribute('order.id', orderId);
      span.setAttribute('user.id', userId);

      const job = await orderQueue.add('fulfill-order', { orderId, userId });

      span.setAttribute('bullmq.job.id', job.id!);
      span.setStatus({ code: SpanStatusCode.OK });
      span.end();
      return job;
    },
  );
}

And the worker that processes them:

// worker.ts
import { Worker } from 'bullmq';
import { BullMQOtel } from 'bullmq-otel';
import { trace, SpanKind, SpanStatusCode } from '@opentelemetry/api';

const telemetry = new BullMQOtel({ tracerName: 'order-worker', version: '1.0.0' });
const tracer = trace.getTracer('order-worker');

const worker = new Worker(
  'orders',
  async (job) => {
    return tracer.startActiveSpan(
      'orders.fulfill',
      { kind: SpanKind.CONSUMER },
      async (mainSpan) => {
        mainSpan.setAttribute('bullmq.job.id', job.id!);
        mainSpan.setAttribute('bullmq.job.name', job.name!);

        try {
          // Child span: check inventory
          const inventory = await tracer.startActiveSpan(
            'check-inventory',
            async (s) => {
              const r = await checkStock(job.data.orderId);
              s.end();
              return r;
            },
          );

          // Child span: charge payment
          const payment = await tracer.startActiveSpan(
            'charge-payment',
            async (s) => {
              const r = await charge(job.data.userId);
              s.end();
              return r;
            },
          );

          // Child span: send confirmation
          await tracer.startActiveSpan('send-confirmation', async (s) => {
            await sendEmail(job.data.userId);
            s.end();
          });

          mainSpan.setStatus({ code: SpanStatusCode.OK });
          return { inventory, payment };
        } catch (err) {
          mainSpan.recordException(err);
          mainSpan.setStatus({
            code: SpanStatusCode.ERROR,
            message: (err as Error).message,
          });
          throw err;
        } finally {
          mainSpan.end();
        }
      },
    );
  },
  {
    connection: { host: process.env.REDIS_HOST!, port: 6379 },
    telemetry,
    concurrency: 5,
  },
);

This three-file setup gives you end-to-end distributed tracing for every job that moves through your orders queue, with Redis command visibility, custom business-logic sub-spans, and production-ready instrumentation.

Start Tracing Your Queues Today

You now have the tools to build end-to-end traces connecting your API layer → queue → worker → Redis. That's the "see" part of observability — understanding exactly what happens inside each job's lifecycle, at the span level, without relying on dashboard aggregates that average away the details.

For the "act" part — when you need to inspect a traced job's data, retry a failed batch, pause a queue mid-incident, or simply see what's waiting in your queues right now — sign up for Queue Hub. Queue Hub gives you real-time visibility into your BullMQ and BeeQueue queues, with job detail views, worker monitoring, Redis connection management, and multi-org team support. It's the management layer that completes your observability stack: detect with metrics, diagnose with traces, act with Queue Hub.