Bull - Job Manager for Nodejs
There are some blogs and documentation on Bull for Node.js. But. I would love to share my experience on how we replaced RabbitMQ with Bull.
My friend is working for Heath and fitness app. Backend is written in Node.js. They were using RabbitMQ for job processing and needs to handle around 500k+ jobs every day for processing users activities, sending notifications to users and much more. Some of the notifications fails to deliver and there was no proper retry mechanism to process failed notifications. Also, they were using node-schedule to schedule jobs. He approached me to overcome this challenge.
I am highly fascinated by Resque in ruby. It's a redis based job manager, which provides features like job retries, scheduling, hooks, priority jobs and much more. I understood, we need a job manager rather than just a message queue to overcome the above problem and my search ended with Bull - The fastest, most reliable, Redis-based queue for Node.
We convinced, Bull is the right solution to overcome the above problem because of its rich of features:
- Minumal CPU usage due to a polling-free design
- Robust design based on Redis
- Delayed jobs
- Schedule and repeat jobs according to a corn specification
- Rate limiter for jobs
- Retries
- Priority
- Concurrency
- Pause/resume - globally or locally
- Multiple job types per queue
- Threaded
- Automatic recovery from process crashes
Let's discuss on POC and how we implemented in production It provides a very simple mechanism to create queues and publish jobs to queue.
const Queue = require('bull');
const queue = new Queue('queue1', { redis: { port: 6379, host: '127.0.0.1' } });
We just created object of Queue class by passing the queue name and redis connection details. This made us to think, each queue establishes a separate connection to redis. No wonder, bull provides a provision to reuse the one redis connection for all queues.
Next, to submit job to the queue,
queue.add({ data: 'job data' }, { attempts: 2, removeOnComplete: true, removeOnFail: true }));
Queue's add() accepts job data as json parameter and optional configuration for that job. Here, I would like to highlight the key feature Queue provides fro jobs. Each job can have it's own configuration, event thought it uses the same queue.
Example: web have two jobs - Job1 and Job2. Job1 should be retried in case of failure but Job2 should not. That's possible, just pass relevant configurations while adding job to queue. Here are the following optional parameters configurations for jobs:
- priority: number, ranges from 1 to MAX_INT
- dealy: number; The amount of milliseconds to wait until this job can be processed.
- attempts: number; The total number of attempts to try the job until it completes
- repeat: RepeatOpts; Repeat job according to a cron specification.
backoff: number | BackoffOpts; Backoff setting for automatic retries if the job fails, default strategy:
fixed. - lifo: boolean; if true, adds the job to the right of the queue instead of the left (default false) timeout: number; The number of milliseconds after which the job should fail with a timeout error
- jobId: number | string; Override the job ID — by default, the job ID is a unique integer, but you can use this setting to override it.
- removeOnComplete: boolean | number | KeepJobs; If true, removes the job from redis when it successfully
- removeOnFail: boolean | number | KeepJobs; If true, removes the job from redis when it fails after all attempts.
- stackTraceLimit: number; Limits the amount of stack trace lines that will be recorded in the stacktrace.