Background Worker - Replov Docs
Skip to main content

Background Worker

Long-running job processors for asynchronous tasks

Background Workers process jobs from a queue, handle long-running tasks, and execute asynchronous operations without blocking your main application.

Perfect For

  • Email sending and notifications
  • Image/video processing
  • Data import/export operations
  • Report generation
  • Webhook processing

Key Features

Process jobs from Redis/RabbitMQ queues
Automatic retries on failure
Concurrent job processing
Job priority and scheduling
Queue Integration
Background Workers work best with job queues like BullMQ (Redis), RabbitMQ, or AWS SQS for reliable job processing.

Example: Email Worker with BullMQ

worker.jsjavascript
const { Worker } = require('bullmq');
const nodemailer = require('nodemailer');
// Configure email transporter
const transporter = nodemailer.createTransport({
host: process.env.SMTP_HOST,
port: process.env.SMTP_PORT,
auth: {
user: process.env.SMTP_USER,
pass: process.env.SMTP_PASS
}
});
// Create worker
const emailWorker = new Worker('email-queue', async (job) => {
const { to, subject, html } = job.data;
console.log(`Processing email job ${job.id} to ${to}`);
try {
await transporter.sendMail({
from: process.env.FROM_EMAIL,
to,
subject,
html
});
console.log(`Email sent successfully to ${to}`);
return { success: true, sentAt: new Date() };
} catch (error) {
console.error(`Failed to send email to ${to}:`, error);
throw error; // Will trigger retry
}
}, {
connection: {
host: process.env.REDIS_HOST,
port: process.env.REDIS_PORT
},
concurrency: 5, // Process 5 jobs concurrently
limiter: {
max: 100, // Max 100 jobs
duration: 60000 // per minute
}
});
// Event handlers
emailWorker.on('completed', (job) => {
console.log(`Job ${job.id} completed`);
});
emailWorker.on('failed', (job, err) => {
console.error(`Job ${job.id} failed:`, err);
});
console.log('Email worker started');

Adding Jobs to Queue

Your web service can add jobs to the queue for the worker to process:

api-server.jsjavascript
const { Queue } = require('bullmq');
// Create queue connection
const emailQueue = new Queue('email-queue', {
connection: {
host: process.env.REDIS_HOST,
port: process.env.REDIS_PORT
}
});
// API endpoint to send email
app.post('/api/send-email', async (req, res) => {
const { to, subject, html } = req.body;
// Add job to queue
const job = await emailQueue.add('send-email', {
to,
subject,
html
}, {
attempts: 3, // Retry up to 3 times
backoff: {
type: 'exponential',
delay: 2000 // Start with 2s delay
}
});
res.json({
message: 'Email queued',
jobId: job.id
});
});

Example: Image Processing Worker

image-worker.jsjavascript
const { Worker } = require('bullmq');
const sharp = require('sharp');
const { S3Client, PutObjectCommand } = require('@aws-sdk/client-s3');
const s3 = new S3Client({ region: process.env.AWS_REGION });
const imageWorker = new Worker('image-queue', async (job) => {
const { imageUrl, sizes } = job.data;
console.log(`Processing image: ${imageUrl}`);
// Download original image
const response = await fetch(imageUrl);
const buffer = Buffer.from(await response.arrayBuffer());
const results = [];
// Generate multiple sizes
for (const size of sizes) {
const resized = await sharp(buffer)
.resize(size.width, size.height, { fit: 'cover' })
.jpeg({ quality: 80 })
.toBuffer();
// Upload to S3
const key = `images/${size.name}/${Date.now()}.jpg`;
await s3.send(new PutObjectCommand({
Bucket: process.env.S3_BUCKET,
Key: key,
Body: resized,
ContentType: 'image/jpeg'
}));
results.push({
size: size.name,
url: `https://${process.env.S3_BUCKET}.s3.amazonaws.com/${key}`
});
}
return { results };
}, {
connection: {
host: process.env.REDIS_HOST,
port: process.env.REDIS_PORT
},
concurrency: 3
});
console.log('Image processing worker started');

Deployment Configuration

When deploying a Background Worker:

  • • Service Type: Background Worker
  • • Start Command: node worker.js
  • • Environment Variables: Queue connection details, API keys
  • • Concurrency: Configure based on your plan's resources
Long-Running Process
Background Workers should run continuously and listen for jobs. They should not exit after processing a single job.

Monitoring & Logging

worker-with-monitoring.jsjavascript
const worker = new Worker('my-queue', processJob, {
connection: redisConnection
});
// Monitor worker health
worker.on('completed', (job, result) => {
console.log(`✓ Job ${job.id} completed in ${Date.now() - job.timestamp}ms`);
});
worker.on('failed', (job, error) => {
console.error(`✗ Job ${job.id} failed:`, error.message);
// Send alert to monitoring service
});
worker.on('error', (error) => {
console.error('Worker error:', error);
// Critical error - alert immediately
});
// Graceful shutdown
process.on('SIGTERM', async () => {
console.log('Shutting down worker...');
await worker.close();
process.exit(0);
});

Pricing

Background Workers are billed based on compute time:

  • • Nano Compute: ₹199 per service / 30 days - Light job processing
  • • Micro: ₹0.83/hour - Standard workers
  • • Small: ₹1.67/hour - Heavy processing
  • • Medium: ₹3.33/hour - High-throughput workers
Choose your plan based on job complexity and concurrency needs. Image/video processing typically needs more resources.

Best Practices

  • Idempotency: Design jobs to be safely retried
  • Timeouts: Set reasonable job timeouts to prevent hanging
  • Error Handling: Log errors and implement retry logic
  • Monitoring: Track job success/failure rates
  • Graceful Shutdown: Handle SIGTERM for clean shutdowns