Background Worker
Long-running job processors for asynchronous tasks
Background Workers process jobs from a queue, handle long-running tasks, and execute asynchronous operations without blocking your main application.
Perfect For
- Email sending and notifications
- Image/video processing
- Data import/export operations
- Report generation
- Webhook processing
Key Features
Process jobs from Redis/RabbitMQ queues
Automatic retries on failure
Concurrent job processing
Job priority and scheduling
Queue Integration
Background Workers work best with job queues like BullMQ (Redis), RabbitMQ, or AWS SQS for reliable job processing.
Example: Email Worker with BullMQ
worker.jsjavascript
const { Worker } = require('bullmq');const nodemailer = require('nodemailer');// Configure email transporterconst transporter = nodemailer.createTransport({host: process.env.SMTP_HOST,port: process.env.SMTP_PORT,auth: {user: process.env.SMTP_USER,pass: process.env.SMTP_PASS}});// Create workerconst emailWorker = new Worker('email-queue', async (job) => {const { to, subject, html } = job.data;console.log(`Processing email job ${job.id} to ${to}`);try {await transporter.sendMail({from: process.env.FROM_EMAIL,to,subject,html});console.log(`Email sent successfully to ${to}`);return { success: true, sentAt: new Date() };} catch (error) {console.error(`Failed to send email to ${to}:`, error);throw error; // Will trigger retry}}, {connection: {host: process.env.REDIS_HOST,port: process.env.REDIS_PORT},concurrency: 5, // Process 5 jobs concurrentlylimiter: {max: 100, // Max 100 jobsduration: 60000 // per minute}});// Event handlersemailWorker.on('completed', (job) => {console.log(`Job ${job.id} completed`);});emailWorker.on('failed', (job, err) => {console.error(`Job ${job.id} failed:`, err);});console.log('Email worker started');
Adding Jobs to Queue
Your web service can add jobs to the queue for the worker to process:
api-server.jsjavascript
const { Queue } = require('bullmq');// Create queue connectionconst emailQueue = new Queue('email-queue', {connection: {host: process.env.REDIS_HOST,port: process.env.REDIS_PORT}});// API endpoint to send emailapp.post('/api/send-email', async (req, res) => {const { to, subject, html } = req.body;// Add job to queueconst job = await emailQueue.add('send-email', {to,subject,html}, {attempts: 3, // Retry up to 3 timesbackoff: {type: 'exponential',delay: 2000 // Start with 2s delay}});res.json({message: 'Email queued',jobId: job.id});});
Example: Image Processing Worker
image-worker.jsjavascript
const { Worker } = require('bullmq');const sharp = require('sharp');const { S3Client, PutObjectCommand } = require('@aws-sdk/client-s3');const s3 = new S3Client({ region: process.env.AWS_REGION });const imageWorker = new Worker('image-queue', async (job) => {const { imageUrl, sizes } = job.data;console.log(`Processing image: ${imageUrl}`);// Download original imageconst response = await fetch(imageUrl);const buffer = Buffer.from(await response.arrayBuffer());const results = [];// Generate multiple sizesfor (const size of sizes) {const resized = await sharp(buffer).resize(size.width, size.height, { fit: 'cover' }).jpeg({ quality: 80 }).toBuffer();// Upload to S3const key = `images/${size.name}/${Date.now()}.jpg`;await s3.send(new PutObjectCommand({Bucket: process.env.S3_BUCKET,Key: key,Body: resized,ContentType: 'image/jpeg'}));results.push({size: size.name,url: `https://${process.env.S3_BUCKET}.s3.amazonaws.com/${key}`});}return { results };}, {connection: {host: process.env.REDIS_HOST,port: process.env.REDIS_PORT},concurrency: 3});console.log('Image processing worker started');
Deployment Configuration
When deploying a Background Worker:
- • Service Type: Background Worker
- • Start Command:
node worker.js - • Environment Variables: Queue connection details, API keys
- • Concurrency: Configure based on your plan's resources
Long-Running Process
Background Workers should run continuously and listen for jobs. They should not exit after processing a single job.
Monitoring & Logging
worker-with-monitoring.jsjavascript
const worker = new Worker('my-queue', processJob, {connection: redisConnection});// Monitor worker healthworker.on('completed', (job, result) => {console.log(`✓ Job ${job.id} completed in ${Date.now() - job.timestamp}ms`);});worker.on('failed', (job, error) => {console.error(`✗ Job ${job.id} failed:`, error.message);// Send alert to monitoring service});worker.on('error', (error) => {console.error('Worker error:', error);// Critical error - alert immediately});// Graceful shutdownprocess.on('SIGTERM', async () => {console.log('Shutting down worker...');await worker.close();process.exit(0);});
Pricing
Background Workers are billed based on compute time:
- • Nano Compute: ₹199 per service / 30 days - Light job processing
- • Micro: ₹0.83/hour - Standard workers
- • Small: ₹1.67/hour - Heavy processing
- • Medium: ₹3.33/hour - High-throughput workers
Choose your plan based on job complexity and concurrency needs. Image/video processing typically needs more resources.
Best Practices
- Idempotency: Design jobs to be safely retried
- Timeouts: Set reasonable job timeouts to prevent hanging
- Error Handling: Log errors and implement retry logic
- Monitoring: Track job success/failure rates
- Graceful Shutdown: Handle SIGTERM for clean shutdowns