Laravel Queue Configuration Guide

Complete Laravel Queue Setup and Management

Laravel queues allow you to defer time-consuming tasks, improving your application's response time. This comprehensive guide covers queue configuration, job creation, worker management, and monitoring for production environments.

Time Required: 30-45 minutes
Difficulty: Intermediate
Prerequisites: Laravel application, Redis or database access, server access

Queue System Overview
Queue Configuration
Creating and Dispatching Jobs
Worker Management
Supervisor Setup
Monitoring and Debugging
Production Best Practices
Troubleshooting

Queue System Overview

Laravel's queue system allows you to defer time-consuming tasks to background processes:

Queue Benefits

Improved Response Time: Offload slow tasks from web requests
Better User Experience: Instant feedback while processing continues
Scalability: Handle more concurrent requests
Reliability: Retry failed jobs automatically
Resource Management: Control CPU and memory usage

Queue Drivers Comparison

Driver	Performance	Persistence	Best For	Setup Complexity
Redis	Excellent	In-memory + persistence	High-traffic applications	Medium
Database	Good	Persistent	Small to medium apps	Easy
SQS	Good	AWS managed	AWS-based applications	Medium
Sync	N/A	No queuing	Development/testing	None

Queue Configuration

Step 1: Environment Configuration

Configure your queue driver in .env:

# Redis Queue Configuration (Recommended)
QUEUE_CONNECTION=redis
REDIS_HOST=127.0.0.1
REDIS_PASSWORD=null
REDIS_PORT=6379
REDIS_DB=0

# Database Queue Configuration (Alternative)
# QUEUE_CONNECTION=database

# SQS Configuration (AWS)
# QUEUE_CONNECTION=sqs
# AWS_ACCESS_KEY_ID=your-key
# AWS_SECRET_ACCESS_KEY=your-secret
# AWS_DEFAULT_REGION=us-east-1
# SQS_QUEUE=https://sqs.us-east-1.amazonaws.com/123456789012/queue-name

Step 2: Database Queue Setup (if using database driver)

# Generate queue table migration
php artisan queue:table

# Generate failed jobs table migration
php artisan queue:failed-table

# Run migrations
php artisan migrate

Step 3: Redis Configuration

Install Redis PHP extension and configure Laravel:

# Install Redis (Ubuntu/Debian)
sudo apt update
sudo apt install redis-server php-redis

# Install Redis (CentOS/RHEL)
sudo yum install redis php-redis

# Start and enable Redis
sudo systemctl start redis-server
sudo systemctl enable redis-server

# Test Redis connection
redis-cli ping

Configure Redis in config/database.php:

'redis' => [
    'client' => env('REDIS_CLIENT', 'predis'),
    'options' => [
        'cluster' => env('REDIS_CLUSTER', 'redis'),
        'prefix' => env('REDIS_PREFIX', Str::slug(env('APP_NAME', 'laravel'), '_').'_database_'),
    ],
    'default' => [
        'url' => env('REDIS_URL'),
        'host' => env('REDIS_HOST', '127.0.0.1'),
        'password' => env('REDIS_PASSWORD', null),
        'port' => env('REDIS_PORT', '6379'),
        'database' => env('REDIS_DB', '0'),
    ],
    'cache' => [
        'url' => env('REDIS_URL'),
        'host' => env('REDIS_HOST', '127.0.0.1'),
        'password' => env('REDIS_PASSWORD', null),
        'port' => env('REDIS_PORT', '6379'),
        'database' => env('REDIS_CACHE_DB', '1'),
    ],
    'queue' => [
        'url' => env('REDIS_URL'),
        'host' => env('REDIS_HOST', '127.0.0.1'),
        'password' => env('REDIS_PASSWORD', null),
        'port' => env('REDIS_PORT', '6379'),
        'database' => env('REDIS_QUEUE_DB', '2'),
    ],
]

Step 4: Queue Configuration

Configure queues in config/queue.php:

'connections' => [
    'redis' => [
        'driver' => 'redis',
        'connection' => 'queue',
        'queue' => env('REDIS_QUEUE', 'default'),
        'retry_after' => 60,
        'block_for' => null,
        'after_commit' => false,
    ],
    
    'high-priority' => [
        'driver' => 'redis',
        'connection' => 'queue',
        'queue' => 'high-priority',
        'retry_after' => 60,
        'block_for' => null,
        'after_commit' => false,
    ],
    
    'emails' => [
        'driver' => 'redis',
        'connection' => 'queue',
        'queue' => 'emails',
        'retry_after' => 300,
        'block_for' => null,
        'after_commit' => false,
    ],
]

Creating and Dispatching Jobs

Step 1: Create a Job Class

# Generate a job class
php artisan make:job SendWelcomeEmail

Example job class (app/Jobs/SendWelcomeEmail.php):

<?php

namespace AppJobs;

use AppModelsUser;
use AppMailWelcomeEmail;
use IlluminateBusQueueable;
use IlluminateContractsQueueShouldQueue;
use IlluminateFoundationBusDispatchable;
use IlluminateQueueInteractsWithQueue;
use IlluminateQueueSerializesModels;
use IlluminateSupportFacadesMail;
use IlluminateSupportFacadesLog;

class SendWelcomeEmail implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public $user;
    public $tries = 3;
    public $maxExceptions = 3;
    public $timeout = 60;
    public $backoff = [10, 30, 60];

    public function __construct(User $user)
    {
        $this->user = $user;
        $this->onQueue('emails');
    }

    public function handle()
    {
        Log::info('Sending welcome email to user: ' . $this->user->email);
        
        Mail::to($this->user->email)->send(new WelcomeEmail($this->user));
        
        Log::info('Welcome email sent successfully to: ' . $this->user->email);
    }

    public function failed(Throwable $exception)
    {
        Log::error('Failed to send welcome email to: ' . $this->user->email, [
            'error' => $exception->getMessage(),
            'user_id' => $this->user->id,
        ]);
        
        // Notify administrators or take other action
    }

    public function retryUntil()
    {
        return now()->addMinutes(10);
    }
}

Step 2: Dispatch Jobs

Dispatch jobs from controllers, commands, or other classes:

# Simple dispatch
SendWelcomeEmail::dispatch($user);

# Dispatch to specific queue
SendWelcomeEmail::dispatch($user)->onQueue('emails');

# Dispatch with delay
SendWelcomeEmail::dispatch($user)->delay(now()->addMinutes(5));

# Dispatch to specific connection
SendWelcomeEmail::dispatch($user)->onConnection('redis');

# Conditional dispatch
SendWelcomeEmail::dispatchIf($user->wants_emails, $user);

# Batch dispatch
Bus::batch([
    new SendWelcomeEmail($user1),
    new SendWelcomeEmail($user2),
    new SendWelcomeEmail($user3),
])->dispatch();

Step 3: Job Middleware

Create job middleware for rate limiting or other concerns:

# Generate middleware
php artisan make:job-middleware RateLimitEmails

class RateLimitEmails
{
    public function handle($job, $next)
    {
        Redis::throttle('emails')
            ->allow(10)
            ->every(60)
            ->then(function () use ($job, $next) {
                $next($job);
            }, function () use ($job) {
                $job->release(60);
            });
    }
}

Apply middleware to jobs:

public function middleware()
{
    return [new RateLimitEmails];
}

Worker Management

Starting Queue Workers

# Start worker for default queue
php artisan queue:work

# Start worker for specific queue
php artisan queue:work --queue=emails,default

# Start worker with specific connection
php artisan queue:work redis --queue=high-priority,default

# Worker with configuration options
php artisan queue:work redis   --queue=high-priority,emails,default   --sleep=3   --tries=3   --max-time=3600   --memory=512

Worker Configuration Options

Option	Description	Default	Recommended
--sleep	Seconds to sleep when no jobs	3	1-5
--tries	Number of attempts	1	3
--timeout	Seconds before timeout	60	60-300
--memory	Memory limit in MB	128	256-512
--max-time	Max worker runtime	∞	3600

Worker Process Management

# Check queue status
php artisan queue:monitor

# Restart all workers
php artisan queue:restart

# Clear all jobs
php artisan queue:clear

# Flush failed jobs
php artisan queue:flush

# Retry failed jobs
php artisan queue:retry all

# Retry specific failed job
php artisan queue:retry 1

Supervisor Setup

Supervisor ensures your queue workers stay running in production.

Step 1: Install Supervisor

# Ubuntu/Debian
sudo apt update
sudo apt install supervisor

# CentOS/RHEL
sudo yum install supervisor

# Start and enable supervisor
sudo systemctl start supervisord
sudo systemctl enable supervisord

Step 2: Create Supervisor Configuration

Create /etc/supervisor/conf.d/laravel-worker.conf:

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/artisan queue:work redis --sleep=3 --tries=3 --max-time=3600 --memory=512
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=4
redirect_stderr=true
stdout_logfile=/var/www/html/storage/logs/worker.log
stopwaitsecs=3600

Step 3: Configure Multiple Queue Workers

For high-traffic applications, create separate workers for different queues:

# High priority queue worker
[program:laravel-high-priority]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/artisan queue:work redis --queue=high-priority --sleep=1 --tries=3
autostart=true
autorestart=true
user=www-data
numprocs=2
redirect_stderr=true
stdout_logfile=/var/www/html/storage/logs/high-priority-worker.log

# Email queue worker
[program:laravel-emails]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/artisan queue:work redis --queue=emails --sleep=3 --tries=5
autostart=true
autorestart=true
user=www-data
numprocs=2
redirect_stderr=true
stdout_logfile=/var/www/html/storage/logs/email-worker.log

# Default queue worker
[program:laravel-default]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/artisan queue:work redis --queue=default --sleep=5 --tries=3
autostart=true
autorestart=true
user=www-data
numprocs=4
redirect_stderr=true
stdout_logfile=/var/www/html/storage/logs/default-worker.log

Step 4: Manage Supervisor

# Reload supervisor configuration
sudo supervisorctl reread
sudo supervisorctl update

# Start workers
sudo supervisorctl start laravel-worker:*

# Check worker status
sudo supervisorctl status

# Stop workers
sudo supervisorctl stop laravel-worker:*

# Restart workers (after code deployment)
sudo supervisorctl restart laravel-worker:*

Monitoring and Debugging

Queue Status Monitoring

# Check queue size
php artisan queue:monitor redis:default,redis:emails --max=100

# List failed jobs
php artisan queue:failed

# Monitor queue in real-time
watch -n 1 'php artisan queue:monitor'

Redis Queue Monitoring

# Connect to Redis CLI
redis-cli

# Check queue length
LLEN queues:default
LLEN queues:emails
LLEN queues:high-priority

# View queue jobs (without removing)
LRANGE queues:default 0 -1

# Check failed jobs
LLEN queues:default:failed

# Monitor Redis in real-time
redis-cli monitor

Custom Queue Monitoring

Create a monitoring command (app/Console/Commands/QueueMonitor.php):

<?php

namespace AppConsoleCommands;

use IlluminateConsoleCommand;
use IlluminateSupportFacadesRedis;
use IlluminateSupportFacadesDB;

class QueueMonitor extends Command
{
    protected $signature = 'queue:status';
    protected $description = 'Display queue status information';

    public function handle()
    {
        $queues = ['high-priority', 'emails', 'default'];
        
        $this->info('Queue Status Report');
        $this->info('==================');
        
        foreach ($queues as $queue) {
            $size = Redis::llen("queues:$queue");
            $this->line("Queue '$queue': $size jobs pending");
        }
        
        // Failed jobs count
        $failedJobs = DB::table('failed_jobs')->count();
        $this->line("Failed jobs: $failedJobs");
        
        // Recent job statistics
        $recentJobs = DB::table('jobs')
            ->where('created_at', '>', now()->subHour())
            ->count();
        $this->line("Jobs created in last hour: $recentJobs");
    }
}

Queue Dashboard

Install Laravel Horizon for advanced monitoring:

# Install Horizon
composer require laravel/horizon

# Publish Horizon assets
php artisan horizon:install

# Configure Horizon in config/horizon.php
php artisan vendor:publish --provider="LaravelHorizonHorizonServiceProvider"

# Start Horizon
php artisan horizon

Production Best Practices

Performance Optimization

Queue Prioritization: Use separate queues for different priority levels
Job Batching: Group related jobs to reduce overhead
Memory Management: Set appropriate memory limits for workers
Connection Pooling: Use persistent connections for Redis

Error Handling and Logging

// Job with comprehensive error handling
class ProcessOrder implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public $tries = 3;
    public $maxExceptions = 2;
    public $backoff = [60, 300, 900]; // 1min, 5min, 15min

    public function handle()
    {
        try {
            // Process order logic
            $this->processOrderLogic();
            
        } catch (PaymentException $e) {
            // Don't retry payment failures
            $this->fail($e);
            
        } catch (TemporaryException $e) {
            // Retry temporary failures
            $this->release(60);
            
        } catch (Exception $e) {
            // Log and re-throw for retry
            Log::error('Order processing failed', [
                'order_id' => $this->order->id,
                'error' => $e->getMessage(),
                'trace' => $e->getTraceAsString(),
            ]);
            throw $e;
        }
    }

    public function failed(Throwable $exception)
    {
        Log::critical('Order processing permanently failed', [
            'order_id' => $this->order->id,
            'error' => $exception->getMessage(),
        ]);
        
        // Notify administrators
        // Mark order as failed
        // Send customer notification
    }
}

Deployment Strategy

#!/bin/bash
# deployment script with queue handling

echo "Starting deployment..."

# Put application in maintenance mode
php artisan down

# Pull latest code
git pull origin main

# Update dependencies
composer install --no-dev --optimize-autoloader

# Clear caches
php artisan config:cache
php artisan route:cache
php artisan view:cache

# Run migrations
php artisan migrate --force

# Restart queue workers
php artisan queue:restart

# Wait for workers to restart
sleep 10

# Bring application back online
php artisan up

echo "Deployment completed!"

Troubleshooting Common Issues

Issue 1: Jobs Not Processing

Symptoms: Jobs stuck in queue, workers not processing

Diagnosis:

# Check worker processes
ps aux | grep "queue:work"

# Check supervisor status
sudo supervisorctl status

# Check Redis connection
redis-cli ping

# Check job payload
php artisan tinker
>>> DB::table('jobs')->first();

Solutions:

Restart queue workers: php artisan queue:restart
Check supervisor configuration and restart
Verify Redis/database connectivity
Check application logs for errors

Issue 2: Memory Leaks

Symptoms: Workers consuming increasing memory