What Are Database-Backed Queues?

Why AI Coders Need to Know This

Almost every real app needs background jobs at some point: send a welcome email after signup, resize an uploaded image, generate a PDF report, sync data to a third-party API. These tasks shouldn't block the HTTP response — you want to hand them off, return a "we're on it" to the user, and process them later.

When you ask AI to add a background job system, it almost always reaches for Redis + BullMQ (or Celery if you're on Python). That's the standard playbook. Redis is fast, BullMQ is battle-tested, and there are a thousand tutorials for it.

But here's the thing: Redis is a whole separate service. You need to run it, configure it, secure it, pay for it, and make sure it's available everywhere your app runs. For a side project processing a few hundred jobs a day, that's a lot of overhead for what is essentially "put a task in a list and process it later."

If you already have PostgreSQL, you have everything you need for a job queue. Database-backed queues use rows in a table as jobs, lock them during processing, and poll for new work. It's slower than Redis and less suitable for massive scale — but for most apps being built today, it's more than fast enough, and it's one less service to operate.

The Mental Model: A Construction Dispatch Board

Imagine a construction company dispatcher with a whiteboard. Every morning, new jobs get written on the board: "Install drywall at unit 12," "Plumbing inspection at unit 7," "Electrical rough-in at unit 3." Workers come in, grab a job off the board, write their name next to it, and go do it. When they finish, they erase the job.

Two rules keep it from becoming chaos:

1. Once a worker claims a job, no one else touches it — you don't want two plumbers showing up to the same unit.
2. If a worker goes home sick mid-job, the dispatcher marks that job as abandoned and puts it back on the board for someone else.

A database-backed queue is that dispatch board, except the whiteboard is a database table, each row is a job, workers claim rows by locking them, and the database's own transaction system enforces the "one worker per job" rule. The fancy SQL term for "skip jobs someone else already grabbed" is SKIP LOCKED.

Real Scenario: AI Hands You Redis, You Have Postgres

Add background job processing to my Node.js app. When a user
signs up, I want to send them a welcome email without blocking
the signup response. I already have PostgreSQL — use that
instead of Redis if possible.

That last sentence is key. Without it, AI defaults to Redis every time. With it, you'll get a pg-boss setup. Here's what a well-configured AI generates — and then we'll break down every piece.

What AI Generated

Step 1: Install pg-boss

npm install pg-boss

pg-boss is the most popular PostgreSQL-backed job queue for Node.js. It handles all the hard parts automatically: creating the jobs table, locking rows so two workers don't claim the same job, retrying failed jobs, expiring stale jobs, and scheduling recurring jobs.

Step 2: Initialize pg-boss

// lib/queue.js
import PgBoss from 'pg-boss';

// pg-boss connects to your existing PostgreSQL database
const boss = new PgBoss(process.env.DATABASE_URL);

// Handle errors so they don't crash the process
boss.on('error', (error) => console.error('pg-boss error:', error));

export default boss;

pg-boss takes your existing database connection string — the same one your app already uses — and creates a schema called pgboss with the tables it needs. No separate database, no new credentials, nothing to configure separately.

Step 3: Start the Queue and Register a Worker

// workers/emailWorker.js
import boss from '../lib/queue.js';
import { sendWelcomeEmail } from '../lib/email.js';

async function startEmailWorker() {
  await boss.start();

  // Register a handler for the 'send-welcome-email' queue
  await boss.work('send-welcome-email', async (jobs) => {
    for (const job of jobs) {
      const { userId, email, name } = job.data;

      try {
        await sendWelcomeEmail({ email, name });
        console.log(`Welcome email sent to ${email}`);
        // pg-boss automatically marks the job as complete when the
        // handler returns without throwing
      } catch (error) {
        // Throwing causes pg-boss to mark the job as failed
        // and retry it based on your retry configuration
        throw error;
      }
    }
  });

  console.log('Email worker started');
}

startEmailWorker();

The worker registers a handler for a named queue. When jobs arrive in that queue, pg-boss calls the handler with a batch of jobs. If the handler completes without throwing, the job is marked done. If it throws, the job is marked failed and retried.

Step 4: Enqueue Jobs from Your API

// routes/auth.js
import boss from '../lib/queue.js';

app.post('/api/signup', async (req, res) => {
  const { email, name, password } = req.body;

  // Create the user in the database
  const user = await db.users.create({ email, name, password });

  // Enqueue the welcome email job — this is near-instant
  // The actual email is sent by the worker in the background
  await boss.send('send-welcome-email', {
    userId: user.id,
    email: user.email,
    name: user.name
  });

  // Respond immediately — don't wait for the email to send
  res.status(201).json({
    status: 'success',
    message: 'Account created! Check your email.',
    data: { userId: user.id }
  });
});

The key insight is boss.send() just inserts a row into the jobs table — it's fast. The actual email sending happens in the background worker. Your API responds in milliseconds instead of waiting for the email provider to accept the message.

Step 5: Configure Retries and Options

// Send with retry options
await boss.send(
  'send-welcome-email',
  { userId: user.id, email: user.email, name: user.name },
  {
    retryLimit: 3,        // Retry up to 3 times on failure
    retryDelay: 30,       // Wait 30 seconds between retries
    expireInHours: 24,    // Abandon if not completed within 24 hours
    priority: 0           // Higher numbers = higher priority
  }
);

// Schedule a recurring job (like a daily digest email)
await boss.schedule(
  'daily-digest',
  '0 9 * * *',          // Cron: every day at 9am
  {},                    // No specific data for recurring jobs
  { tz: 'America/New_York' }
);

Understanding SKIP LOCKED: The Secret Sauce

The hardest problem with any job queue is: how do you prevent two workers from processing the same job? With Redis and BullMQ, Redis itself handles this with atomic operations. With a database queue, you need a different mechanism.

Naive approach — don't do this:

-- ❌ RACE CONDITION — two workers can both select the same row
SELECT * FROM jobs WHERE status = 'pending' LIMIT 1;
UPDATE jobs SET status = 'processing' WHERE id = $1;

Between the SELECT and the UPDATE, another worker can read the same row. You now have two workers processing the same job. Classic race condition.

The right approach using SKIP LOCKED:

-- ✅ SAFE — each worker atomically locks and claims one job
-- No other worker can see rows that are locked
BEGIN;

SELECT * FROM jobs
WHERE status = 'pending'
ORDER BY created_at ASC
LIMIT 1
FOR UPDATE SKIP LOCKED;   -- Lock the row; skip rows already locked

UPDATE jobs SET status = 'processing' WHERE id = $1;

COMMIT;

Here's what FOR UPDATE SKIP LOCKED does:

• FOR UPDATE — lock the selected row so no other transaction can modify it
• SKIP LOCKED — if a row is already locked by another worker, skip it entirely instead of waiting

Think of it like a supermarket checkout line where instead of waiting for a slow cashier, customers are magically routed to the next open register. Each worker instantly gets an available job without blocking on jobs that are already being processed. pg-boss runs this exact SQL pattern internally — you don't have to write it yourself, but understanding it helps you debug when things go wrong.

The DIY Polling Pattern

If you don't want to add pg-boss as a dependency, you can build a basic queue yourself. It's a useful exercise to understand how database queues work under the hood.

Create the Jobs Table

CREATE TABLE jobs (
  id          BIGSERIAL PRIMARY KEY,
  queue       TEXT NOT NULL,
  payload     JSONB NOT NULL DEFAULT '{}',
  status      TEXT NOT NULL DEFAULT 'pending',  -- pending, processing, done, failed
  attempts    INT NOT NULL DEFAULT 0,
  max_attempts INT NOT NULL DEFAULT 3,
  created_at  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  run_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),  -- for delayed jobs
  locked_at   TIMESTAMPTZ,
  locked_by   TEXT,   -- worker ID for debugging
  done_at     TIMESTAMPTZ,
  error       TEXT
);

-- Index for fast polling: only look at pending/failed, only in the right queue
CREATE INDEX idx_jobs_queue_status ON jobs (queue, status, run_at)
  WHERE status IN ('pending', 'failed');

The Worker Poll Loop

// Simple polling worker — checks every 5 seconds
const WORKER_ID = `worker-${process.pid}-${Date.now()}`;
const POLL_INTERVAL_MS = 5000;

async function pollAndProcess(queue, handler) {
  while (true) {
    const job = await claimNextJob(queue);

    if (job) {
      await processJob(job, handler);
    } else {
      // No jobs available — wait before polling again
      await sleep(POLL_INTERVAL_MS);
    }
  }
}

async function claimNextJob(queue) {
  const result = await db.query(`
    UPDATE jobs
    SET status = 'processing',
        locked_at = NOW(),
        locked_by = $1,
        attempts = attempts + 1
    WHERE id = (
      SELECT id FROM jobs
      WHERE queue = $2
        AND status = 'pending'
        AND run_at <= NOW()
      ORDER BY run_at ASC
      LIMIT 1
      FOR UPDATE SKIP LOCKED
    )
    RETURNING *
  `, [WORKER_ID, queue]);

  return result.rows[0] || null;
}

async function processJob(job, handler) {
  try {
    await handler(job.payload);

    await db.query(
      `UPDATE jobs SET status = 'done', done_at = NOW() WHERE id = $1`,
      [job.id]
    );
  } catch (error) {
    const newStatus = job.attempts >= job.max_attempts ? 'failed' : 'pending';
    await db.query(
      `UPDATE jobs SET status = $1, error = $2, locked_at = NULL, locked_by = NULL WHERE id = $3`,
      [newStatus, error.message, job.id]
    );
  }
}

const sleep = (ms) => new Promise(resolve => setTimeout(resolve, ms));

This DIY approach is fine for low-volume use cases. The pg-boss library is basically a much more robust version of this same pattern, with battle-tested handling for edge cases you haven't thought of yet.

Database Queue vs. Redis Queue: When to Use Which

This isn't about which one is "better" — it's about which one is right for your situation right now.

Use a Database Queue When

• You already have PostgreSQL and don't want to operate another service. One database to manage is better than two services.
• Your job volume is moderate — hundreds or low thousands of jobs per minute. Database queues handle this easily.
• You need transactional job enqueuing — creating a user and enqueuing the welcome email in the same database transaction means either both succeed or neither does. You can't do this with Redis.
• You're on a constrained budget — managed PostgreSQL is cheaper than managed PostgreSQL plus managed Redis.
• You're deploying to platforms that don't offer Redis easily (some PaaS environments, edge runtimes).
• Job durability matters more than speed — PostgreSQL writes are durable by default; Redis persistence is optional and configurable.

Use Redis (BullMQ) When

• You need high throughput — tens of thousands of jobs per minute or more. Redis is in-memory and orders of magnitude faster for queue operations.
• You need real-time job processing — Redis pub/sub means workers are notified instantly when a job arrives. Database queues poll on an interval, so there's always a small delay.
• You need advanced queue features — rate limiting, flow control, delayed jobs with millisecond precision, job progress tracking, and priority queues are all first-class features in BullMQ.
• You're already using Redis for caching or sessions. Adding BullMQ to an existing Redis instance costs nothing extra.
• You're building something that scales to large job volumes and want the most commonly supported ecosystem.

Pros and Cons

Pros of Database-Backed Queues

• No extra infrastructure — one database to operate, back up, and monitor instead of two services
• Transactional enqueuing — enqueue a job inside a database transaction; if the transaction rolls back, the job is never enqueued
• Familiar tooling — query your job table directly with SQL; use your existing database GUI to inspect jobs
• ACID guarantees — jobs are durable the moment they're committed; no risk of data loss if the process crashes
• Simpler deployment — no Redis to configure, no connection string to manage, no Redis AUTH tokens to rotate
• Easy debugging — SELECT * FROM pgboss.job WHERE name = 'send-welcome-email' AND state = 'failed' is all it takes to see what went wrong

Cons of Database-Backed Queues

• Polling overhead — workers check the database on an interval even when there are no jobs; this adds read load to your database
• Lower throughput — PostgreSQL isn't optimized for high-frequency queue operations; at tens of thousands of jobs per minute, you'll feel the difference
• Notification latency — there's always a delay between job creation and processing equal to your poll interval (typically 1–10 seconds)
• Table bloat — completed jobs accumulate as rows unless you regularly clean them up; pg-boss handles this automatically, DIY solutions need maintenance
• Adds load to your primary database — high job throughput means more queries hitting the same Postgres instance your app depends on

A Note on Quirrel

Quirrel is another database-backed job queue worth knowing about — it's designed specifically for serverless environments where you can't run a long-lived polling worker. Instead of a worker process that polls forever, Quirrel uses HTTP callbacks: when a job is ready to run, Quirrel makes an HTTP request to a URL in your app.

// With Quirrel (serverless-friendly)
import { Queue } from 'quirrel/next';  // or /express, /fastify, etc.

// Define a queue as a Next.js API route
export default Queue('api/queues/email', async (job) => {
  await sendWelcomeEmail(job);
});

// Enqueue from anywhere
import EmailQueue from './queues/email';

await EmailQueue.enqueue(
  { userId: user.id, email: user.email },
  { delay: '5min' }  // Send 5 minutes after signup
);

Quirrel is a good fit if you're on Vercel, Netlify, or another serverless platform where you can't run a persistent Node.js worker process. For traditional servers and containers, pg-boss is generally the better choice.

What AI Gets Wrong About Database Queues

1. Reaching for Redis Without Asking

The default AI response to "add background jobs" is Redis + BullMQ. This is correct in many cases but not all. AI rarely asks whether you already have a database that could serve as a queue or whether the job volume justifies the extra service. If you don't specify, you'll get Redis. If you want to evaluate database queues, you have to ask explicitly — like the prompt above.

2. Missing the SKIP LOCKED Clause

When AI writes a DIY polling queue, it frequently forgets SKIP LOCKED. You'll see something like this:

-- ❌ RACE CONDITION — two workers will claim the same job
SELECT id, payload FROM jobs
WHERE status = 'pending'
LIMIT 1;

UPDATE jobs SET status = 'processing' WHERE id = $1;

This looks fine and works fine with a single worker. The moment you add a second worker for scale or redundancy, you get duplicate processing. Always make sure the query uses FOR UPDATE SKIP LOCKED inside a single atomic operation. Ask AI to verify this pattern explicitly: "Does this query handle concurrent workers safely? Show me the SKIP LOCKED SQL."

3. No Cleanup for Completed Jobs

AI generates the happy path but often forgets that your jobs table will grow forever. Completed and failed jobs accumulate as dead rows, bloating the table and slowing queries. pg-boss handles this automatically. DIY queues need a cleanup job:

// Run this periodically — e.g., a scheduled job or a cron
async function cleanupOldJobs() {
  await db.query(`
    DELETE FROM jobs
    WHERE status IN ('done', 'failed')
      AND created_at < NOW() - INTERVAL '7 days'
  `);
}

A good rule: keep failed jobs for 7 days (so you can debug them), keep successful jobs for 1 day (for audit purposes), and delete everything older than that.

4. Polling Without Backoff

A naive polling loop runs at the same interval regardless of job volume:

// ❌ Hammers the database even when there's nothing to do
while (true) {
  const job = await claimNextJob();
  if (job) await processJob(job);
  await sleep(1000);  // Always polls every second
}

Under low load this wastes database resources. The better pattern uses exponential backoff when there are no jobs:

// ✅ Backs off when idle, responds quickly when busy
let pollInterval = 1000;  // Start at 1 second
const MAX_INTERVAL = 30000;  // Cap at 30 seconds

while (true) {
  const job = await claimNextJob();

  if (job) {
    await processJob(job);
    pollInterval = 1000;  // Reset to fast polling when jobs exist
  } else {
    await sleep(pollInterval);
    pollInterval = Math.min(pollInterval * 2, MAX_INTERVAL);  // Back off
  }
}

How to Debug Database Queues with AI Tools

In Cursor

When jobs aren't being processed, highlight your worker code and ask: "This polling worker isn't picking up jobs — check whether the SKIP LOCKED query is correct, whether the worker is actually connecting to the right queue name, and whether the job status transitions are correct." Cursor can trace the entire flow from enqueue to process and spot where jobs are getting stuck.

For pg-boss specifically: "My pg-boss worker is connected but jobs in the 'send-welcome-email' queue stay in state 'created' — what's wrong with this setup?" Knowing the pg-boss state machine (created → active → completed/failed) helps Cursor diagnose the issue.

In Windsurf

Use Cascade for inspecting your job flow across multiple files: "Trace the job lifecycle from when boss.send() is called in auth.js to when the handler completes in emailWorker.js — check for any places the job could silently fail or never be claimed." Windsurf's cross-file analysis is excellent for following data through a distributed system like a job queue.

In Claude Code

Paste your schema and worker code together and ask: "Review this DIY database queue for race conditions — specifically, is the SKIP LOCKED pattern implemented correctly, and are there any scenarios where two workers could process the same job?" Claude Code is particularly good at spotting concurrency bugs that look fine in single-threaded review but fail under load.

Direct Database Debugging

The best thing about database queues is that you can inspect them directly with SQL. These queries cover 90% of debugging scenarios:

-- See all pending jobs
SELECT id, name, data, createdon, startedon
FROM pgboss.job
WHERE state = 'created'
ORDER BY createdon ASC;

-- See failed jobs with error details
SELECT id, name, data, output, createdon, completedon
FROM pgboss.job
WHERE state = 'failed'
ORDER BY completedon DESC
LIMIT 20;

-- Count jobs by state
SELECT state, COUNT(*) FROM pgboss.job GROUP BY state;

-- Check if any jobs are stuck in 'active' (might mean a worker crashed)
SELECT id, name, startedon, NOW() - startedon AS running_for
FROM pgboss.job
WHERE state = 'active'
ORDER BY running_for DESC;

Stuck jobs in 'active' state usually mean a worker crashed mid-job without marking it complete. pg-boss has a built-in expiration mechanism to reclaim these — check your expireInHours configuration if jobs are stuck.

Bonus: Transactional Job Enqueuing

This is the superpower that Redis queues can't match. Because your jobs live in the same database as your application data, you can enqueue a job inside the same database transaction as the data change it depends on:

// Enqueue the job only if the user was successfully created
// If the INSERT fails (duplicate email, constraint violation),
// the job is never enqueued — no orphan jobs, no email to a user who
// doesn't exist in your database

await db.transaction(async (trx) => {
  // Create the user
  const user = await trx('users').insert({
    email: req.body.email,
    name: req.body.name,
    password_hash: await bcrypt.hash(req.body.password, 12)
  }).returning('*');

  // Enqueue the job — same transaction
  await boss.sendWithConnection(trx.client, 'send-welcome-email', {
    userId: user[0].id,
    email: user[0].email,
    name: user[0].name
  });
  // If anything above fails and the transaction rolls back,
  // the job row is also rolled back. No email sent to a ghost user.
});

pg-boss exposes sendWithConnection() specifically for this pattern. With Redis, you'd have to handle this coordination manually — create the user, check if it succeeded, then enqueue the job — with a window where the user could be created but the job never enqueued.

What to Learn Next

Frequently Asked Questions