Background Jobs and Queues, Explained for Founders
A background job is a piece of work your application does after it has answered the user, not while they wait. A queue is the list where those jobs sit until a separate worker process picks them up. This keeps slow or unreliable tasks, such as sending emails, building reports, processing uploads and handling payment webhooks, out of the web request. Pages stay fast, and a failed task can be retried without the user seeing an error.
Why should slow work run outside the web request?
When someone clicks a button, the application receives a request and has to send back a response. Everything the code does before responding adds to the time the user waits. If that includes generating a large PDF or calling an outside email provider, the user sits watching a spinner. If the provider is slow or down, the whole action may fail.
Moving that work to the background separates two questions. The first is whether the user's request was accepted. The second is whether the follow-up work has finished. The application records the order, puts a "send confirmation email" job on the queue and responds straight away. A worker sends the email a moment later.
This is one practical use of asynchronous processing. For the broader trade-offs between doing work immediately and doing it later, see synchronous vs asynchronous software processing.
How does a queue hold jobs?
Think of a queue as a durable to-do list shared between parts of your system:
- Producer. The web application creates a job, a small message describing the work, such as "resize image 42" or "generate March invoice report for account 17".
- Queue. The job is stored, usually in a database table or a dedicated queue service, until a worker is free.
- Worker. A separate process takes the next job, does the work and marks it complete.
- Result. The outcome is saved somewhere the application can read, such as a status field, a file link or a notification to the user.
Because the job is stored, a deploy or a temporary crash of a worker does not lose it, provided the queue itself is durable. You can also add more workers when the queue grows, which is part of building scalable and robust architectures. Managed queue services and self-hosted options both work. The choice usually follows your hosting approach, covered in managed services vs self-hosting.
What belongs in the background and what does not?
| Work | Background? | Reason |
|---|---|---|
| Sending confirmation and notification emails | Yes | Depends on an outside provider; the user does not need to wait for delivery |
| Generating large reports or exports | Yes | Can take a long time; notify the user when the file is ready |
| Resizing images, transcoding video, scanning uploads | Yes | Heavy processing that would slow the request |
| Processing incoming payment webhooks | Yes, after a quick acknowledgement | Acknowledge receipt promptly, then do the business logic in a job |
| Syncing records with a third-party system | Yes | Outside APIs can be slow, rate-limited or unavailable |
| Scheduled clean-up and nightly summaries | Yes | No user is waiting at all |
| Checking a password at sign-in | No | The user needs the answer before continuing |
| Validating a form and showing errors | No | Feedback must be immediate |
| Checking stock or a balance before confirming a purchase | Usually no | The decision depends on the result |
| Showing a page the user just requested | No | This is the response itself; consider [caching](/blog/caching-explained-for-web-applications) if it is slow |
A useful test is to ask: does the user need the result before they can take their next step? If not, it is a candidate for the background.
What are retries, and why do they need limits?
Background work often fails for temporary reasons. An email provider times out, or a partner API is briefly unavailable. A retry means the queue puts a failed job back to be tried again, usually after a growing delay so the struggling service has time to recover.
Retries need three decisions:
- How many attempts before giving up.
- How long to wait between attempts.
- Which failures are worth retrying. A timeout is worth retrying. A job referring to a deleted record will fail every time.
Retries create a risk that work happens twice, for example a job that charged a card but crashed before recording success. Jobs should be safe to run more than once. That property is called idempotency, explained in API idempotency and duplicate transactions. The same applies to webhooks, which providers may deliver more than once. See webhooks vs API polling for how they work.
What is dead-letter handling?
When a job has used up its retries, it should not vanish and it should not loop forever. A dead-letter queue is a separate holding area for jobs that have failed for good. They wait there until a person investigates.
Good dead-letter handling means:
- The failed job keeps its original data and the last error message.
- Someone is alerted when jobs land there.
- There is a safe, documented way to replay a job after the cause is fixed, or to discard it deliberately.
Without this, failures go unnoticed. A customer never receives an invoice, and nobody knows until they complain.
What should you monitor?
A queue can look healthy while work quietly piles up. Ask for visibility into:
- Queue depth. How many jobs are waiting.
- Age of the oldest job. This often matters more than depth, because it shows how late work is running.
- Failure and retry rates by job type.
- Dead-letter arrivals, with an alert and an owner.
- Workers that have stopped. Silence should not be read as success.
These belong in your wider monitoring plan after launch.
For a worked business case, see automating customer payment reminders.
Worked example: an export feature
*Illustrative scenario.* A customer clicks "Export all orders". Without a queue, the request runs until the file is built and may time out on a large account. With a queue, the application creates an export job, shows "We are preparing your file" and responds immediately. A worker builds the file and stores it, then emails a download link. If the email provider fails, the email job retries on its own without rebuilding the file. If it still fails after the agreed attempts, it moves to the dead-letter queue and support is alerted. The customer can still download the file from their account page.
When does a queue not make sense?
- Very small, fast tasks. A queue adds moving parts: another process to deploy, monitor and pay for.
- Work the user must see right away. Hiding it in the background only adds polling and confusion.
- Early prototypes. Simple scheduled tasks may be enough until real volume appears. Note the decision so it is revisited later.
What common mistakes should founders watch for?
Calling slow outside services inside the request. One slow provider then slows your whole product.
Retrying forever. Endless retries hide permanent failures and can overload a service that is already struggling.
No dead-letter process. Failed jobs disappear and nobody hears about the problem.
Jobs that are unsafe to repeat. Retries then send duplicate emails or duplicate charges.
Not telling the user what is happening. Background work needs a visible status, such as "processing", "ready" or "failed, please try again".
Monitoring depth but not age. A short queue of stuck jobs can still be hours late.
What questions should you ask your development team?
- Which tasks in our product run in the background, and which run during the request?
- Where are jobs stored, and is anything lost if a worker or server restarts?
- How many retries does each job type get, and with what delay?
- Is every job safe to run twice? How do you prevent duplicate emails or charges?
- What happens to a job that fails permanently, and who is told?
- How do we replay a failed job safely once the cause is fixed?
- What do we monitor: queue depth, oldest job age, failures? Who owns those alerts?
- How does the user see the status of work that is still in progress?
- How do workers scale when volume grows, and what does that cost to host?
Frequently asked questions
Is a background job the same as a cron job?
Not quite. A cron job, or scheduled task, runs at set times. A background job is usually triggered by an event, such as a sign-up or an upload. Many systems use the same queue and workers for both.
Do background jobs make my application faster?
They make responses faster for users, because slow work no longer holds up the page. The total work is the same. It is done somewhere else and can be spread across workers.
Do we need a separate queue service?
Not always. Smaller products often store jobs in their existing database. Dedicated queue services help when volume, throughput or reliability needs grow. Ask your team to record the choice in an architecture decision record.
How do payment webhooks fit in?
Acknowledge the webhook quickly so the payment provider does not treat it as failed. Then put the processing in a job, and make sure handling the same event twice has no extra effect.
*Planning a product that needs reliable background processing? Start a conversation →*