Introduction

This episode brings Celery into the expense-tracking app: heavy work such as sending monthly reports runs in the background with workers, repeats on schedule with Beat, and stays visible through Flower. By the end, you will have a queue that runs reliably under systemd, with failing tasks retrying a limited number of times before landing in a dead-letter queue. Building on episode 9, we reuse Redis — previously set up as a cache — as the broker.

1. Installing and Configuring Celery

Install Celery with Redis support, then create celery.py next to settings.py so the app discovers tasks automatically.

# config/celery.py
import os
from celery import Celery
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.prod")
app = Celery("expense")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()
# config/settings/prod.py
CELERY_BROKER_URL = env("REDIS_URL")
CELERY_RESULT_BACKEND = env("REDIS_URL")
CELERY_TASK_TIME_LIMIT = 300
CELERY_TASK_SOFT_TIME_LIMIT = 240
CELERY_TASK_ACKS_LATE = True
CELERY_WORKER_PREFETCH_MULTIPLIER = 1

Worth noting: acks_late means a task is only acknowledged after it finishes, so if a worker dies mid-task, the task is handed out again; prefetch_multiplier = 1 stops a worker from hoarding several tasks at once.

2. Writing Your First Task

The monthly-report task gathers the data and sends the email, running outside the request, so users never have to wait for it.

# expenses/tasks.py
from celery import shared_task
from django.contrib.auth import get_user_model
from django.core.mail import send_mail
from .services import monthly_summary
@shared_task(
    autoretry_for=(Exception,),
    retry_backoff=True,
    max_retries=3,
)
def send_monthly_report(user_id):
    user = get_user_model().objects.get(pk=user_id)
    body = monthly_summary(user)
    send_mail("Bao cao chi tieu thang", body,
              "no-reply@example.com", [user.email])

Pass only the primary key into the task and query the object again inside it. Tasks must be idempotent: running one twice should produce the same result, because Celery may deliver a task more than once.

3. Scheduling with Celery Beat

Beat is the system’s clock: it fires tasks on schedule and workers execute them. Declare the schedule in settings using crontab.

# config/settings/prod.py
from celery.schedules import crontab
CELERY_BEAT_SCHEDULE = {
    "send-monthly-reports": {
        "task": "expenses.tasks.send_monthly_reports",
        "schedule": crontab(day_of_month=1, hour=7, minute=0),
    },
    "remind-due-bills": {
        "task": "expenses.tasks.remind_due_bills",
        "schedule": crontab(hour=8, minute=0),
    },
}

The send_monthly_reports task loops over users and calls send_monthly_report.delay(user.id) for each one. Splitting it up means one failing email does not block the whole batch, and each report retries independently.

4. Running Workers and Beat with systemd

In production, the worker and Beat are operating-system services that restart automatically when they die. Each process gets its own systemd unit, and Beat runs as a single instance only.

# /etc/systemd/system/celery-worker.service
[Unit]
Description=Celery worker - quan ly chi tieu
After=network.target
[Service]
User=deploy
WorkingDirectory=/srv/expense
EnvironmentFile=/srv/expense/.env
ExecStart=/srv/expense/venv/bin/celery -A config worker -l info --concurrency=2
Restart=always
[Install]
WantedBy=multi-user.target
sudo systemctl enable --now celery-worker celery-beat
sudo systemctl status celery-worker

Two Beat instances would fire the same tasks twice, so with multiple servers only one machine runs Beat, while workers can be replicated freely. Check worker logs with journalctl -u celery-worker, following the centralized-logging approach from episode 7.

5. Monitoring with Flower

Flower is a web interface for watching workers, running tasks, failed tasks, and processing throughput.

/srv/expense/venv/bin/celery -A config flower --port=5555

Never expose Flower to the Internet: put it behind Nginx with basic authentication, the same way episode 3 locked down internal endpoints. A quick look at Flower each morning tells you how many reports succeeded overnight and which tasks were unusually slow.

6. Bounded Retries and Dead-Letter

Unlimited retries clog the queue, because a failing task keeps coming back and hogging workers. The rule here: retry at most three times with increasing backoff, and after the third failure move the task to a dead-letter queue for manual handling.

# expenses/tasks.py
from celery import Task
class DeadLetterTask(Task):
    def on_failure(self, exc, task_id, args, kwargs, einfo):
        self.app.send_task(
            "expenses.tasks.handle_dead_letter",
            args=[self.name, args, kwargs, str(exc)],
            queue="dead_letter",
        )
@shared_task(base=DeadLetterTask, bind=True, max_retries=3)
def remind_due_bills(self):
    try:
        send_due_reminders()
    except Exception as exc:
        raise self.retry(exc=exc, countdown=60 * 2 ** self.request.retries)

The handle_dead_letter task only writes a log entry and sends an alert through Sentry, as set up in episode 7. The dead-letter queue is the single place to check for stuck work, instead of digging through logs.

Putting It Into Practice

Start with one genuinely low-risk task: send the monthly report email with Beat on the 1st of the month. Once you trust it, gradually move other heavy work into the queue: payment-due reminders, expense summaries, CSV exports. Keep this rule: a request should only do fast work — anything slower than a few hundred milliseconds deserves to become a task.

Related Posts

FAQ

Is it OK to share one Redis instance for both cache and broker?

Yes, but use different database indexes: cache on db 1, broker on db 0. If the task volume grows large, give the broker its own dedicated Redis instance.

How much worker concurrency should I set?

Start with the number of CPU cores and increase gradually based on what Flower shows. I/O-heavy tasks such as sending email can handle high concurrency, but CPU-heavy tasks should not exceed the core count.

What happens when a task runs too long?

task_time_limit hard-kills a task after 300 seconds, while soft_time_limit gives it a chance to clean up 60 seconds earlier. If a task keeps hitting the limit, split it up or optimize it — do not just raise the limit.

Sources