This final episode wraps up the journey of taking the expense tracker to production: by the end, you will have a complete operations runbook, an incident checklist you can use on your very next on-call shift, and a clear upgrade path. Following episode 19 on scaling with more workers and a read replica, this episode pulls the whole series together into a sustainable operations routine.

What an Operations Runbook Is and What It Contains

A runbook is a living document that records every operations task: who does what, in which order, when deploying, when something breaks, and when you need to scale. A good runbook has four parts: the standard deploy procedure, how monitoring and alerting work, the backup and restore procedure, and an incident checklist by severity. Keep the runbook in the repo and review it every quarter, or after every major incident.

Incident Checklist by Severity

When an incident happens, the first job is not to fix code — it is to determine the severity. A P1 incident means the site is completely down: check the /health endpoint, see whether Gunicorn is still running, check the PostgreSQL connection, and if a deploy just finished, roll back immediately instead of debugging on production. A P2 incident means the system is slow: look at the slow query log, check the Redis hit rate, and see whether the Celery queue is backed up. A P3 incident is an isolated error: read the logs in Sentry and work out whether the error repeats in a pattern. Record every incident with its time, symptoms, cause, and fix, so next time you only need to open the record.

#!/bin/bash
# scripts/incident-triage.sh — chạy khi site có dấu hiệu bất thường
set -e
curl -fsS https://example.com/health/ || echo "ALERT: health endpoint khong phan hoi"
pg_isready -h db.example.com -p 5432 || echo "ALERT: PostgreSQL khong san sang"
redis-cli -h cache.example.com ping || echo "ALERT: Redis khong phan hoi"
systemctl is-active gunicorn celery || echo "ALERT: service dung"

What’s Next on the Upgrade Path

Once the runbook is stable, the three most worthwhile investments are Kubernetes, observability, and deeper automation. Kubernetes replaces manual deploys when you have many services, need pods to recover on their own, and want to scale with load. Observability takes the monitoring from episode 8 to the next level: distributed tracing with OpenTelemetry, structured logs, and one unified dashboard for Django, Celery, and Nginx. Finally, automate the runbook itself: turn the manual checklist into a check script, then into automated alerts. Only take each upgrade step when there is a real need for it.

# core/management/commands/ops_status.py
from django.core.management.base import BaseCommand
from django.db import connections
from django.core.cache import cache
class Command(BaseCommand):
help = "Kiem tra nhanh DB va cache truoc khi deploy"
def handle(self, *args, **options):
with connections["default"].cursor() as cursor:
cursor.execute("SELECT 1")
cursor.fetchone()
self.stdout.write(self.style.SUCCESS("PostgreSQL: OK"))
cache.set("ops_probe", "ok", timeout=10)
assert cache.get("ops_probe") == "ok"
self.stdout.write(self.style.SUCCESS("Redis cache: OK"))

Apply It to Your Project

Spend this afternoon writing your first runbook for the expense tracker: start from your current deploy procedure, write down every command you now run by hand, then add the P1, P2, and P3 checklists using the template above. A good production system is not one that never fails — it is one where every failure is handled faster than the last.

Related Posts

Frequently Asked Questions

How is a runbook different from system documentation?

A runbook focuses on what to do when deploying or when something breaks, while system documentation describes the architecture and the reasons behind design decisions. When you are on call at 2 a.m. during an incident, you need the runbook, not an architecture diagram.

When should you move to Kubernetes?

When you have many services to manage together, need pods to recover on their own when they fail, or need to scale with load in ways systemd and manual deploys cannot handle. For a single Django app, Gunicorn plus systemd is still good enough.

How short should an incident checklist be?

Short enough to read through in five minutes under stress — usually five to seven steps per severity level. Put the longer detail in the explanation below each step.

References