Roll a Django App’s Database Credential

Change the password, the login or the database address a Django app connects with, without downtime. The one exception is moving to a different database instance, which still needs an outage (see Endpoint roll). The app holds two credentials (“slots”) in its vault, and a roll moves it from the live slot to the spare:

1. Put the new credential in the spare slot. The app isn’t using it, so nothing breaks.
2. Point DATABASE_ACTIVE_SLOT at the spare.
3. Restart every service, so each container picks the new slot when it starts.
4. Check every container is on the new slot, and the old login is idle.
5. Retire the old credential.

The live slot is never touched until the roll is confirmed, so rolling back is the same steps in the other direction.

The worked example throughout is hb-buzz staging. Adapting to another app lists what changes for the others. The design and its reasons are in the two-slot credentials RFC; this runbook is only the procedure.

hb-buzz at a glance

The commands below use the staging values. For production, swap in the right-hand column.

In both, the vault keys are DATABASE_URL_A, DATABASE_URL_B and DATABASE_ACTIVE_SLOT. DATABASE_URL stays as the fallback for builds without slot support; leave it alone. Slot A held the owner login hb_buzz_user when each vault moved to slots.

Before you start

• The app reads slots. hb-buzz does from master. Production deploys from release, so check the running build includes it: /ping must report a database field. An app without slot support ignores the slot keys, so a roll does nothing.
• The vault is in slot mode: DATABASE_URL_A and DATABASE_ACTIVE_SLOT are set. Moving a vault to slot mode is a separate one-off step, in the RFC’s “Adopting in Other Apps”.
• Tools:
  • the Cloud SQL Auth Proxy and Application Default Credentials, set up as in add_db_slot_user’s prerequisites;
  • a psql client;
  • an hb-ansible checkout with the vault password for the environment;
  • Papertrail, in the browser or through ./bin/swo.
• Access:
  • roles/cloudsql.client on the-helper-bees;
  • read on the admin password secret;
  • read and add-version on the app’s vault.

Start the proxy in its own terminal and leave it running:

cloud-sql-proxy --port 6543 the-helper-bees:us-central1:ansible-hb-psql-stage

Then, in the terminal you run the roll from (Git Bash, from the infrahive root):

export PGPASSWORD=$(gcloud secrets versions access latest --secret=stage_hb_psql_admin_user_password --project=prj-n-secrets-817e)
export PGCONN="host=127.0.0.1 port=6543 dbname=hb_buzz_db user=stage_hb_psql_admin_user"

For production:

cloud-sql-proxy --port 6544 the-helper-bees:us-central1:ansible-hb-psql-prod

export PGPASSWORD=$(gcloud secrets versions access latest --secret=prod_hb_psql_admin_user_password --project=prj-p-secrets-3c7d)
export PGCONN="host=127.0.0.1 port=6544 dbname=hb_buzz_db user=prod_hb_psql_admin_user"

Each instance has its own port, so a staging command can’t reach production even with both proxies running. Pass --port 6544 to add_db_slot_user for production.

1. Find the live slot

Show the pointer and each slot’s login and host, without printing passwords:

gcloud secrets versions access latest --secret=hb-buzz-vault --project=prj-bu1-n-vault-secrets-9d03 | ./bin/jq '
  def login: if . == null then null else (split("://")[1] | split("@") | {user: (.[0] | split(":")[0]), host: (.[1] | split("/")[0])}) end;
  {active: .DATABASE_ACTIVE_SLOT, A: (.DATABASE_URL_A | login), B: (.DATABASE_URL_B | login)}'

The slot active names is the live slot; the other is the spare. Confirm the database agrees: only the live login should have sessions.

psql "$PGCONN" -c "SELECT usename, application_name, state, backend_start FROM pg_stat_activity WHERE usename LIKE 'hb_buzz_user%' ORDER BY 1, 4;"

If the spare’s login has sessions, a container is still on it, or fell back to it. Find out why before going on: Check every container shows how.

2. Prepare the spare slot

Do one of the following. Each writes the spare slot only and leaves the pointer where it is, so the app keeps running on the live slot.

Password roll

Give the spare slot’s login a new password and write it into the spare slot. With A live and B spare:

./zig/zig build scripts -- add_db_slot_user --database hb_buzz_db --owner hb_buzz_user --user hb_buzz_user_b --admin-user stage_hb_psql_admin_user --admin-password-secret prj-n-secrets-817e/stage_hb_psql_admin_user_password --vault prj-bu1-n-vault-secrets-9d03/hb-buzz-vault --slot B --rotate-password

Read the plan and the target line it prints, then rerun with --apply. With B live, use --user hb_buzz_user_a --slot A.

The first roll onto A. While slot A still holds the owner login hb_buzz_user, the script won’t rotate it: the owner owns the tables, and hb-ansible’s backup and restore playbooks use its password. Run the slot A command without --rotate-password instead. That creates hb_buzz_user_a and writes it to slot A, and hb_buzz_user stays as it is.

User roll

Create a new login and write it into the spare slot. With A live and B spare:

./zig/zig build scripts -- add_db_slot_user --database hb_buzz_db --owner hb_buzz_user --user hb_buzz_user_b2 --admin-user stage_hb_psql_admin_user --admin-password-secret prj-n-secrets-817e/stage_hb_psql_admin_user_password --vault prj-bu1-n-vault-secrets-9d03/hb-buzz-vault --slot B

Dry run, then --apply. The login the spare slot held before (hb_buzz_user_b here) is now in no slot. Drop it in step 6.

Create logins with this script, not gcloud sql users create. The gcloud command makes the login a member of cloudsqlsuperuser, so the script refuses to use it.

Endpoint roll

The spare slot points at a different address.

Same database, new address (a new proxy endpoint, a new IP for the same instance): no write fencing is needed, because both slots reach the same data. Edit the spare slot’s URL by hand, as in step 3, changing only the host and port. Before you point at it, check that the new address reaches the right database. Use the admin login, whose password is already in PGPASSWORD, so the spare’s password never goes on a command line: psql "host=<new host> port=<new port> dbname=hb_buzz_db user=stage_hb_psql_admin_user" -c "SELECT current_database(), inet_server_addr();". The boot probe checks the spare login itself when the containers restart.

A different instance (an upgraded copy, a migration) is not zero-downtime yet. During the restart, old containers keep writing to the old instance while new ones write to the new one, and a new container whose probe fails falls back to the old instance. Writes would end up split across two databases. A PostgreSQL standby also passes the boot probe while read-only, so a passing probe doesn’t prove the target takes writes. Until the apps have read-only mode, fence writes with an outage:

1. Follow Execute a Planned Outage and scale the database services to 0.
2. Copy the database and confirm it’s writable and current.
3. Create the slot logins on the new instance. Run add_db_slot_user against a proxy for the new instance, with --print-password instead of --vault, and write the spare URL by hand.
4. Point the vault at the spare and scale the services back up.

The zero-downtime version of this, read-only mode during the copy, is the RFC’s read-only rung and hasn’t been done yet.

3. Point the vault at the spare

Add one vault version that changes only DATABASE_ACTIVE_SLOT. This is a read-modify-write with no locking, so make sure nobody else is adding vault versions while it runs.

Set NEW to the spare slot, and VERSION to the vault version that holds the spare’s new URL:

• If you just ran add_db_slot_user, use the version it printed (wrote DATABASE_URL_B to …/versions/N).
• Otherwise, use latest.

Don’t use latest straight after the script writes: it can briefly return the previous version. The flip would then write the spare’s old URL back with the pointer on it, and the new password, which the script never prints with --vault, would be lost.

NEW=B
VERSION=latest
SECRET=hb-buzz-vault
PROJECT=prj-bu1-n-vault-secrets-9d03
(
  tmp=$(mktemp -d) && trap 'rm -rf "$tmp"' EXIT \
  && gcloud secrets versions access "$VERSION" --secret="$SECRET" --project="$PROJECT" > "$tmp/old.json" \
  && ./bin/jq -e --arg s "$NEW" '(.["DATABASE_URL_" + $s] | type == "string" and length > 0) and .DATABASE_ACTIVE_SLOT != $s' "$tmp/old.json" \
  && ./bin/jq --arg s "$NEW" '.DATABASE_ACTIVE_SLOT = $s' "$tmp/old.json" > "$tmp/new.json" \
  && ./bin/jq -e -n --slurpfile a "$tmp/old.json" --slurpfile b "$tmp/new.json" '($a[0] | del(.DATABASE_ACTIVE_SLOT)) == ($b[0] | del(.DATABASE_ACTIVE_SLOT))' \
  && gcloud secrets versions add "$SECRET" --project="$PROJECT" --data-file="$tmp/new.json" \
  || { echo "VAULT UPDATE FAILED: stop here, and rerun step 1 to see whether the pointer changed" >&2; exit 1; }
) && echo "Vault updated"

The first jq refuses an empty spare slot, and a slot that’s already active. The second makes the change, and the third proves nothing else changed. A failed check stops the block before the vault is written, and the copies of the vault are deleted however the block ends. If gcloud secrets versions add itself fails, the new version may still have been stored, so check with step 1 before retrying.

Before restarting anything, rerun step 1’s first command until active names the spare. Reads of latest can briefly return the previous version, and a container restarted then would stay on the old slot.

Nothing has changed for the app yet. Running containers chose their slot when they started and keep it.

4. Restart every service

Note the time first, in UTC. The checks in step 5 count log lines from this moment:

date -u +%Y-%m-%dT%H:%M:%SZ

Redeploy the commit that’s already running, from the hb-ansible checkout. Staging runs master:

uv run ansible-playbook gc_deploy_with_gcp_secrets.yml -e "app_name=hb_buzz env=staging remote_branch=master" --vault-password-file ~/.secrets/hb-ansible/staging/ansible-vault-pass -v

Production runs release. Without use_release_branch=true, hb-ansible deploys origin/<remote_branch> whatever the environment, so a production redeploy of master would ship unreleased code:

uv run ansible-playbook gc_deploy_with_gcp_secrets.yml -e "app_name=hb_buzz env=production use_release_branch=true" --vault-password-file ~/.secrets/hb-ansible/production/ansible-vault-pass -v

hb-ansible sees the same commit and force-updates every service except Redis, the Cloud SQL proxy and Caddy, start-first. Each new container fetches the vault and picks the new slot; the old one keeps serving until the new one is healthy.

hb-ansible deploys a branch, so it can’t be pinned to a commit. Before you run it, check that the branch you deploy is still the commit that’s running: master on staging, release on production. If it has moved, the redeploy would also ship new code, which muddies the roll’s evidence; restart by hand instead (below). A redeploy also restarts both cloudflared replicas at once. On hb-buzz production on 2026-10-29, that caused about 90 seconds of failed requests while Swarm’s DNS caught up. Restarting by hand avoids it.

• Don’t pass -e replicas=…. A replicas override turns the run into a full stack deploy, which skips the force update. With nothing else changed, no container restarts and nothing changes slot.
• celerybeat is briefly down on every roll. It has no healthcheck, so the old scheduler stops as soon as the new container starts. A task that falls due in the gap runs once the new scheduler is up.

To restart by hand instead, on a swarm manager:

sudo docker service update --force --update-order start-first hb_buzz_django
sudo docker service update --force --update-order start-first hb_buzz_admin_django
sudo docker service update --force --update-order start-first hb_buzz_celeryworker
sudo docker service update --force --update-order stop-first hb_buzz_celerybeat

celerybeat goes last, with stop-first: two schedulers at once would enqueue every periodic task twice. Say so explicitly, because a redeploy can leave the service set to start-first.

Each command waits for its service to converge only in an interactive terminal. When scripting, or with --quiet, check sudo docker service inspect --format '{{.UpdateStatus.State}}' <service> shows completed before starting the next.

5. Check every container

/ping isn’t enough. From the VPN it only reaches admin_django, and celeryworker and celerybeat have no /ping. Each container logs one line when it chooses a slot, so the logs are the check.

In Papertrail, search the hb_buzz services for db-slots from the restart time onward. Or use the CLI:

./bin/swo -c <your staging swo config> logs get --min-time <restart time> --max-time <now, UTC> "db-slots"

The roll is good when:

• Each restarted container logged exactly one [db-slots] serving on active slot <new> line. On staging, at one replica each, that’s four: django, admin_django, celeryworker, celerybeat.
• No [db-slots] line says WARNING or fallback. A fallback means the new slot failed its test, and that container is serving on the old slot. Read the reason in the line before changing anything:
  • a wrong credential;
  • a missing grant;
  • the wrong database;
  • an outage or network failure that may clear on its own. Then roll back or fix the spare slot and restart again.
• Sentry has no new “Database slot fallback” warning.

/ping is a quick extra check. It shows the container’s slot and whether it fell back, and never the login:

curl -s https://buzz.thb.sh/ping

{"version": "...", "pong": "123", "replica": "3f9a1c2e", "database": {"slot": "B", "fallback": false}}

From the database

pg_stat_activity shows which login each session logged in as, even though the session then runs as hb_buzz_user. It’s the check that names the login, which neither the logs nor /ping do. As the admin:

psql "$PGCONN" -c "SELECT usename, state, backend_start, now() - state_change AS in_state_for FROM pg_stat_activity WHERE usename LIKE 'hb_buzz_user%' ORDER BY usename, backend_start;"

After a roll onto hb_buzz_user_b:

• The new login has sessions that started after the restart. Compare backend_start with the restart time from step 4.
• The old login has no sessions started after the restart. Any older ones belong to containers that are still stopping, and they disappear within a minute or so.

Reading it:

• Zero rows for the new login straight after the restart can just mean no traffic yet. Django opens a connection on its first request and keeps it for CONN_MAX_AGE, and celery connects only while it runs a task. Request /ping, which queries the database, and run the query again.
• Before a roll, zero rows for the spare login is expected. Nothing uses the spare. add_db_slot_user already proved the login works when it created it.
• Sessions don’t map to containers. The stack reaches the database through its Cloud SQL proxy, so client_addr is the same for every container. Count containers with the [db-slots] lines, not here.

6. Retire the old credential

Wait until the old login has no sessions:

psql "$PGCONN" -c "SELECT usename, count(*) FROM pg_stat_activity WHERE usename LIKE 'hb_buzz_user%' GROUP BY 1;"

CONN_MAX_AGE (500 s in hb-buzz) only matters for processes that haven’t restarted. Once every container has restarted, the old login’s sessions should be gone within a minute. A celery worker connects only while it runs a task, so a zero here means something only after step 5 has shown no container is on the old slot.

Then, by roll:

• Password roll: the old slot is the next roll’s spare. Give its login a new password now, so the old one stops working. Rerun Password roll for the old slot. The script refuses while the login still has sessions.
• User roll: drop the login that left the vault in step 2, once it has no sessions:

  psql "$PGCONN" -c "DROP ROLE hb_buzz_user_b;"

  Slot logins never own anything, so the drop can’t fail on ownership. Never drop hb_buzz_user: it owns the tables.
• Endpoint roll: keep the old instance until the roll has been confirmed and backups run from the new one.

Roll back

Until step 6, the old slot is unchanged, so rolling back is a roll onto it: step 3 with NEW set to the old slot, then steps 4 and 5. Containers that fell back to the old slot are already on it, but restart them anyway so every container decides again from the same vault version.

To undo only the vault change before restarting anything, add a version with the old pointer, as in step 3, rather than disabling the new version.

After step 6 the old credential no longer works: its password changed, or its login was dropped. To go back then, prepare that slot again as a new roll (Password roll or User roll), then point at it.

Evidence for a pilot roll

For a roll that has to prove zero downtime, record:

• The roll’s start and end time (UTC), the commit, and which slot the stack moved from and to.
• The [db-slots] lines from step 5, one per container, with no WARNING.
• The pg_stat_activity output from steps 1, 5 and 6.
• Requests polled throughout, with the failure count. Start this before step 4 and stop it after step 5:

  while true; do echo "$(date -u +%H:%M:%S) $(curl -s -o /dev/null -m 5 -w '%{http_code}' https://buzz.thb.sh/ping)"; sleep 2; done | tee ping-poll.log

  Afterward, grep -vc ' 200$' ping-poll.log is the failure count. From the VPN this polls admin_django only; poll from off the VPN as well to cover django.
• For the per-replica proof, deploy the stack at two replicas first (-e replicas=prod, a full deploy), then roll. That means 6 [db-slots] lines: two each for django and admin_django, and one each for the celery services.

Adapting to another app

To run the commands above for hbcrm or bea_hbcrm, swap these values in. The instance, proxy port and admin are the same as hb-buzz’s.

For any app, look up:

• its owner role and database, in the instance or hb-ansible’s vars;
• its stack name and services, from docker stack services <stack>;
• whether its vault is in slot mode, with step 1.

hbcrm’s bees_ro and bees_write connections aren’t slots. A roll doesn’t touch them.

Gotchas

• Never rotate the live login’s password. PostgreSQL keeps one password per role, so new connections from processes that haven’t restarted would fail. add_db_slot_user refuses to write the live slot, and refuses a login that has open sessions.
• The owner role stays out of rolls. hb_buzz_user owns the tables, and hb-ansible keeps its password for backups and restores. Rotating it means updating hb-ansible in the same change.
• Scaling up doesn’t move existing containers. New replicas pick the current pointer, while untouched ones keep the slot they started with. A roll restarts every service explicitly.
• A Secret Manager blip can boot a container on an old vault version. fetch_secrets.sh falls back to a Redis cache of the last successful fetch. The container’s [db-slots] line and /ping show the old slot.
• The slot is decided once per container. Gunicorn workers, celery and manage.py reuse the first decision, so one [db-slots] line per container is expected, not one per process.