Runbooks

Step-by-step guides for common operations.

Remote Access

• SSH Into GCP VMs — connect to Google Cloud instances via IAP tunnels
• SSH Into Azure VMs — connect to Azure instances via Bastion or direct SSH

Development

• Dev Windows VM — spin up a temporary Windows dev box (VS, .NET, SQL Server, Claude CLI pre-installed) to build/debug HA apps

Maintenance

• Upgrade Caddy and Cloudflared — update reverse proxy and tunnel agents on production VMs
• Execute a Planned Outage — standardized process for scheduled downtime with stakeholder notification
• Create a New Custom GCP Boot Image — build a new ubuntu-2204-docker-falcon boot image via gcp_compute_image
• Rolling Boot Image Upgrade for Swarm Managers — roll a new boot image onto the three swarm managers without losing quorum
• Rolling Update of Swarm Worker Nodes — replace swarm worker VMs one at a time after a template change, scripted via roll_swarm_workers
• Recover Swarm Manager Quorum Loss — recover Raft quorum after losing two or more swarm managers
• Validate Swarm Worker Autoscale — exercise the end-to-end auto-scale lifecycle (MIG provision, swarmctl rebalance, worker drain) before phase rollouts or after autoscaler config changes
• HA Windows Worker Rolling Reboot — blue-green reboot of HA Windows worker VMs with zero customer downtime
• Rotate the HA SQL Admin Password — rotate haadmin on the staging and production Azure SQL servers, either in place (outage) or by first moving apps to haadmin2 (no outage)
• Recreate an Azure DevOps Environment — rebuild a corrupt ADO Environment (“Linked environment pool is null”) by renaming it aside and re-registering swarm-manager agents
• THB-P-SFTP-WIN Storage Cleanup — find what’s filling the HA SFTP server’s drives and move data off C: safely; covers the meal referral retention jobs and the low-disk alert

Debugging

• Debug Docker Container Issues — diagnose and recover stopped or failed containers
• Verify Azure Secret Values — check that Key Vault secrets match expected values
• Debug Cloudflare 403 Blocks — diagnose API requests blocked by Cloudflare using the Ray ID

Launches

• HA Admin Portal — end-to-end guide for launching a new partner HA Admin Portal
• Legacy CoPo — launch a new legacy Consumer Portal partner
• Benefits Hub — launch a new Benefits Hub tenant on the shared VM

Decommissions

• Legacy CoPo — retire a legacy Consumer Portal partner stack, including archiving its databases before teardown

Services

• FileMage SFTP Gateway — web portal + SFTP, tunneled via Cloudflare, with deploy and rotation procedures
• FileMage Mailbox Slack Notifier — Slack alerts when partners upload files, config-driven via a Secret Manager secret

Edit this page