RFC: HomeAlign (HA) Swarm Consolidation, Azure-Native vs GCP Migration

Status: Draft | Author: DevOps Team | Created: 2026-06-29 | Updated: 2026-07-08 Tracking: HB-8750 (Cost Benefit Analysis on HA Swarm Consolidation) | Parent context: HB-8200 (Docker Swarm Consolidation, GCP, ~87% complete)

Overview

The GCP Docker Swarm consolidation (HB-8200) is in production and delivering its promised savings: ~45 single-purpose VMs collapsed into a shared 3-manager + autoscaling-worker cluster with autohealing, memory-based autoscaling, and graceful drain. The HomeAlign (HA) platform did not get any of that. HA runs its own hand-built Docker Swarm on Azure (10 Linux managers + 9 Windows Server 2022 workers), statically sized, with no autoscaling, no autohealing, and a manual cordon/drain story.

This RFC decides how to bring the consolidation value to HA. Two paths:

• Path 1, Azure-native. Keep HA on Azure and build the missing elastic layer natively: the Azure equivalent of our GCP Managed Instance Groups (VM Scale Sets + autoscale + health-probe autohealing + Azure Monitor alerting), and re-point our swarmctl controller at Azure.
• Path 2, GCP migration. Relocate HA’s Windows worker tier to GCP, provisioning a Windows worker pool that joins a GCP swarm and reuses the consolidation automation we already built. HA’s Azure SQL databases stay on Azure (the Django tier already reaches them cross-cloud today).

Recommendation: Path 1 (Azure-native), now on risk grounds rather than cost, gated on a cheap cross-cloud latency spike for the .NET tier. With AHB treated as unavailable (an audit found it applied to none of the 9 Windows workers, and eligibility is unconfirmed), the Windows premium is identical on both clouds, so hard dollars land at near-parity with a modest Path 2 edge (roughly $25k cheaper at 3-year midpoints after the egress measurement, bands still overlapping). Path 1 is still recommended because it avoids the higher-impact risks: no cross-cloud database hop for the chatty .NET PMS, no surgery on the live shared GCP production swarm, no PHI re-homing, and it plays to the team’s existing Azure and Windows expertise. The trade is now explicit: Path 1 pays a modest cost premium to avoid an unproven latency risk and a PHI blast-radius change. If leadership prioritizes hard-dollar cost and the latency spike comes back clean, Path 2 becomes the better choice (conditions in the Decision section). Confirming AHB eligibility would erase the premium on Azure and restore Path 1’s cost parity, strengthening this recommendation.

---

Executive Summary

The decision in one line: Without AHB, both clouds bill the same Windows premium and hard dollars land at near-parity (a modest Path 2 edge, ~$25k at 3-year midpoints, bands overlapping); Path 1 is recommended because it avoids an unproven cross-cloud database latency risk for the chatty .NET tier and keeps PHI off a shared production swarm. The decision now turns on whether that risk avoidance is worth a modest cost premium, and on confirming AHB eligibility, which would erase the premium and restore Path 1’s cost parity.

---

Background

What we built on GCP (HB-8200)

Two consolidated swarms (non-production and production), each a fixed 3-manager control plane plus a regional Managed Instance Group of n2-highmem-4 workers. The pieces that matter for this decision:

• Worker MIG (infra/hb-infra/modules/swarm_worker/main.tf): a regional instance group manager, autoscaling on the Ops Agent memory metric agent.googleapis.com/memory/percent_used (70% target), autohealing on a TCP health check against dockerd’s :9323 metrics port, OPPORTUNISTIC rolling updates.
• swarmctl (src/services/swarmctl/, ~3,600 LOC Go): a manager-side controller that rebalances services onto new workers and drains a worker’s tasks before it leaves. GCP coupling is isolated to two small clients (~220 LOC, Cloud Monitoring + Compute API) behind dependency-injected interfaces, active only at rebalancer Phase 2 and above.
• Worker lifecycle (worker-startup.sh.tftpl, 509 lines of bash; systemd drain unit; swarm-reaper timer): join via token from Secret Manager, drain on scale-in, reap ghost nodes.

This is Linux and Ubuntu throughout. There is zero Windows support in the GCP swarm stack.

Current HA state on Azure (infra/ha-infra)

• No elastic layer. A repo-wide search finds zero VM Scale Sets, autoscalers, or health-probe autohealing in ha-infra. Every VM is a static for_each entry. Swarm formation is partly manual (association scripts run by hand after terraform apply).
• Two applications, the same per-tenant databases. The Windows workers run the HealthAlign PMS, the primary chatty transactional system (ASP.NET Core containers such as haap-webapp and haap-interfacetaskservice), deployed via per-tenant Azure DevOps release pipelines (YAML; the older Classic Release / deployment-group model has been retired). Separately, django_homealign, a lighter Django API/auth layer over the same per-tenant databases, was migrated onto the GCP swarm in HB-8200.
• 38 per-tenant Azure SQL databases (ha-prod1-azsqldb, elastic pools ep01/ep02), plus Azure storage and file shares the Windows app mounts.
• Deployment is Azure DevOps (self-hosted ADO agent VM, SSH service endpoints to every node on port 2290, Azure Container Registry, Azure Functions artifacts).
• django_homealign already runs GCP-compute against Azure SQL cross-cloud. The mechanism: GCP swarm Cloud NAT egress IPs are allowlisted on the Azure SQL Server firewall (src/scripts/add_az_sql_fw_rule.sh, imperative, not Terraform). The path is public internet plus Azure SQL firewall allowlist, no VPN or Interconnect. This proves the cross-cloud database pattern works for the light Django layer.

The problem

HA’s Windows tier has none of the consolidation benefits. It is statically over-provisioned (a 32 vCPU D32ds_v5 sits in the fleet permanently), has no automatic recovery when a Windows worker dies, and no graceful drain on planned replacement. We want to extend the HB-8200 elastic model to HA. The two paths differ only in which cloud the HA Windows swarm lives on.

---

Path 1: Azure-Native

Build the elastic layer that ha-infra is missing, on Azure, on top of HA’s already-working Windows swarm. HA stays exactly where it is. The PMS stays co-located with its databases.

flowchart TB
  ADO["Azure DevOps
per-tenant release pipelines"]:::ext
  subgraph MGRS["Manager plane (right-sized 10 to 3)"]
    M1["mgr-1 (Linux)"]:::mgr
    M2["mgr-2 (Linux)"]:::mgr
    M3["mgr-3 (Linux)"]:::mgr
  end
  SC["swarmctl
Azure Monitor client
(220 LOC swap)"]:::ctl
  subgraph VMSS["Windows worker VMSS (autoscale)"]
    W1["win worker"]:::wkr
    W2["win worker"]:::wkr
    Wn["...autoscale"]:::wkr
  end
  AS["Azure autoscale
memory via Azure Monitor Agent"]:::az
  HC["Automatic Instance Repair
TCP probe :9323"]:::az
  TN["Terminate Notification
5 to 15 min drain"]:::az
  SQL["Azure SQL
38 per-tenant DBs (co-located)"]:::data
  ADO -->|deploy over SSH| MGRS
  MGRS -.->|runs| SC
  SC -->|rebalance / drain| VMSS
  AS -.->|scales| VMSS
  HC -.->|autoheals| VMSS
  VMSS -->|ExecStop-equivalent drain| TN
  VMSS -->|in-region, sub-ms| SQL
  classDef mgr fill:#bfdbfe,stroke:#1d4ed8,color:#0b1e3b
  classDef wkr fill:#bbf7d0,stroke:#15803d,color:#06281a
  classDef ctl fill:#fde68a,stroke:#b45309,color:#3a2400
  classDef az fill:#99f6e4,stroke:#0f766e,color:#04241f
  classDef data fill:#ddd6fe,stroke:#6d28d9,color:#241152
  classDef ext fill:#e5e7eb,stroke:#4b5563,color:#111827

What we build:

• VMSS for the Windows workers with azurerm_monitor_autoscale_setting. Memory scaling needs guest-OS counters, which platform metrics do not include: the current (2026) mechanism is Azure Monitor Agent + a Data Collection Rule routing perf counters into the custom Virtual Machine Guest metrics namespace, which the autoscale rule must reference. The classic diagnostics-extension (WAD) path was deprecated 2026-03-31; do not build on it. Note the AMA-to-Metrics destination is still docs-flagged preview (risk table).
• Automatic Instance Repair with the Application Health extension probing dockerd :9323 (TCP handshake suffices), matching our GCP autohealing one-for-one. Pin the VMSS orchestration mode explicitly: Flexible orchestration (the default for new VMSS) does not support LB-probe health monitoring, so health goes through the App Health extension either way. Known blind spot carried over from GCP (HB-8725): a port-based TCP check bills a join-failed worker as healthy; same on Azure.
• Terminate Notification (5 to 15 minute window) as the scale-in drain hook, replacing the systemd ExecStop unit.
• Metrics backend: Azure Monitor managed service for Prometheus (backed by an Azure Monitor Workspace) as the GMP equivalent. The in-swarm Prometheus collector (cAdvisor, node-exporter, dockerd :9323, the Grafana stack) is already a set of Docker Swarm services and is cloud-portable; we keep it and change only its export target from the GMP sidecar flags to a native remote_write block pointed at the workspace ingestion endpoint (managed-identity auth, since the managers stay on Azure VMs). This is a config change to one service, not a self-hosted Prometheus storage stack: Azure Monitor managed Prometheus is GA, PromQL-compatible, and supports remote_write from plain VMs (not just AKS), with 18-month retention and no storage charge. A single in-swarm Grafana carries both clusters as datasources (the existing GCP GMP plus the new Azure Monitor managed Prometheus workspace), so dashboards stay one pane of glass rather than splitting into a second Grafana to babysit.
• Azure Monitor / Log Analytics alert rules to replace the 38 GCP Cloud Monitoring alert policies (10 worker + 10 manager + 18 observability; this expands toward ~56 monitoring objects once the per-manager uptime checks fan out across 10 managers). The ~24 PromQL policies become Azure Monitor Prometheus rule groups in the workspace; the ~8 native-metric threshold alerts become Azure Monitor metric alerts; the per-manager uptime check and the worker_join_failed log-based metric become an availability test and a Log Analytics KQL alert. The 6 metric-absence alerts (worker + manager condition_absent, the class that catches a dead agent or dead pipeline) have no native Azure metric-alert equivalent and need an explicit workaround (e.g. absent() in a Prometheus rule group for agent-shipped metrics, or a scheduled KQL heartbeat query); this is the one non-mechanical part of the port. With the manager plane right-sized to 3, the per-manager uptime fan-out shrinks and the total object count lands near ~41 to 43, not the ~56 a 10-manager fan-out would imply. ha-infra already uses Log Analytics + data collection rules, so this is a port, not a greenfield.
• swarmctl Azure client swap: reimplement the two thin clients (~220 LOC) against Azure Monitor metrics and the Azure Compute API, behind the existing interfaces. The drain server and rebalance state machine port unchanged.

What we reuse: the entire swarmctl control logic, the existing Windows swarm bootstrap (~2,000 lines of working PowerShell; ~850 in the bootstrap scripts proper), the existing Azure VM modules (azure_windows_vm, azure_linux_vm), and the existing ADO deploy path. The PMS data path is unchanged.

Opportunity (path-independent): right-size the over-provisioned manager plane (10 Linux managers down to 3 or 5; Raft needs an odd quorum, not ten), worth roughly ~$12k/yr. To be honest about attribution: this requires no elastic layer and is available today under any path, including doing nothing. It belongs with the other decision-independent quick wins (AHB flip if eligible, D32 right-size, spare deallocation), not in Path 1’s column.

Primary risk: Azure autoscale and repair semantics differ from GCP MIG (no exact OPPORTUNISTIC analog; memory needs the AMA guest-metrics pipeline, which is still docs-flagged preview; terminate notification fires on scale-in and delete but not on stop or deallocate). All are known and designable, but the preview status of the guest-metrics-to-autoscale path is a real dependency to spike early.

---

Path 2: GCP Migration

Stand up a Windows worker pool on GCP that joins a swarm and runs the PMS, reusing the GCP consolidation automation. The 38 Azure SQL databases stay on Azure; the PMS reaches them cross-cloud the way Django does today.

Two sub-variants:

• 2a, join the existing shared prod swarm. Cheapest in raw build, but it requires recreating the ingress overlay on the live cluster that already runs every consolidated GCP app, and it places HA’s PHI workloads in the same Raft cluster and ingress mesh as everything else.
• 2b, dedicated HA swarm on GCP. Stand up a separate 3-manager GCP swarm for HA to preserve isolation. Cleaner on blast radius and compliance, but it forfeits the “reuse the existing managers” benefit and adds ~1 week to build three new managers. If Path 2 is chosen, 2b is the responsible form.

flowchart TB
  ADO["Azure DevOps
(re-point SSH endpoints to GCP IPs)"]:::ext
  subgraph MGRS["Dedicated HA managers on GCP (2b)"]
    M1["mgr-1"]:::mgr
    M2["mgr-2"]:::mgr
    M3["mgr-3"]:::mgr
  end
  SC["swarmctl
(reused, GCP-native)"]:::ctl
  subgraph MIG["NEW Windows worker MIG"]
    W1["win worker"]:::wkr
    W2["win worker"]:::wkr
    Wn["...autoscale"]:::wkr
  end
  AS["MIG autoscaler"]:::gcp
  HC["MIG REPAIR :9323"]:::gcp
  NAT["Cloud NAT egress"]:::gcp
  SQL["Azure SQL (stays on Azure)
38 per-tenant DBs"]:::data
  FW["Azure SQL firewall
allowlist GCP NAT IPs"]:::data
  ADO -->|deploy over SSH| MGRS
  MGRS -.->|runs| SC
  SC -->|rebalance / drain| MIG
  AS -.->|scales| MIG
  HC -.->|autoheals| MIG
  MIG --> NAT
  NAT -->|public internet, tens of ms| FW
  FW --> SQL
  classDef mgr fill:#bfdbfe,stroke:#1d4ed8,color:#0b1e3b
  classDef wkr fill:#bbf7d0,stroke:#15803d,color:#06281a
  classDef ctl fill:#fde68a,stroke:#b45309,color:#3a2400
  classDef gcp fill:#99f6e4,stroke:#0f766e,color:#04241f
  classDef data fill:#ddd6fe,stroke:#6d28d9,color:#241152
  classDef ext fill:#e5e7eb,stroke:#4b5563,color:#111827

What we build:

• A new Windows worker MIG. A GCP MIG is single-OS by construction (one instance template, one boot image, one startup script), so Windows needs its own MIG. The Terraform scaffolding copies from swarm_worker (~1 to 2 days), but the 509-line bash startup script is a from-scratch PowerShell rewrite (~1 to 2 weeks): dockerd :9323 metrics, Ops Agent via googet, GCR auth for Windows, drain without systemd, swarm join. Realistic total for the pool: ~2 weeks.
• Ingress handling (2a: recreate the overlay on live prod; 2b: configure it on the new swarm) and cutover (DNS, deploy re-point, parallel run).
• Azure SQL firewall allowlisting for the new GCP NAT egress IPs (extends the existing imperative add_az_sql_fw_rule.sh pattern).
• A container registry decision. The ADO pipelines push Windows images (multi-GB) to ACR today. Either GCP workers pull from ACR cross-cloud on every deploy (recurring Azure egress, unmodeled until now) or the images migrate to Artifact Registry (a one-time work item, and Windows GCR/AR auth was its own 534-line problem on the Linux side). Folded into the cutover row of the effort table.
• (2b) three new GCP managers.

What we reuse: swarmctl directly (already GCP-native), the GCP MIG/autoscaler/autohealing pattern, the GCP monitoring stack.

What stays on Azure (and is therefore not “off Azure”): the 38 SQL databases, Azure storage and file shares, and Azure DevOps. HA’s system of record remains Azure-resident. Path 2 relocates compute, not the platform.

Primary risk (the linchpin): the PMS is the chatty transactional tier, and the current architecture deliberately co-locates it with its databases on Azure (per-tenant Windows VM next to per-tenant Azure SQL DB; databasedeploy runs migrations directly against the DB). Only the light Django layer was moved cross-cloud. Pushing the PMS to GCP puts the chatty tier a public-internet hop (tens of ms per round trip) away from its databases, which is exactly the arrangement the design avoids today. Whether the PMS tolerates this is unproven and the rationale for co-location is undocumented (tribal). This must be settled by a measurement spike before committing, because if it fails, Path 2 either degrades the PMS or forces the 38-database migration after all (months).

---

Cost Analysis

Both paths deliver the same elastic capability, so the comparison is cloud placement, one-time effort, and recurring cost, measured against a do-nothing baseline.

Baseline and path-independent quick wins

The baseline for all payback math is HA as it runs today: static fleet, no autoscaling, no autohealing, manual cordon/drain. Against that baseline, the elastic layer’s dollar return is thin. Autoscaling sheds ~20 vCPU (static 60 running to a ~40 vCPU average, itself an unvalidated assumption pending the per-tenant-pinning question), which at the verified E4bds_v5 Windows rate (0.518/hrper4 − vCPUVM)is * * 22.7k/yr gross**; net of the $10k to 20k/yrtwo − stackopstaxthatleaves * * 3k to 13k/yr * *, against * *39k to 58kone − time * * : * * payback 3yearsinthebestcase, 20 + yearsatthepessimisticend * *.WithAHBflippedon, shedVMssaveonlytheLinuxrate( 14.6k/yr gross), the net spans roughly -5kto+5k/yr, and there is likely no dollar payback at all. The elastic layer’s primary value is therefore reliability (autohealing, graceful drain, no manual 2 AM recovery for a 2-person team), with cost savings secondary at best. Both paths can be phased to front-load that value: Automatic Instance Repair plus the alert port first (~2 to 3 weeks, most of the reliability benefit), autoscaling second.

Separately, four savings require no elastic layer at all and proceed under any decision, including no decision. They belong to neither path and are not counted in either path’s column:

Operating assumption (updated 2026-06-30): AHB treated as unavailable. A portal/CLI audit found Azure Hybrid Benefit applied to zero of the 9 production Windows workers, and eligibility could not be confirmed from Azure (it lives in VLSC and the EA, not the data plane). Per direction, the cost case below now assumes no AHB as the baseline. This removes Path 1’s single biggest cost advantage and changes the conclusion of this section. The AHB-eligible scenario is retained as a reference at the end, because confirming eligibility would revive that advantage.

The Windows premium (no longer a differentiator without AHB)

GCP bills Windows at **0.046/vCPU/hourontopoftheLinuxmachinerate, undiscountable * *(SustainedUseandCommittedUseDiscountsapplytocomputeonly; thereisnoWindowslicenseCUD).AzurecoulderasethiswithAzureHybridBenefit, but * *withoutAHBAzurechargestheidenticalpremium * *(0.046/vCPU/hr, cross-validated to the cent across E4s_v5, E4bds_v5, and D32ds_v5). So both paths pay the same Windows license premium on the same running vCPU:

The premium is real and permanent on both clouds (~16k/yrautoscaledprod, plus 12.9k/yr non-prod), so it remains a reason to autoscale aggressively and right-size the D32 node, but it no longer separates the paths. Path 1’s decisive cost advantage in the AHB-eligible case is gone without AHB. Note the non-prod Windows fleet (THB-N-WEB1..4-WIN) was previously uncounted in this RFC; whichever path is chosen must also cover non-prod (standing decision keeps swarm replicas identical across environments), and the AHB upside includes it.

SKU note. The per-VM base-compute figures in the backing analysis use Azure E4s_v5 as the reference SKU, but HA’s actual workers are E4bds_v5 (the local-disk variant, roughly 13% higher base rate). This does not affect the premium figures above: they are pure vCPU × 0.046/vCPU/hrandthereforeSKU − independent, thebasecomputeistreatedasnettingoutacrosscloudsintheTCO, andtheAHBdelta(0.184/hr per 4-vCPU VM) is unchanged. The $0.046/vCPU/hr premium was cross-validated against Azure’s Windows-minus-Linux delta on E4s_v5, E4bds_v5, and D32ds_v5, all three resolving to $0.046/vCPU/hr to the cent.

Confirming AHB eligibility remains the single highest-value cost input, because it is the one lever that swings the comparison back toward Path 1: with SA-covered Windows Server licenses, Path 1 drops the premium to 0(a 16k to 37k/yrswingacrossprodandnon − prod)whilePath2cannot(GCPBYOLexistsonlyviasole − tenantnodes, whosesmallestnoderuns 4.5k/mo, clearly uneconomical at this fleet size). Eligibility is plausible (HA runs Windows on Azure under an existing Enterprise Agreement, and one VM already carries the Windows_Server flag), but unconfirmed, and it lives in VLSC/the EA rather than the Azure data plane (see Open Question #1). The Azure Reserved Instances sign-off in HB-8048 is a separate billing commitment and does not establish license ownership. Until licensing confirms SA coverage, this RFC costs the conservative no-AHB case above.

Cross-cloud egress (Path 2 only, permanent)

If the PMS runs on GCP and queries Azure SQL, query traffic and result sets cross clouds on every transaction, forever (GCP egress $0.12/GB for the first 1 TB then $0.11/GB to 10 TB, Azure egress 0.087/GBforthefirst10TBaftera100GBfreeallotment, ingressfreebothsides).Thestandardpublic − internetegresstiersabovearetherelevantonesfortheCloudNATpath; the2026 − 05 − 01GCPpeering − rateincreaseappliesonlytoDirect/CarrierPeeringandCDNInterconnect, notthispath.ForthelightDjangolayerthisistolerabletoday. * *ForthePMSthevolumeisnowmeasured, anditissmall. * *A14 − daypullofAzureMonitorNICcountersacrossall8runningWindowsworkers(2026 − 06 − 24to2026 − 07 − 08, deliberatelyspanningamonth − end)shows 1, 737GB/moreceivedand 1, 951GB/mosent, fleet − wide.Pricedascross − cloud : eventhehardupperbound(assuming100376/mo ≈ 4.5k/yr * *; therealisticAzure − boundsharelandsat * * 1.5k to $2.6k/yr**. The earlier $3k to 12k/yrplaceholderwas2to4xtoohighatitstopend.TheTCOcarries * *2k to 5k/yr * *.Isolatingthe * *databasetrafficspecifically * *(subtractingmeasuredfile − share, registry, andoverlaycomponentsfromtheNICtotals) : SQLresultsetsare 690to1, 240GB/moandqueries/writes 100to400GB/mo, so * *cross − cloudDBtrafficalonepricesat 0.9k to $1.9k/yr, versus 0withcomputeco − locatedinAzure * *.Thedatabaseegressquestionissettled : itdoesnotmovethedecision.Remainingtailrisk : any * new * bulkflowoverthelink(anightlyexportorbackup, +130 to $180/mo per 50 GB/night) is additive and should stay carved out. Measurement detail: .plans/HB-8750-ha-swarm-cba/egress-measurement.md. Path 1 has zero cross-cloud egress.

Observability layer (small, roughly symmetric)

Path 1’s observability is a managed-service port, not a self-hosted stack: the in-swarm Prometheus collector remote-writes to Azure Monitor managed service for Prometheus (the GMP equivalent), exactly as it writes to GMP today. Hard-dollar service cost for the ~12-node HA swarm is small: managed-Prometheus ingestion ~$30 to $100/mo, alert rules $0 to $20/mo (Prometheus rule groups are billed as negligible query samples), workspace base $0, modest Log Analytics, and Grafana at 0(keptself − hostedin − swarm). * *Leantotal 50 to 150/mo( 600 to 1, 800/yr); * * AzureManagedGrafana, ifadopted, wouldadd 80 to 95/mobutisoptional.ThisisroughlysymmetricwithPath2, wheremonitoringthesamenodesfoldsintotheexistingGCPGMPstackatasimilarmarginalingestioncost( 300 to $900/yr), so the service-fee delta between paths is a few hundred dollars a year, well inside the noise. The material Path-1 observability cost is the labor of operating a second managed stack, which is captured in the two-stack ops tax below, not in service fees. Detail: .plans/HB-8750-ha-swarm-cba/observability-azure.md.

One-time effort

(Earlier drafts stated Path 2 at ~3 to 5 weeks, which did not sum to its own rows; corrected.) At ~108/hrloadedfor 1.5engineers(6,480/wk), one-time engineering is roughly $39k to 58k(Path1) * *vs * *32k to 39k(Path2) * *, adeltaofabout * *7k to $19k, not the larger gap earlier drafts implied.

3-Year TCO (no-AHB operating case)

Assumptions: ~40 Windows vCPU autoscaled average (prod), no AHB (the premium is paid on both clouds, so it cancels), base compute netting out only under matched 3-year commitments (Azure RI 41%/62% vs GCP CUD 37%/55% almost exactly cancel GCP’s ~18.5% cheaper on-demand list price, a ~$7.3k/yr gap otherwise; if the HB-8048 Azure RIs cover these workers, Azure’s marginal cost is below GCP on-demand until a GCP CUD is signed, and mid-term migration strands the commitment; actual RI coverage must be pulled from Cost Management), egress at its measured band of $2k to $5k/yr (see the egress section; formerly a $3k to $12k/yr placeholder), two-stack ops tax for Path 1 at $10k to $20k/yr (the labor of a second monitoring stack, autoscaler, swarmctl deployment, and runbooks for a 2-person team), and a residual ops tax for Path 2b of $5k to $10k/yr (it is also a second cluster: its own managers, alerts, and runbooks, plus the imperative cross-cloud SQL firewall allowlist, egress monitoring, and Windows host/container build lockstep; 0wasnotdefensible).TheAzureobservability * service * fees( 0.6k to $1.8k/yr, see above) are small and roughly symmetric, so not broken out.

**Without AHB, hard dollars are near-parity with a modest Path 2 edge: roughly $25k cheaper at midpoints over three years, with overlapping bands.** (Earlier drafts claimed "$32k to $62k cheaper" from arithmetic errors since corrected, then "$13k“ using the pre-measurement egress placeholder; the measured egress tightened Path 2’s band.) Recurring spend alone is likewise comparable: Path 1’s ops tax ($10k to 20k/yr)vsPath2^′segressplusresidualops(7k to $15k/yr). The honest no-AHB read: cost leans modestly toward Path 2 but does not decide this RFC; risk and strategy do.

Reference, AHB-eligible scenario. If licensing confirms SA coverage, the Windows premium row becomes Path 1 $0 / Path 2 48k, movingthetotalstoroughly * *69k to 118k(Path1) * *vs * *104k to 153k(Path2) * * : aclear 35k midpoint edge to Path 1. Recompute the moment eligibility is settled; it remains the single input that most changes the cost picture.

---

Risks and Mitigations

---

Decision

Recommendation

Proceed with Path 1 (Azure-native) on the conservative no-AHB baseline, subject to one pre-decision gate and one cost-upside check:

1. Run a cross-cloud latency spike for the PMS (the gate). Even though we expect to stay on Azure, it de-risks any future GCP option and quantifies the co-location assumption that is currently undocumented. If it comes back clean and leadership prioritizes hard dollars, Path 2 becomes viable.
2. Confirm AHB eligibility (cost upside, no longer a gate). We now cost the conservative no-AHB case. If licensing confirms SA-covered Windows Server licenses, Path 1 drops the Windows premium to 0, givingitaclearcostedgeandcapturinga 24k/yr prod plus ~$12.9k/yr non-prod saving on the existing fleet regardless of which path is chosen.

This RFC presents two build options. If leadership judges neither return acceptable (the payback math in the Cost Analysis baseline is deliberately candid), HA simply stays as it is, and the path-independent quick wins proceed regardless. Between the two:

Path 1 wins on the factor that most separates the build paths: risk (no chatty cross-cloud database hop, no surgery on the live shared production swarm, no PHI re-homing). On cost, the picture has shifted: without AHB both clouds bill the identical Windows premium, and the corrected TCO lands at near-parity (Path 2 ahead by ~$25k at 3-year midpoints; egress, once measured, turned out too small to offset its lower one-time effort and smaller ops tax). Path 1 also plays to the team’s Azure and Windows strengths (Miko) and keeps HA consistent with the standing “Azure hosts HomeAlign only” decision. The reusable crown jewel, swarmctl, ports to Azure for ~220 lines, so we do not forfeit the consolidation IP by staying on Azure.

The genuine cost in Path 1’s column is now twofold: maintaining a second swarm stack (the two-stack ops tax), and, without AHB, forgoing Path 2’s slightly lower hard-dollar midpoint. Both are real and bounded. The recommendation holds because the unproven cross-cloud PMS latency risk and the PHI blast-radius change weigh more heavily than a cost gap that the corrected arithmetic shows to be within noise. That judgment reverses if the latency spike clears and leadership prioritizes cost over the risk premium.

When Path 2 would be the better choice

Path 2 (specifically 2b, a dedicated HA swarm on GCP) becomes the right call if all of the following hold:

• The team is not AHB-eligible — now the operating assumption, so the Windows premium delta is already near zero and this condition is currently satisfied, and
• The PMS latency spike comes back clean (cross-cloud Azure SQL is acceptable for the chatty tier), and
• Leadership weights single-stack consolidation and a GCP-centric future highly enough to accept a permanent cross-cloud data path and the cross-cloud egress that comes with it.

With the no-AHB assumption already satisfying the first condition, the decision now rests on the remaining two: the latency spike and leadership’s cost-versus-risk weighting. If the spike clears and leadership prioritizes the modestly lower hard-dollar cost, Path 2 is the better call; otherwise Path 1’s risk avoidance wins. (Confirming AHB eligibility would un-satisfy the first condition and swing the cost case back to Path 1.)

Decision Matrix

Success Criteria (Path 1)

---

Open Questions / Pre-Decision Spikes

1. AHB eligibility (cost-upside lever; the RFC now baselines the conservative no-AHB case, so this no longer gates the recommendation, but confirming it would restore Path 1’s cost parity). Owner: licensing/EA contact. Update (2026-06-30, Azure CLI audit of subscription 5a7e7941): AHB is usable in the tenant but currently unused on the workers. Of 34 Windows VMs, exactly one (Test-VM, deallocated) has licenseType=Windows_Server set; all 9 production swarm workers (THB-P-WEB{1..9}-WIN) have it unset, and the 8 running ones are paying full pay-as-you-go Windows rates today (the 9th, the blue-green spare THB-P-WEB8-WIN, is deallocated and bills nothing). The 4 non-prod workers (THB-N-WEB{1..4}-WIN, D8s_v3, all running) also have it unset. The Test-VM attestation is soft evidence the org holds qualifying Windows Server + Software Assurance licenses (Azure requires that self-attestation to set the flag), but it does not prove the fleet is covered (Azure does not validate at set-time; compliance is audited separately, and the prod+non-prod fleet needs roughly 128 core-licenses at the 8-core-per-VM minimum). The EA/licensing owner must still confirm fleet-wide coverage. Two consequences for this RFC: (a) Path 1’s “AHB removes the premium” is a state we would switch on, not today’s baseline, so the consolidation savings include capturing AHB; (b) decision-independent quick win: if coverage is confirmed, flipping AHB on the existing workers (az vm update --license-type Windows_Server, no downtime) captures roughly 24.2k/yrontherunningprodfleetplus 12.9k/yr on non-prod (~$37k/yr total) immediately, regardless of Path 1 vs Path 2.
2. PMS cross-cloud latency spike. Stand up one Windows PMS container on GCP (or simulate), point it at a non-prod Azure SQL DB via the NAT-allowlist path, and measure end-to-end request latency vs the co-located baseline. Cheap, decisive, de-risks both paths.
3. Current Azure HA spend baseline (HB-7312, currently unstarted). Needed to quantify consolidation savings on top of the path choice.
4. Cross-cloud egress volume for the PMS tier Resolved 2026-07-08 by measurement: ~3.7 TB/mo fleet-wide NIC traffic; upper bound ~4.5k/yr, realistic 1.5k to $2.6k/yr. Not decision-relevant. See .plans/HB-8750-ha-swarm-cba/egress-measurement.md. Residual watch item: new bulk flows (exports/backups) over the link are additive.
5. Overlay encryption check (Path 2 only): are the GCP swarm overlays --opt encrypted? Encrypted overlays silently sever Windows containers, a hard blocker for 2a.
6. Swarm vs Kubernetes (orthogonal but worth raising): both clouds route Windows containers to managed Kubernetes; a future modernization could make this whole question moot. Out of scope here, flagged for the roadmap.

---

Sources

Internal: docs/rfcs/docker-swarm-consolidation.md, docs/swarm.md, docs/services/swarmctl.md, docs/specs/docker-swarm-outbound-ip-audit.md, infra/hb-infra/modules/swarm_worker/main.tf, src/services/swarmctl/, infra/ha-infra/business_unit_1/production/{main.tf,databases.tf}, infra/ha-infra/scripts/, ~/nexus/decisions/log.md. Jira: HB-8200, HB-8750, HB-8048, HB-7312, HB-8725. Full analysis and external pricing/feature citations: .plans/HB-8750-ha-swarm-cba/research.md.

Confidence: VM sizing, GCP coupling, and the cross-cloud Django-vs-PMS distinction are grounded in the code. Windows premium math is exact ($0.046/vCPU/hr). Azure dollar figures use third-party aggregators (official pages are JS-rendered) but cross-validate with GCP to the cent on the implied premium. Effort and ops-tax figures are order-of-magnitude planning estimates. The RFC now baselines the conservative no-AHB case (audit found AHB applied to none of the 9 workers, eligibility unconfirmed), under which both clouds bill the identical Windows premium, Path 2 is modestly cheaper on hard dollars, and Path 1’s case is risk-driven; confirming AHB eligibility would erase the Azure premium and restore Path 1’s cost parity. Current Azure HA spend (HB-7312) remains unverified.

---

Revision Notes

2026-06-30 (rev 2): Verified the cost case against live Azure and GCP pricing and recosted the conclusion.

• Pricing verification. Sourced current (June 2026) GCP and Azure pricing (two parallel research passes, aggregators cross-checked against official docs). Every figure in the original cost section held: the GCP Windows premium ($0.046/vCPU/hr, undiscountable), n2-highmem-4 base rate, N2 CUD/SUD percentages, and both clouds’ egress tiers all confirmed. The $0.046/vCPU/hr premium cross-validated to the cent against Azure’s Windows-minus-Linux delta on three separate SKUs (E4s_v5, E4bds_v5, D32ds_v5). Added a SKU note (the real fleet workers are E4bds_v5, ~13% pricier base than the E4s_v5 stand-in, but the premium is SKU-independent so headline figures are unaffected) and tightened the egress-tier wording. Full detail: .plans/HB-8750-ha-swarm-cba/cost-verification.md.
• AHB audit. A portal/CLI audit of the HealthAlign subscription (5a7e7941) found Azure Hybrid Benefit applied to zero of the 9 production Windows workers (only one deallocated Test-VM carries the flag), and eligibility could not be confirmed from Azure (it lives in VLSC/the EA). The org is currently paying the full Windows premium on Azure too. Recorded in Open Question #1 with a decision-independent quick win: if licensing confirms SA coverage, flipping AHB on the existing workers captures ~$25.8k/yr immediately.
• No-AHB recosting (the substantive change). Per direction, the cost analysis now baselines the conservative no-AHB case. Without AHB the Windows premium is identical on both clouds and cancels between paths, so Path 2 is modestly cheaper on hard dollars (~$32k to $62k less over three years). The recommendation stays Path 1 but its basis moved from “cheaper and safer” to risk-only: Path 1 now pays a modest cost premium to avoid the unproven cross-cloud PMS latency risk and the PHI blast-radius change. AHB was demoted from a pre-decision gate to a cost-upside lever (confirming it restores Path 1’s cost parity). Reconciled the Overview, Executive Summary, decision-in-one-line, Cost Analysis, Decision section, Decision Matrix, “When Path 2 wins,” and confidence note for consistency.

2026-07-06 (rev 3): Adversarial review of the full CBA (three passes: internal consistency against the repo and live Azure, technical fact-check against Microsoft Learn/Docker/Mirantis/GCP docs, cost-assumption fact-check against the Microsoft Retail Prices API). Full findings: .plans/HB-8750-ha-swarm-cba/adversarial-review.md. Corrections applied:

• TCO arithmetic. Path 2’s effort total did not sum to its rows (now ~5 to 6 wk); one-time engineering rows now derive from the stated 108/hrx1.5 − engineerbasis(39k to $58k vs $32k to 39k); egresscarriedasafullrange; Path2bassignedaresidualopstax(5k to $10k/yr) instead of an indefensible $0. Headline changed accordingly: rev 2’s “Path 2 cheaper by $32k to $62k" is corrected to **near-parity, ~$13k Path 2 edge at midpoints with heavily overlapping bands**. Cost no longer decides the RFC in either direction.
• Fleet facts. The blue-green spare (THB-P-WEB8-WIN) is deallocated and bills nothing, so the billed prod premium is 60 vCPU (~24.2k/yr), not64( 25.8k); OQ#1 corrected. The previously uncounted non-prod Windows fleet (THB-N-WEB1..4-WIN, 4x D8s_v3 = 32 vCPU, ~12.9k/yrpremium)isnowscopedintothepremiumtable, theAHBupside( 37k/yr total), effort, and observability sizing.
• Technical mechanics. Memory autoscale must use AMA + DCR into the custom Virtual Machine Guest namespace (WAD deprecated 2026-03-31; pipeline docs-flagged preview, added to risks); VMSS orchestration mode pinned (Flexible has no LB-probe health, App Health extension required); the 6 condition_absent alerts called out as the one non-mechanical part of the alert port; HB-8725’s TCP-health blind spot added to the risk table; Path 2 given a container-registry line (ACR cross-cloud vs Artifact Registry migration).
• Base compute. “Nets out” now stated precisely: true only under matched 3-yr commitments (Azure RI 41/62% vs GCP CUD 37/55% cancel GCP’s ~18.5% cheaper on-demand list); actual HB-8048 RI coverage must be pulled from Cost Management.
• Attribution. Manager right-sizing (~$12k/yr) reframed as path-independent, alongside the other decision-independent quick wins.
• Fact-checks that held: the $0.046/vCPU/hr premium and its three-SKU cross-validation, egress tiers, AHB rules, terminate-notification semantics, encrypted-overlay-severs-Windows, Mirantis Swarm support through 2030, ~28 to 40 ms us-central1 to East US latency, GCP BYOL uneconomical via sole-tenant, deallocated-VM billing, and all LOC counts in this RFC.
• Still open (structural, needs direction): reconciling per-tenant node pinning and per-tenant ADO pipelines with fungible autoscaled workers (both paths, likely the largest unpriced work item); pricing the latency spike from GCP us-east4 (single-digit ms to East US 2) instead of us-central1; a quantitative pass/fail definition for the spike; PHI-in-transit treatment for 2b.

2026-07-08 (rev 4): Added the missing baseline, scoped per direction: the RFC presents two build options only (Path 1 and Path 2); “do nothing” is not pitched as a third path, it is what happens if leadership judges neither viable. Concretely: a “Baseline and path-independent quick wins” subsection now opens the Cost Analysis, carrying the candid payback math against today’s static fleet (the elastic layer nets ~$3k to $13k/yr after ops tax for $39k to 58kone − time, soitsprimaryvalueisreliability, notcost)andthefourquickwinsthatproceedunderanydecision(AHBflip 37k/yr if eligible, manager right-size ~$12k/yr, D32 right-size, spare deallocation), attributed to neither path. A phasing note (Automatic Instance Repair + alert port first, autoscaling second) replaces the earlier idea of a separate partial option. The Decision section states plainly that if neither path’s return is acceptable, HA stays as-is and the quick wins proceed regardless.

2026-07-08 (rev 5): Corrected the baseline payback figure. The earlier “2.5 to 8 year payback” rested on a ~$27k/yr gross autoscaling-savings ballpark that was never derived from the verified rates. Derived properly (20 vCPU shed at the verified E4bds_v5 Windows rate of 0.518/hrperVM= 22.7k/yr gross; net ~$3k to $13k/yr after the ops tax), the payback is **~3 years best case, 20+ years at the pessimistic end, and likely no dollar payback at all if AHB is enabled** (shed VMs then save only the Linux rate). The correction strengthens, not weakens, the existing conclusion that the elastic layer is a reliability purchase. The ~40 vCPU autoscaled average remains an unvalidated assumption pending the per-tenant-pinning question.

2026-07-08 (rev 6): Measured the Path 2 cross-cloud egress question (Open Question #4) instead of estimating it. Azure Monitor NIC counters for all 8 running Windows workers over 14 days (spanning a month-end) show ~1,737 GB/mo in and ~1,951 GB/mo out fleet-wide. Priced as cross-cloud, the hard upper bound is ~4.5k/yrandtherealisticAzure − boundshareis 1.5k to $2.6k/yr; the $3k to $12k/yr placeholder was 2 to 4x too high at the top. TCO egress row tightened to $6k to $15k (3 yr), moving Path 2’s total to $101k to 132kandthemidpointgapto 25k in Path 2’s favor (from ~$13k). Direction unchanged: near-parity, risk still decides. The egress argument against Path 2 is now empirically dead; the remaining egress risk is any new bulk flow (exports, backups) added over the link. Raw data and method: .plans/HB-8750-ha-swarm-cba/egress-measurement.md.

Edit this page