RFC: Deploy the InterfaceTaskWorker to the GCP Swarm

Table of Contents

• Summary
• Open Items
• Why This Workload
• Current State
  • The Azure Runtime Today
  • The Runtime Contract
• What the GCP Swarm Already Provides
• Relationship to healthAlignPMS#2423
  • What the First Deploy Taught Us
• Gaps and Proposed Changes
  • 1. Config and Secret Delivery
  • 2. Placement and Networking
  • 3. Image Pull
  • 4. Deploy Path
  • 5. The Azure Files Share
  • 6. Rebalance Churn
• Proposed Compose File
• Extending to IdentityServer
• Phased Rollout
• Risks and Open Questions
• Effort
• Alternatives Considered
• Verification Checklist
• References

Summary

The .NET 10 InterfaceTaskWorker is the first HomeAlign workload that can move onto the consolidated GCP swarm without the Windows worker-pool build-out that the HA consolidation RFC prices at 5 to 6 weeks. It is Linux, headless, single-replica, and its image is already built on ubuntu-latest and pushed to GCP Artifact Registry. The cross-cloud Azure SQL firewall allowlisting it depends on is already in place in both environments.

None of the required work is in the application. It is six changes to the deployment contract, all of which follow from one root cause: the worker today runs on a two-node swarm where the Linux node is simultaneously the Raft manager, the deploy target, and the box the container runs on. Every piece of its configuration arrives as a host bind mount placed there moments earlier by the same deploy. On the GCP swarm the task lands on an ephemeral autoscaled MIG worker, and none of those host paths exist.

The recommendation is a phased move that starts with the InterfaceTypes needing no file-share access, which yields a real cross-cloud latency measurement on production data with a one-line rollback.

Scope note. healthAlignPMS#2423 was authored independently and in parallel with this document and carries the implementation, which is now live on core-staging and core-prod. This RFC is therefore not the implementation plan - it is the decision record for the choices that PR makes, what the first real deploy proved or disproved, and what remains open. See Relationship to healthAlignPMS#2423.

Second workload. The same deployment contract has since been applied to IdentityServer (healthAlignPMS#2487), which runs on the staging swarm as of ADO run 14383 (2026-09-04); production has not been attempted. It is recorded here because it is the first test of whether this is a reusable pattern or a one-off — it is — and because two of its differences are load-bearing. See Extending to IdentityServer.

Open Items

The implementation has shipped to both environments. These are the things that have not been settled, kept here as an action list rather than only as rows in the risk table further down.

1. PHI placement on the shared production swarm

Status: open. No approving owner or risk acceptance is recorded anywhere. This is why the Status field above is qualified rather than a clean “Accepted”.

core-prod has been running on the shared production swarm since ADO run 13460, processing 837 claims files with the HA SFTP share CIFS-mounted into the task. The HA consolidation RFC argues specifically against that arrangement in its Path 2a and recommends a dedicated HA swarm (2b) if HA workloads go to GCP. So the placement this RFC implemented is the one the parent RFC advises against, and nothing on the record says that trade was accepted.

It has now been decided three times operationally — a staging default, a prod parameter override, and a prod release — rather than once deliberately. healthAlignPMS#2423 still leaves its Auth / PHI-PII exposure change box unchecked. Parent ticket HB-8824’s framing (“consolidate HA VMs onto a single swarm”) may settle it above the PR, but that is an inference, not a record.

What closes this — one of:

• A named owner accepting shared-swarm placement for HA PHI workloads, recorded here with a date; or
• A decision to move to a dedicated HA cluster (parent RFC’s Path 2b), with a migration date — in which case this RFC’s placement is superseded and §2’s network and placement choices get revisited; or
• A documented finding that the parent RFC’s concern does not apply here (for example that the swarm’s isolation properties are sufficient for this workload class), with whatever review that requires.

Who decides: not the author of this RFC. It needs DevOps plus whoever owns the HA compliance posture.

Until then: production releases default to deployTarget: azure-vm, so reaching the swarm stays an explicit per-run choice rather than something a routine release does. That is a holding position, not a resolution. Staging is different: healthAlignPMS#2509 defaults both staging releases (worker and IdentityServer) to gcp-swarm, since both paths are proven there and staging carries no PHI question.

2. swarmctl rebalance exclusion

Status: open, and this one is live exposure rather than a decision. Both environments run on autoscaling worker pools with no rebalance exclusion, so every scale-out event is a chance for swarmctl to force-update the single-replica worker mid-task, orphan-reset its queue row, and re-run it with duplicated external side effects. See §6 and risk 6.

Closing it is one pattern per environment in src/services/swarmctl/envs/staging.yaml and production.yaml, under rebalance_excluded_services:

rebalance_excluded_services:
  - "*_redis"
  - "*-interfacetask-stack_worker"

The patterns are path.Match globs against the full service name (<stack>_<service>), so one entry covers every tenant and environment; a bare worker matches nothing and would look configured. Infrahive change, so it does not block the app PR. Owner unassigned.

Note the caveat in the swarmctl doc: a single-replica service is still rebalanced by default, and should be excluded “only when a mid-task restart is worse than the imbalance”. For this worker it is — an orphan reset re-runs the task and repeats its external side effects — but that is the argument to make in the PR rather than assume.

3. Cross-cloud latency measurement

Status: open. Phase 0 was meant to measure this before anything was committed; it was skipped and both environments shipped, so it is now a post-deployment measurement rather than a gate. Scope and exit criterion are in risk 2. Cheap to take now that core-staging is live and cross-cloud. Owner unassigned.

4. IdentityServer swarm deploy is validated on staging only

Status: staging done, production open. healthAlignPMS#2487 deployed to the staging swarm on ADO run 14383 (2026-09-04): all three services converged, caddy resolved its upstream on the stack-scoped overlay, the tunnel served the build publicly, and the driver’s health check passed — on the fourth attempt, after three fixes recorded under What the staging runs taught us. The configure-cloudflared.yml question was settled by duplicating the credential pull in a swarm-specific step rather than parameterising the shared template. The PR merged on 2026-09-08. Still open: a production run, and the same dual-instance hazard the worker has — an Azure and a swarm IdentityServer connected to one tunnel run against one database, so the incumbent must be stopped before the swarm deploy and the swarm stack removed before switching back. Owner unassigned.

5. IdentityServer is not multi-instance safe

Status: open; fix in progress. Planning two swarm replicas (healthAlignPMS#2507, HB-9035) surfaced three pieces of per-process state in IdentityServer that break the moment two instances share auth.myhaapp.com — which is already the case whenever an Azure and a swarm instance are both connected to the tunnel, replicas or not:

1. OpenIddict signing and encryption keys are per container (AddDevelopment{Signing,Encryption}Certificate()): two instances advertise different JWKS and cannot decrypt each other’s authorization codes or refresh tokens, and every container recreate rotates the keys. HB-9040 — healthAlignPMS#2510 loads stable certificates from the vault; the vault keys must exist in both environments before it deploys, or the container exits at startup by design.
2. No shared ASP.NET Data Protection key ring: auth, 2FA and SAML session cookies and antiforgery tokens from one instance are unreadable on another. HB-9041, not started.
3. AuthController hands consumer-portal claims off through IMemoryCache, and the portal’s callback may land on the other instance. HB-9042, not started.

Until all three land, the safe operating mode is one instance per hostname: stop the Azure IdentityServer before a swarm deploy and vice versa. The replica change is held as a draft behind them. See What two replicas need. Owner unassigned.

Why This Workload

The HA consolidation RFC’s Path 2 (GCP migration) carries two costs that do not apply here:

• No Windows tier. Path 2’s dominant line item is a from-scratch Windows MIG with a PowerShell rewrite of the 509-line worker bootstrap. The worker is Linux and joins the existing pool.
• Not the chatty request path. Path 2’s linchpin risk is moving the transactional PMS webapp away from its co-located Azure SQL databases. The worker is a polling batch host — a poll loop on a 60s default interval dispatching discrete tasks — which tolerates tens of milliseconds of added round-trip far better than a per-request web tier. This is closer to django_homealign, which already runs GCP compute against Azure-resident data.

It is also the smallest available blast radius: headless, no ingress, no published ports, no Cloudflare tunnel, no DNS, one replica.

Current State

The Azure Runtime Today

The worker runs on the HA Azure swarm, which is not one cluster but a set of per-WEB-group two-node swarms: one Linux manager plus one Windows worker, joined by hand (see scripts/manual/readme.md). Non-prod is THB-N-WEB3-LINUX, prod is THB-P-WEB5-LINUX.

flowchart LR
  ADO["Azure DevOps
static-key SSH endpoint"]:::ext
  subgraph VM["THB-P-WEB5-LINUX - manager AND runtime"]
    FS["/DevOps/.vars/.core-prod
/etc/gcp/service-account.json
/mnt/ha-sftp-share (CIFS)"]:::data
    C["worker container
replicas: 1"]:::wkr
  end
  SQL["ha-prod1-azsqldb
hACore"]:::data
  ADO -->|"scp vars + pwsh deploy"| VM
  FS -.->|"bind mounts"| C
  C -->|"in-region"| SQL
  classDef wkr fill:#bbf7d0,stroke:#15803d,color:#06281a
  classDef data fill:#ddd6fe,stroke:#6d28d9,color:#241152
  classDef ext fill:#e5e7eb,stroke:#4b5563,color:#111827

The deploy is an ADO SSH service connection into that VM running compose/deploy_interfacetaskworker.ps1, which docker stack deploys compose/deploy_interfacetaskworker.yml. Because manager and runtime are the same box, host bind mounts are a valid configuration channel. That is the assumption the move breaks.

The Runtime Contract

Everything the container needs at start, and where it comes from:

What the GCP Swarm Already Provides

Verified against the code and, where noted, against the live cloud:

The single most expensive-looking prerequisite, cross-cloud database access, is therefore already done.

Relationship to healthAlignPMS#2423

PR 2423 (HB-8825) adds a deployTarget parameter to the worker’s release pipelines, a bash driver, and a deploy_interfacetaskworker.gcpswarm.yml stack file. It reaches the same conclusions as this RFC on the root cause, placement, networking, the image-pull path, the bash driver, and preserving the Azure path - arrived at separately, which is useful corroboration for both.

The PR has since converged on this RFC in two of the three places the two diverged. Current state:

The SSH transport divergence is also resolved: the PR moved to the IAP sequence (83a32782), so §4’s four prerequisites are now the PR’s actual prerequisite 1.

Gaps this RFC still carries alone: swarmctl rebalance churn (§6 - zero mentions in the PR diff, and now a live exposure rather than a hypothetical one), the vars-secret naming above, the Phase 0 latency measurement (skipped), and the PHI decision (risk 1, still open for production).

What the first deploy taught us

Staging took five attempts (13231, 13232, 13235, 13237, 13239). All three real failures were in the CIFS mount, and neither document predicted any of them:

• addr= is load-bearing. Docker’s local driver mounts via the kernel directly rather than shelling out to mount.cifs, and the kernel cannot resolve hostnames. Docker resolves addr= and passes the IP through. Without it the mount fails on name resolution.
• iocharset=utf8 must be dropped. The GCP worker image’s kernel ships no nls_utf8 module, so the option inherited from the Azure fstab line fails the mount with “can not access a needed shared library”. SMB3 is UTF-16 on the wire regardless.
• mount.cifs is not required - only the cifs kernel module, which the current worker image already has. This RFC previously called for adding cifs-utils to the worker bootstrap; that was wrong, and it followed from assuming a host mount rather than a driver mount.

And one prediction that did not hold: this RFC rated the SMB dialect as the likeliest first-run failure and recommended vers=3.1.1,seal before any cross-cloud run. Staging mounted successfully on vers=3.0. secureTransferOnly does require an encrypted channel, but SMB 3.0 satisfies it (AES-128-CCM) and the kernel negotiates encryption from the server’s requirement without seal being stated. The 3.1.1 recommendation stands only as a throughput improvement (AES-128-GCM), not a correctness prerequisite - see §5.

Gaps and Proposed Changes

1. Config and Secret Delivery

Problem. Both /root/.vars and the provisioner key arrive as host bind mounts (the key is the host’s /etc/gcp/service-account.json, mounted read-only at /etc/gcp/provisioner-key.json, which is where fetch_gcp_secrets.ps1 looks by default). A bind mount whose host path is missing does not fail the task — Docker creates an empty directory and starts the container. The observable failure is therefore not “mount missing” but a confusing downstream crash: substitute_variables_json.ps1 finds no vars and no secrets, leaves $(SentryDsn) unresolved, and the app dies on Sentry: Invalid URI.

Change 1a — the vars file becomes a Swarm config. Raft distributes configs to whichever node runs the task, which is exactly the property needed. The pipeline step that SCPs .envs/.<tenant>-<env>/ to the manager stays as-is; only the compose changes:

    configs:
      - source: worker_vars
        target: /root/.vars/.interfacetaskworker
        mode: 0400

configs:
  worker_vars:
    # Explicit name: Swarm prefixes stack-scoped configs and the total is capped
    # at 64 chars. Configs are immutable, so the tag makes each deploy a new object.
    name: "itw-${TENANTENVIRONMENT}-vars-${VARS_HASH}"
    file: "./.vars/.${TENANTENVIRONMENT}/.interfacetaskworker"

Version the name on a content hash, not the image tag. Configs and secrets are immutable, so a changed vars file needs a new object name. Keying on ${TAG} is not enough: flipping Worker.OwnedInterfaceTypeIds without a code change redeploys the same tag and collides. The deploy script should compute VARS_HASH as a short digest of the vars file, and prune itw-${TENANTENVIRONMENT}-vars-* objects no longer referenced by a service.

Versioning is also what avoids a stack teardown. With a fixed name, an in-place update whose content changed fails with “secret is in use”, which forces docker stack rm before every deploy - the shape PR 2423 takes, whose driver comments that remove-then-deploy is load-bearing. It is load-bearing for a fixed name. With a versioned name the service simply references a new object and rolls, so the deploy keeps its rollback target and does not tear down the stack’s networks and secrets. For a queue worker the teardown is not free: it kills the in-flight task and orphan-resets its row.

Set the name explicitly regardless, because the stack-scoped default breaks at scale. Swarm caps config and secret names at 64 characters and prefixes stack-scoped ones with <stack>_. That fits for core (56 chars), but the pipelines are per-tenant and six tenant/environment pairs already exceed the cap:

71  healthallianceplan-staging-interfacetask-stack_interfacetaskworker_vars
68  healthallianceplan-prod-interfacetask-stack_interfacetaskworker_vars
67  identityserver-staging-interfacetask-stack_interfacetaskworker_vars
66  networkhealth-staging-interfacetask-stack_interfacetaskworker_vars
65  transamerica-staging-interfacetask-stack_interfacetaskworker_vars
65  abilishealth-staging-interfacetask-stack_interfacetaskworker_vars

Latent today, since the worker’s release pipelines list only core, but it lands the first time the worker follows the legacy service onto another tenant. Same fix as above.

Change 1b — drop the provisioner key mount; authenticate hop 1 from the metadata server. fetch_gcp_secrets.ps1 does a two-hop fetch: a provisioner identity reads ha-<tenant>-vault-key from the vault-keys project, then that vault SA reads ha-<tenant>-vault from the vault-secrets project. Only hop 1 needs the key file, and on a GCP swarm node the node SA already holds that grant. Add a metadata-server path, keeping the key file as the fallback so one image works on both clouds during the parallel run:

function Get-MetadataAccessToken {
    $uri = 'http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token'
    (Invoke-RestMethod -Uri $uri -Headers @{ 'Metadata-Flavor' = 'Google' }).access_token
}

# Prefer a mounted provisioner key (Azure); fall back to the node's own identity
# (GCP swarm), where the compute SA holds secretAccessor on the vault-keys project.
$provisionerToken = if (Test-Path $ProvisionerKeyPath) {
    $key = Get-Content $ProvisionerKeyPath -Raw | ConvertFrom-Json -AsHashtable
    Invoke-WithRetry -Description 'provisioner auth' -ScriptBlock { Get-GcpAccessToken -ServiceAccountKey $key }
} else {
    Invoke-WithRetry -Description 'metadata auth' -ScriptBlock { Get-MetadataAccessToken }
}

Note the direction of the fallback. The current script exit 1s when the key file is absent; preserve a hard failure when both paths fail. There must be no silent degradation to an unauthenticated fetch.

Adopted, and validated by the staging deploy. PR 2423 initially shipped the key as a pre-created cluster secret and moved to this design in 83a32782; the container now authenticates hop 1 as the node SA on GCP and via the mounted key on Azure, so one image serves both clouds. The rejected alternative - a long-lived private key in Raft, materialising on every node that runs the task, for an identity the node already has - is no longer on the table.

One ordering dependency this RFC missed. The fallback lives in the base-app image (haap-interfacetaskworker), not the app image, so the base image has to be rebuilt and pushed before the first app build that a swarm deploy consumes. Otherwise the container looks for a key file that is not mounted and the fetch hard-fails at startup. The PR captures this as its prerequisite 2; it belongs in any similar port of a base-image-mediated contract.

2. Placement and Networking

node.labels.os == linux is an artifact of HA’s mixed-OS two-node swarms; GCP nodes carry only role=manager, set in manager-startup.sh.tftpl. The frontend overlay is likewise created by HA’s bootstrap and does not exist on the GCP clusters.

The worker is headless, so it needs no shared overlay - only egress. Note that dropping the networks block does not leave the service with no network: docker stack deploy still creates a stack-scoped <stack>_default overlay and attaches the service to it. That is fine (it is isolated per stack and costs nothing), but the accurate statement is “no cluster-wide overlay dependency”, not “no overlay”. Declaring a stack-local network explicitly, as PR 2423 does, makes the same outcome legible. Either way, constrain on role:

      placement:
        constraints:
          - node.role == worker

Keep replicas: 1. The single-replica constraint is a correctness requirement, not a capacity choice: PollingWorker.ExecuteAsync calls ResetStartedTaskQueueItems(_owned) on startup, and that reset is scoped to owned InterfaceTypes but not to the current process, so a second replica would clear a row the first is mid-processing.

3. Image Pull

The GCP swarm deliberately avoids --with-registry-auth because of moby/moby#31063: the token baked into the service spec has a roughly 1-hour TTL, and a MIG worker created more than an hour after the last deploy receives an already-expired credential and never pulls. Instead every node runs docker-gcr-proxy on 127.0.0.1:7676 and injects a locally-refreshed token.

The constraint: the proxy’s upstream is hardcoded to https://gcr.io in proxy.go. The worker image lives at us-central1-docker.pkg.dev/the-helper-bees/healthalignpms/interfacetaskworker, which the proxy cannot reach. Every other swarm stack resolves images as ${IMAGE_REGISTRY:-gcr.io}/the-helper-bees/....

Recommendation: also push the worker image to gcr.io/the-helper-bees/healthalignpms/interfacetaskworker by adding a second tag to the Docker@2 step in build.yml, and parameterise the compose reference:

    image: "${IMAGE_REGISTRY:-us-central1-docker.pkg.dev}/the-helper-bees/healthalignpms/interfacetaskworker:${TAG}"

with the swarm deploy exporting IMAGE_REGISTRY=docker-gcr-proxy:7676. This leaves the Azure deploy on its current path unchanged (no IMAGE_REGISTRY set, so the AR default applies), which matters during the parallel run.

The alternative, teaching the proxy a configurable Artifact Registry upstream, is the cleaner long-term fix and probably a small change, but it touches a component every stack on both clusters depends on for image pulls. It should be its own change, not a rider on this migration.

⚠️ Do not simply keep --with-registry-auth. It succeeds on the first deploy and fails silently on the next scale-out, which is the worst possible failure signature.

4. Deploy Path

Three things break at once: the transport (ADO SSH service connections use a static key, while GCP managers set enable-oslogin=TRUE and block-project-ssh-keys=TRUE and are IAP-only), the interpreter (no pwsh on GCP nodes), and parts of the script that only made sense on a private VM.

Recommendation: keep ownership with the HA ADO pipelines, and drive the manager from the agent pool over IAP. The existing deployFromAgentPool: true shape already runs the deploy from ADO Deploy Helper rather than a VM agent, so only the step template changes — replacing deploy-over-ssh.yml’s SSH@0 task with the ephemeral-key plus OS Login plus IAP sequence that gc_deploy_to_swarm.yml already proves out:

gcloud auth activate-service-account --key-file="$SA_KEY"
ssh-keygen -t ed25519 -N '' -f "$KEY" -C "ado-itw-deploy"
gcloud compute os-login ssh-keys add --key-file="$KEY.pub" --ttl=60m

# Stage the driver, the stack file, and the tenant's vars together. The compose
# file resolves the vars secret as ./.vars/.<tenant>-<env>/.interfacetaskworker
# relative to itself, so all three must land in the same directory.
STAGE=$(mktemp -d)
cp compose/deploy_interfacetaskworker_gcpswarm.sh \
   compose/deploy_interfacetaskworker.gcpswarm.yml "$STAGE/"
mkdir -p "$STAGE/.vars"
cp -r "$ARTIFACT/.envs/.${TENANT}-${ENV}" "$STAGE/.vars/"

# --ssh-key-file is required: without it gcloud uses ~/.ssh/google_compute_engine,
# not the ephemeral key just registered with OS Login. Copy the directory itself,
# not a glob -- "$STAGE/*" would silently skip .vars, which is a dotfile.
gcloud compute scp --tunnel-through-iap --ssh-key-file="$KEY" --zone "$ZONE" \
  --recurse "$STAGE" "$MGR:~/interfacetaskworker"
gcloud compute ssh --tunnel-through-iap --ssh-key-file="$KEY" --zone "$ZONE" "$MGR" \
  --command "sudo bash ~/interfacetaskworker/deploy_interfacetaskworker_gcpswarm.sh core prod $SHORT"

gcloud compute ssh --tunnel-through-iap subsumes hb-ansible’s explicit ProxyCommand; the manager is addressed by instance name and zone, not IP.

The staging step is the easiest part to get subtly wrong. The vars file arrives from the build artifact as .envs/.<tenant>-<env>/ but the compose file expects it at .vars/.<tenant>-<env>/, so the transfer both renames the parent directory and has to keep the tenant directory intact. Getting it wrong does not fail the copy - it fails the driver’s own missing ./.vars/... precondition, which is the good outcome. The bad outcome is a partial copy that leaves a stale vars file from an earlier deploy in place, which deploys successfully with the wrong owned-type set.

Port the deploy script to bash, and while porting, delete three single-VM artifacts:

Keep the convergence wait (IsStackRunning plus deadline) — it is the deploy’s only real health signal.

New IAM required (Terraform, hb-infra). The deploying identity needs three grants, not two: roles/iap.tunnelResourceAccessor on the swarm env project, roles/compute.osAdminLogin (not plain osLogin - the driver needs sudo for docker), and roles/iam.serviceAccountUser on the managers’ runtime service account (google_service_account.swarm_manager). The third is easy to miss and is not optional: OS Login checks iam.serviceAccounts.actAs on a VM’s attached service account at every connection, precisely so SSH access cannot be used to escalate into that account. roles/compute.osAdminLogin already carries the instance and project discovery permissions gcloud compute ssh needs. Which identity is an open question (see Risks): the ADO agent already holds a GCP identity for pushing to Artifact Registry, and there is separate in-flight work on keyless ADO-to-GCP auth worth aligning with rather than minting another long-lived key.

Why not a plain ADO SSH service connection to a manager, the shape PR 2423 originally took before moving to this sequence in 83a32782. It is not impossible, but it carries four prerequisites that are easy to under-count - the PR now folds the first two into its own prerequisite 1:

1. The POSIX username is derived, not chosen. With enable-oslogin=TRUE the account is sa_<uniqueId> for a service account (<user>_<domain>_<tld> for a human), so the endpoint’s username has to match what OS Login mints.
2. Two grants, not one. roles/compute.osAdminLogin for passwordless sudo and iam.serviceAccountUser on the manager’s runtime SA - see the comment block in swarm_manager/main.tf.
3. A firewall rule for :22 from the ADO agent’s egress IP to the manager’s external IP. The swarm’s SSH posture is OS Login plus IAP, and the tag-scoped rules live in the separate thehelperbees/gcp-networks repo - a cross-team change, not a portal click. This is the prerequisite most likely to be missed, because everything else can be staged before it and the failure looks like a hung connection.
4. It pins one manager. Any manager can serve docker stack deploy, so leadership is irrelevant, but the endpoint becomes a deploy SPOF and its host key changes whenever the image-upgrade runbook rebuilds that node.

The IAP path avoids 3 and 4 outright and makes 1 and 2 explicit in the pipeline rather than latent in a service-connection definition.

5. The Azure Files Share

This is the only genuinely hard gap, and it is narrower than it first appears. Only two of the worker’s modules touch the media path — X12837FileWriter (type 60) and ChangeHealthcareImportResponseFiles (type 7). Prod currently owns types 60,55,47, so exactly one type is affected there.

The path is a CIFS mount of //hasftp.file.core.windows.net/ha-sftp-share. The storage account is permissive on the network side — publicNetworkAccess: Enabled, networkRuleSet.defaultAction: Allow, no IP rules (verified live 2026-08-31) — so no allowlisting work is required. What is required:

Mechanism: a per-node CIFS Docker volume, not a host mount. This RFC originally proposed adding the mount to worker-startup.sh.tftpl. PR 2423’s approach is better and is adopted here: declare a local-driver volume with CIFS driver_opts in the stack file, with credentials fetched at deploy time from Secret Manager using the manager’s own metadata identity, so no credential rides through ADO.

volumes:
  hasftp:
    driver: local
    driver_opts:
      type: cifs
      device: "//hasftp.file.core.windows.net/ha-sftp-share/HealthAlign/${ENVIRONMENT}"
      # addr= and the absence of iocharset are both load-bearing - see below.
      o: "addr=hasftp.file.core.windows.net,username=${HASFTP_USERNAME},password=${HASFTP_PASSWORD},file_mode=0777,dir_mode=0777,vers=3.0"

Two properties make this the right shape. Docker establishes the mount on whichever node runs the task, so nothing has to be pre-mounted on an autoscaled pool; and a failed mount fails the task rather than silently presenting an empty directory the way a bind mount over a missing path does. That closes the silent-failure hole this RFC previously had to warn about: X12837FileWriter can no longer report success while writing into the void.

What still has to be true:

1. The cifs kernel module - but not cifs-utils. Docker’s local driver issues the mount syscall itself rather than shelling out to mount.cifs, so only the kernel module is needed and the current worker image already ships it. No swarm_worker bootstrap change is required. (An earlier revision of this RFC called for adding cifs-utils; that followed from assuming a host mount and was wrong.)
2. addr= in the options. Because the kernel performs the mount directly, it cannot resolve the hostname in device:. Docker resolves addr= and passes the address through. Omitting it fails the mount on name resolution - one of the three real staging failures.
3. No iocharset=utf8. The GCP worker kernel ships no nls_utf8 module, so the option inherited from hasftp.sh’s in-region fstab line fails the mount with “can not access a needed shared library”. SMB3 is UTF-16 on the wire regardless, and the default codepage covers the ASCII filenames this share exchanges.
4. Encryption is mandatory but vers=3.0 satisfies it. The hasftp account has secureTransferOnly: true (verified live 2026-08-31), so Azure Files refuses an unencrypted channel. An earlier revision of this RFC concluded that the inherited vers=3.0 was therefore the likeliest first-run failure and prescribed vers=3.1.1,seal. That was wrong: staging mounted cleanly on vers=3.0, because SMB 3.0 carries AES-128-CCM and the kernel negotiates encryption from the server’s requirement without seal being stated. Moving to vers=3.1.1 buys AES-128-GCM and better throughput on the internet path, which is worth doing if the 837 volumes grow - as an optimisation, not a prerequisite.
5. Credential and option changes must change the volume name. Docker volumes keep the driver_opts they were created with, so a fixed name means any change - a cred rotation, a mount-option fix - silently keeps mounting with stale options on every node that already materialised it. PR 2423 solves this by content-addressing the volume name from a fingerprint of the compose file plus the creds (14d35e51), so a change forces every node to materialise fresh and orphaned definitions (which hold no data) die with node recycling. This is the same technique §1a asks for on the vars object, and it is strictly better than this RFC’s original advice to remove the volume by hand after a rotation.

⚠️ The CIFS credential is readable from the node. Docker’s local driver has no credentials-file equivalent, so username=/password= are stored verbatim in the volume’s metadata and returned by docker volume inspect. Anyone who can reach the Docker socket on a worker can read the storage-account key. Docker-socket access is already root-equivalent, so this widens exposure from “root on the node” to “root on the node, plus anything that can query the daemon” rather than creating a new class of access - but it is a real difference from the Azure VMs, which keep the key in a 0600 root-owned credentials file outside Docker. The alternative that avoids it is the host-mount-plus-bind-mount shape this RFC originally proposed, which trades the exposure for the autoscaling and silent-failure problems §5 exists to solve. Worth an explicit decision rather than an accident; at minimum, rotate the key on any node compromise and keep it out of any support bundle that captures docker volume inspect.

No storage-firewall work is required, contrary to a common assumption. publicNetworkAccess: Enabled, networkRuleSet.defaultAction: Allow, ipRules: [] — there is nothing to allowlist for the swarm NAT egress IPs, and GCP does not block outbound 445 (it blocks 25). PR 2423 lists this as a prerequisite; it is already satisfied.

That is a statement about what the migration has to do, not an endorsement of the posture. The share is reachable from any network that holds the storage key, and after this migration it is on the PHI path for two clouds instead of one. Restricting it to the swarm Cloud NAT egress IPs plus the existing Azure consumers is a small, well-understood change (defaultAction: Deny plus ipRules), and a private endpoint is the stronger form. Neither is required to ship, and neither is in this RFC’s scope - but shipping without recording the choice would let the migration quietly inherit an open door. Tracked as risk 10.

Even with a good mechanism the ordering question stands for production. Staging settled it by events - it moved all 20 types at once and debugged the mount in place - but prod owns type 60, so the same choice recurs there:

Splitting types across two worker deployments is safe. Two workers with disjoint owned sets never reset each other’s rows, because ResetStartedTaskQueueItems(int[] owned) filters on the owned set. The legacy service’s ExcludedInterfaceTypeIds becomes the union of both workers’ owned sets; its own reset already skips worker-owned types via BuildExcludedTypeClause. The mechanical cost is a second vars file and a distinct stack name, since Worker.OwnedInterfaceTypeIds is one CSV per deployment.

6. Rebalance Churn

New exposure the Azure box does not have: when the autoscaler adds a worker, swarmctl force-updates services so Swarm spreads tasks onto it. For a single-replica order: stop-first service, a force-update is a restart mid-task — the queue row is orphan-reset and the task re-runs. Module side effects are already required to be idempotent (a single service is at-least-once by design), so this is a churn and duplicate-side-effect concern rather than a corruption one.

swarmctl supports an exclusion list, and as of infrahive#1358 it is wired end to end: rebalance_excluded_services in src/services/swarmctl/envs/<env>.yaml, joined into REBALANCE_EXCLUDED_SERVICES by the deploy script and validated twice: src/scripts/deploy_swarmctl/main.py rejects the list at deploy time unless it is a list of strings with at most 64 entries, no duplicates, and no entry that is empty, has surrounding whitespace, or contains a comma; swarmctl then rejects a malformed path.Match pattern at boot. Entries are path.Match globs against the full service name (<stack>_<service>), so *-interfacetask-stack_worker covers every tenant and environment in one line. A bare worker matches nothing and the exclusion would appear configured while every rebalance still restarts the task. Autoscaler scale-in drain and MIG autohealing still apply, which is the behaviour we want.

Corrected 2026-09-03. An earlier revision of this RFC said to set REBALANCE_EXCLUDED_SERVICES directly in src/services/swarmctl/compose.yml, where it was then unset, using one full service name per tenant. That was accurate when written and is now wrong on both counts. infrahive#1358 moved the list into the per-environment files and made it glob-capable — prompted by prod rebalance passes on 2026-08-30 and 09-01 restarting 14 per-app redis brokers and silently dropping their queued Celery work, which is this failure class arriving on a different service first. Both environments now carry *_redis.

PR 2423 does not address this — a diff-wide search for rebalance returns nothing — so it is the one item in this RFC with no counterpart in the implementation.

Proposed Compose File

Phase-1 shape: no file share, no frontend, config-based vars, metadata-server secrets.

services:
  worker:
    # IMAGE_REGISTRY is set to docker-gcr-proxy:7676 by the swarm deploy; unset on
    # the Azure path, which keeps pulling directly from Artifact Registry.
    image: "${IMAGE_REGISTRY:-us-central1-docker.pkg.dev}/the-helper-bees/healthalignpms/interfacetaskworker:${TAG}"
    environment:
      - TZ=America/Chicago
      - ENVIRONMENT=${ENVIRONMENT}
      # Reuses each tenant's existing legacy vault: VAULT_APP selects
      # `ha-<tenant>-vault` (+ `-vault-key`); ENVIRONMENT selects the n/p project.
      - VAULT_APP=ha-${TENANT}
      # Ownership is NOT set here — this file is tenant/env-agnostic. Each
      # deployment sets Worker.OwnedInterfaceTypeIds in its vars config below.
    configs:
      - source: worker_vars
        target: /root/.vars/.interfacetaskworker
        mode: 0400
    deploy:
      # Single replica is a correctness requirement, not a capacity choice: the
      # startup reset is scoped to owned types but not to this process.
      replicas: 1
      placement:
        constraints:
          - node.role == worker
      restart_policy:
        condition: any
        max_attempts: 1
        window: 120s
        delay: 5s
      resources:
        limits:
          cpus: "2.0"
          memory: 4G
      update_config:
        parallelism: 1
        delay: 10s
        failure_action: rollback
        order: stop-first

configs:
  worker_vars:
    # VARS_HASH is a short digest of the vars file, computed by the deploy
    # script: the tag alone collides when only the vars content changed (§1a).
    name: "itw-${TENANTENVIRONMENT}-vars-${VARS_HASH}"
    file: "./.vars/.${TENANTENVIRONMENT}/.interfacetaskworker"

Removed relative to today: the /root/.vars and provisioner-key bind mounts, the /data bind mount, the frontend network, and the os == linux constraint.

Extending to IdentityServer

Status: deployed to staging (healthAlignPMS#2487, ADO run 14383, 2026-09-04); not yet attempted in production. The two load-bearing differences below now have runtime evidence behind them; the worker’s numbers still do not transfer.

IdentityServer is the natural second candidate — same repo, same Linux base image, same vault contract, same ADO release shape — and a much less forgiving one. The worker is a headless single-replica batch poller with no ingress: if a deploy goes wrong the queue waits. IdentityServer is three services (caddy, cloudflared, identityserver), four bind mounts, a Cloudflare tunnel, and the auth service, so a deploy that goes wrong is an outage.

What transfers unchanged

All six deployment-contract changes in Gaps and Proposed Changes apply as written: vars as a Swarm object rather than a bind mount, metadata-server auth instead of the provisioner key (§1), node.role == worker (§2), the gcr.io push for the proxy (§3), the IAP deploy transport (§4), the content-addressed CIFS volume with addr= and no iocharset (§5), and the rebalance question (§6).

Two prerequisites are already satisfied from the worker rollout and need no repeat: the ADO agent IAM trio (infrahive #1351, since relocated by #1355) and the managers’ access to hasftp-cifs-creds (infrahive #1353). That is the first real evidence the per-environment setup cost is one-time rather than per-service.

Two differences that are load-bearing

1. No stack teardown, so versioned secret names stop being optional. The worker’s driver does remove-then-deploy because Swarm secrets are immutable and a fixed name fails “secret is in use” — acceptable for a batch poller, an outage for auth. IdentityServer’s compose is tuned for order: start-first with monitor windows sized around cloudflared’s tunnel handshake, so the deploy has to be a rolling update and the secrets have to roll with it. Both secret names are therefore content-addressed from their own contents.

This is the same argument §1a makes for the worker, where it was a preference the implementation declined. Here it is a constraint. Worth noting for the next service: the versioned-name approach generalises and the teardown approach does not.

2. frontend cannot simply be dropped. For the headless worker, removing the external cluster-wide frontend overlay cost nothing (§2). For IdentityServer it is load-bearing: caddy reaches the app across it by fully-qualified Swarm name. The draft makes that network stack-scoped rather than creating a cluster-wide frontend on the GCP swarm, which would put unrelated tenants’ stacks on a shared overlay on a shared cluster. internal keeps its existing role and scope, so the isolation rule in the app repo’s AGENTS.md (cloudflared uses the short name caddy, so its network must be stack-scoped) is preserved rather than collapsed.

This was the highest-risk item in the extension and the thing a staging run had to prove. It depends on caddy’s baked config resolving its upstream by a name that works on a stack-scoped overlay. If it resolved something else, routing would break quietly rather than failing the deploy — the same silent-failure shape §5 worries about, in a different place. Proven across three runs. Run 14379 showed the resolution itself — cloudflared’s entrypoint logged caddy DNS resolved and origin chain ready — but that run then failed on the tunnel credential’s file mode, so it is evidence for the overlay, not for the path end to end. Runs 14381 and 14383 served the deployed build at auth.myhaapp.com through the swarm’s tunnel; 14383 also passed the driver’s own health check.

Smaller deltas

The tunnel being outbound-only is the piece of luck in this extension: it means no published ports and no routing-mesh surgery on the shared cluster, which is what would have made a web-facing service genuinely hard.

The pipeline wiring decision

configure-cloudflared.yml is shared across webapp, identityserver and webappcore, and its transport is hardwired to CopyFilesOverSSH@0 with a /DevOps target while staging three artefacts (tunnel credential, certificate, config.yml). The PR did not parameterise it. A swarm-specific deploy-swarm.yml step reuses the artifact’s pull_cloudflare_tunnel_from_gcp.sh — the same script the Azure path runs, on the same agent identity, against the same secrets project — and ships the result with gcloud compute scp, so only the copy transport differs and no new secret access is introduced. It stages only tunnel.json, because that is all the stack mounts: config.yml is baked into the cloudflared image and cert.pem is not mounted by this stack. The release template forks on the same deployTarget parameter as the worker’s, defaulting to azure-vm in both environments.

What the staging runs taught us

It took four attempts (ADO runs 14370, 14379, 14381, 14383). The transport, the stack deploy, the proxy pull and the compose shape were right first time; every failure was in a place the worker’s rollout could not have exercised, and each generalises to the next service.

1. §1 has to be applied per base image, not once. The metadata-server fallback lives in fetch_gcp_secrets.ps1, and each haap-* base image carries its own copy of that script. healthAlignPMS#2423 patched the haap-interfacetaskworker copy; haap-identityserver still hard-failed on Provisioner key not found (run 14370). The fix needs a base-image rebuild before the app build that a swarm deploy consumes, exactly as the worker’s prerequisite 2 did. Any further service needs the same audit of its base image.

2. Swarm secrets are root-owned; non-root images need 0444. The tunnel credential was declared mode: 0400, mirroring the vars secret, but the cloudflared image runs as USER cloudflared — a useradd --system account with no fixed uid — so it could not read it (run 14379). A uid: pin would rot on the next base-image rebuild; docker’s default 0444 is correct because the secret tmpfs is private to the container. The worker never hit this because its container runs as root.

3. Port health checks for fidelity, not shape. The endpoint serialises camelCase ({"version": ...}); the Azure PowerShell check tolerates that because PowerShell property lookup is case-insensitive, and the bash/python port did not (run 14381 — the build was live and correct, only the check misread it).

4. A changed base image gets a new tag. Both rollouts first rebuilt haap-<app>:10.0.0 in place to carry the §1 fallback. Review asked for the honest version: the base images are now 10.0.1 (build_linux_docker_images.ps1 and each app Dockerfile’s FROM), so the change is a new artefact rather than a rewrite of an existing one — at the cost of an ordering rule: the base-image build must run before the next app build, or the app build fails at FROM.

The dual-instance hazard is also sharper here than for the worker: while both an Azure and a swarm IdentityServer are connected to the same tunnel, Cloudflare load-balances between them, so a public health check is a coin flip and two auth instances run against one database. The Azure path tolerates that overlap only for the seconds between start-first and the old task stopping.

What two replicas need

The natural next step is two replicas per service on the swarm, and healthAlignPMS#2507 (HB-9035) drafts it: replicas: 2 with max_replicas_per_node: 1 on all three services, so losing any one worker leaves the swarm path serving (cloudflared runs two connectors on the one tunnel; caddy and identityserver are reached through their VIPs). Two details are worth recording:

• update_config.parallelism stays 1, unlike django_homealign’s UPDATE_PARALLELISM=2: with start-first, one untouched replica keeps serving through the other’s monitor window, so a bad build is caught by the rollback canary while capacity never drops below one good task. The cost is one extra delay + monitor cycle per service.
• The driver’s public health check changes meaning. With the Azure instance on the same tunnel, a probe answering with a different version is no longer a failure, only another connector; the check retries until it sees the expected build once, and a new per-replica verdict (docker service ps shows every running task on this build’s tag) carries the correctness burden instead.

The compose change is the easy part. What actually gates it is Open Item 5: the application keeps per-process state (token keys, the Data Protection key ring, an in-memory claims handoff) that no scheduler setting can share.

Sizing followed the same rule as everything else here: measure, then set. healthAlignPMS#2509 adds reservations and right-sizes limits from an 18-hour docker stats sample of the production Azure stacks (one instance of each service carrying all traffic): identityserver 458 MiB p50 / 521 max and 0.09 core p95 / 0.92 max; caddy and cloudflared under 40 MiB and 0.12 core. Reservations are set at roughly p99 (identityserver 0.25 CPU / 768M), limits at 3–4× the observed max — the inherited 16G identityserver limit was ~30× actual use. Production runs the three IdentityServer services with no memory limit, a deliberate trade-off rather than a safe default: a hard limit turns a leak into an OOM-kill of the auth container at a moment of Docker’s choosing, while no limit lets it run until the node is under pressure — at which point the kernel’s OOM killer picks the largest process on the node, which is as likely to be a neighbouring tenant’s task as IdentityServer itself. What contains that on the shared swarm: the reservations stay in place (768M for identityserver), so the scheduler still accounts for the footprint; max_replicas_per_node: 1 keeps the two replicas on different nodes; and the measured working set is ~0.5 GB on 32 GB n2-highmem-4 workers. The alerting is honest rather than protective: swarm_node_memory_high pages once a node has been above 85% memory for 10 minutes, which is advance notice of a leak, not a guard against the kill; swarm_service_oom reports a container OOM kill after the kernel has already chosen; and swarm_container_mem_near_limit cannot fire for these services at all, because it is a ratio to the container limit and there is none — so the per-container early warning that a limit would have bought is the thing this trade-off gives up. Staging keeps the measured ceilings so a regression shows there first. If a leak ever shows up in staging, the answer is a production limit above the observed peak, not a bigger node. The worker’s limits stay at 2 CPU / 4G because the sample caught almost no batch activity, and a batch processor’s peak is what a limit is for. Values are ${VAR:-default} in the compose files with a per-environment override block in each driver.

Phased Rollout

Handoff within each phase follows the established order — deploy legacy-excludes-first, drain in-flight rows to zero, then start the worker — because exclusion stops the legacy service claiming new tasks but does not stop it finishing current ones.

What actually happened. PR 2423 collapsed phases 2 through 4 for staging - deployTarget defaulted to gcp-swarm there at the time, and staging owns both file-touching types - so the first run exercised the new transport, the GCR pull, the metadata-server secret fetch, and cross-internet CIFS at once. It took five attempts (ADO runs 13231 through 13239), and all three real failures were mount options (see What the first deploy taught us). Phase 0 was skipped, so there is still no cross-cloud latency measurement.

Production then did the same thing: run 13460 moved all three owned types at once, including type 60, rather than taking Phase 3’s file-types-last split. Its two failures were both prerequisites rather than mount options - the ADO agent IAM trio (run 13450) and the manager’s access to hasftp-cifs-creds (run 13458) - because staging had already absorbed the mount debugging. Both were hand-applied and are now codified in infrahive#1351 and infrahive#1353.

Read honestly, the collapsed rollout argues for neither position outright. It cost four failed staging runs on an environment where failed runs are cheap, and it surfaced three findings a phased approach would have hit later and one at a time. What it did cost is the Phase 0 measurement: production shipped without a cross-cloud latency number, so the assumption this RFC most wanted evidence for in advance is now something to measure after the fact rather than a gate that was passed. Taking it against either environment is still close to free and worth doing.

Rollback at every phase is a vars change plus a redeploy: hand the types back to the Azure deployment and shrink the GCP deployment’s owned set. No data migration is involved at any point, because the database never moves.

Risks and Open Questions

| 8 | Swarm config/secret names exceed the 64-char cap on six tenant/environment pairs once the worker moves past core | Explicit name: on the object (§1a), which is also what enables the rolling update | | 9 | Cross-internet SMB has never been exercised. Closed. Exercised by ADO run 13239 on vers=3.0; the dialect concern was a throughput question, not a correctness one (§5 item 4) | Retained as a record of a wrong call. The real mount hazards were addr= and iocharset, which this RFC did not anticipate | | 10 | The hasftp share is reachable from any network holding the storage key, and this migration puts it on a second cloud’s PHI path | Restrict to the swarm NAT egress IPs plus existing Azure consumers (defaultAction: Deny + ipRules), or a private endpoint. Not required to ship; record the decision either way | | 11 | The CIFS credential is exposed via docker volume inspect on any worker that materialises the volume (§5) | Docker-socket access is already root-equivalent, so treat as exposure widening rather than a new class. Rotate on node compromise; keep docker volume inspect out of support bundles |

Effort

Compare against the HA consolidation RFC’s Path 2 at 5 to 6 weeks, which buys the Windows tier this workload does not need.

Alternatives Considered

• Leave it on Azure. Zero cost, and legitimate if the answer to risk 1 is “no HA workloads on the shared swarm”. The worker keeps a statically-sized VM with no autohealing and no graceful drain — the same gap the consolidation RFC describes for the whole HA tier.
• Wait for the dedicated HA GCP swarm (Path 2b). Correct if the PHI-isolation answer forces it; this work then becomes the first stack on that cluster rather than an addition to the shared one. Most of §1 through §4 is reusable either way.
• Deploy via AWX and gc_deploy_to_swarm.yml. Reuses a proven IAP path, but that playbook is heavily Django-shaped (migration detection, celery scaling, Caddy, canary) and would need real surgery. A short bash script driven from the existing ADO pipeline is smaller and keeps HA deploys in one place.
• Ship the provisioner SA key to GCP workers as a Swarm secret. Works, but puts a long-lived private key in Raft to obtain an identity the node already has.

Verification Checklist

Per phase, before declaring done:

• The startup log reads InterfaceTaskWorker starting. Owned InterfaceType ids: [...] with exactly the intended set
• The legacy service’s startup Excluded [...] equals the union of all worker deployments’ owned sets
• docker service ps shows the task on the intended node class, 1/1, with no restart loop
• Papertrail: the Dispatching count reconciles with Completed S plus F after a settle window, using absolute --min-time and --max-time
• No secret placeholder survives substitution (grep the running container’s appsettings.json for $()
• A deliberate MIG scale-out does not restart the worker task (the swarmctl exclusion holds)
• A worker created more than one hour after the deploy still pulls the image (the --with-registry-auth regression test)
• Phase 4 only: a deliberately broken CIFS mount fails the task rather than writing into an empty directory
• The stack’s config and secret object names are under 64 characters for the tenant being deployed
• The CIFS mount is present on the node running the task (findmnt -t cifs -o TARGET,SOURCE,OPTIONS) and the session is actually encrypted - findmnt reports the options that were requested, not what was negotiated, so confirm against /proc/fs/cifs/DebugData on that node

References

• healthAlignPMS#2423 — the pipeline and compose implementation of this design (HB-8825)
• ADO run 13239 — the first successful core-staging deploy onto the GCP swarm
• HomeAlign Swarm Consolidation: Azure-Native vs GCP Migration — the parent decision and its cost model
• Docker Swarm Consolidation — HB-8200, the GCP cluster this targets
• Swarm architecture overview
• docker-gcr-proxy — why --with-registry-auth is avoided
• swarmctl — rebalance and drain
• App Service Accounts — the no-key-file secret-fetch pattern
• Docker Swarm Outbound IP Audit — NAT egress IPs and the Azure SQL firewall inventory
• InterfaceTaskService/RFC/0001-dotnet-linux-migration.md (healthAlignPMS) — the worker’s own design, module status, and the at-least-once/orphan-recovery follow-up