RFC: Deploy the InterfaceTaskWorker to the GCP Swarm

Table of Contents

• Summary
• Open Items
• Why This Workload
• Current State
  • The Azure Runtime Today
  • The Runtime Contract
• What the GCP Swarm Already Provides
• Relationship to healthAlignPMS#2423
  • What the First Deploy Taught Us
• Gaps and Proposed Changes
  • 1. Config and Secret Delivery
  • 2. Placement and Networking
  • 3. Image Pull
  • 4. Deploy Path
  • 5. The Azure Files Share
  • 6. Rebalance Churn
• Proposed Compose File
• Phased Rollout
• Risks and Open Questions
• Effort
• Alternatives Considered
• Verification Checklist
• References

Summary

The .NET 10 InterfaceTaskWorker is the first HomeAlign workload that can move onto the consolidated GCP swarm without the Windows worker-pool build-out that the HA consolidation RFC prices at 5 to 6 weeks. It is Linux, headless, single-replica, and its image is already built on ubuntu-latest and pushed to GCP Artifact Registry. The cross-cloud Azure SQL firewall allowlisting it depends on is already in place in both environments.

None of the required work is in the application. It is six changes to the deployment contract, all of which follow from one root cause: the worker today runs on a two-node swarm where the Linux node is simultaneously the Raft manager, the deploy target, and the box the container runs on. Every piece of its configuration arrives as a host bind mount placed there moments earlier by the same deploy. On the GCP swarm the task lands on an ephemeral autoscaled MIG worker, and none of those host paths exist.

The recommendation is a phased move that starts with the InterfaceTypes needing no file-share access, which yields a real cross-cloud latency measurement on production data with a one-line rollback.

Scope note. healthAlignPMS#2423 was authored independently and in parallel with this document and carries the implementation, which is now live on core-staging and core-prod. This RFC is therefore not the implementation plan - it is the decision record for the choices that PR makes, what the first real deploy proved or disproved, and what remains open. See Relationship to healthAlignPMS#2423.

Open Items

The implementation has shipped to both environments. These are the things that have not been settled, kept here as an action list rather than only as rows in the risk table further down.

1. PHI placement on the shared production swarm

Status: open. No approving owner or risk acceptance is recorded anywhere. This is why the Status field above is qualified rather than a clean “Accepted”.

core-prod has been running on the shared production swarm since ADO run 13460, processing 837 claims files with the HA SFTP share CIFS-mounted into the task. The HA consolidation RFC argues specifically against that arrangement in its Path 2a and recommends a dedicated HA swarm (2b) if HA workloads go to GCP. So the placement this RFC implemented is the one the parent RFC advises against, and nothing on the record says that trade was accepted.

It has now been decided three times operationally — a staging default, a prod parameter override, and a prod release — rather than once deliberately. healthAlignPMS#2423 still leaves its Auth / PHI-PII exposure change box unchecked. Parent ticket HB-8824’s framing (“consolidate HA VMs onto a single swarm”) may settle it above the PR, but that is an inference, not a record.

What closes this — one of:

• A named owner accepting shared-swarm placement for HA PHI workloads, recorded here with a date; or
• A decision to move to a dedicated HA cluster (parent RFC’s Path 2b), with a migration date — in which case this RFC’s placement is superseded and §2’s network and placement choices get revisited; or
• A documented finding that the parent RFC’s concern does not apply here (for example that the swarm’s isolation properties are sufficient for this workload class), with whatever review that requires.

Who decides: not the author of this RFC. It needs DevOps plus whoever owns the HA compliance posture.

Until then: production releases default to deployTarget: azure-vm, so reaching the swarm stays an explicit per-run choice rather than something a routine release does. That is a holding position, not a resolution.

2. swarmctl rebalance exclusion

Status: open, and this one is live exposure rather than a decision. Both environments run on autoscaling worker pools with no rebalance exclusion, so every scale-out event is a chance for swarmctl to force-update the single-replica worker mid-task, orphan-reset its queue row, and re-run it with duplicated external side effects. See §6 and risk 6.

Closing it is one line in src/services/swarmctl/compose.yml — REBALANCE_EXCLUDED_SERVICES — using the full Swarm service name (<tenant>-<env>-interfacetask-stack_worker); a bare worker matches nothing and looks configured. Infrahive change, so it does not block the app PR. Owner unassigned.

3. Cross-cloud latency measurement

Status: open. Phase 0 was meant to measure this before anything was committed; it was skipped and both environments shipped, so it is now a post-deployment measurement rather than a gate. Scope and exit criterion are in risk 2. Cheap to take now that core-staging is live and cross-cloud. Owner unassigned.

Why This Workload

The HA consolidation RFC’s Path 2 (GCP migration) carries two costs that do not apply here:

• No Windows tier. Path 2’s dominant line item is a from-scratch Windows MIG with a PowerShell rewrite of the 509-line worker bootstrap. The worker is Linux and joins the existing pool.
• Not the chatty request path. Path 2’s linchpin risk is moving the transactional PMS webapp away from its co-located Azure SQL databases. The worker is a polling batch host — a poll loop on a 60s default interval dispatching discrete tasks — which tolerates tens of milliseconds of added round-trip far better than a per-request web tier. This is closer to django_homealign, which already runs GCP compute against Azure-resident data.

It is also the smallest available blast radius: headless, no ingress, no published ports, no Cloudflare tunnel, no DNS, one replica.

Current State

The Azure Runtime Today

The worker runs on the HA Azure swarm, which is not one cluster but a set of per-WEB-group two-node swarms: one Linux manager plus one Windows worker, joined by hand (see scripts/manual/readme.md). Non-prod is THB-N-WEB3-LINUX, prod is THB-P-WEB5-LINUX.

flowchart LR
  ADO["Azure DevOps
static-key SSH endpoint"]:::ext
  subgraph VM["THB-P-WEB5-LINUX - manager AND runtime"]
    FS["/DevOps/.vars/.core-prod
/etc/gcp/service-account.json
/mnt/ha-sftp-share (CIFS)"]:::data
    C["worker container
replicas: 1"]:::wkr
  end
  SQL["ha-prod1-azsqldb
hACore"]:::data
  ADO -->|"scp vars + pwsh deploy"| VM
  FS -.->|"bind mounts"| C
  C -->|"in-region"| SQL
  classDef wkr fill:#bbf7d0,stroke:#15803d,color:#06281a
  classDef data fill:#ddd6fe,stroke:#6d28d9,color:#241152
  classDef ext fill:#e5e7eb,stroke:#4b5563,color:#111827

The deploy is an ADO SSH service connection into that VM running compose/deploy_interfacetaskworker.ps1, which docker stack deploys compose/deploy_interfacetaskworker.yml. Because manager and runtime are the same box, host bind mounts are a valid configuration channel. That is the assumption the move breaks.

The Runtime Contract

Everything the container needs at start, and where it comes from:

What the GCP Swarm Already Provides

Verified against the code and, where noted, against the live cloud:

The single most expensive-looking prerequisite, cross-cloud database access, is therefore already done.

Relationship to healthAlignPMS#2423

PR 2423 (HB-8825) adds a deployTarget parameter to the worker’s release pipelines, a bash driver, and a deploy_interfacetaskworker.gcpswarm.yml stack file. It reaches the same conclusions as this RFC on the root cause, placement, networking, the image-pull path, the bash driver, and preserving the Azure path - arrived at separately, which is useful corroboration for both.

The PR has since converged on this RFC in two of the three places the two diverged. Current state:

The SSH transport divergence is also resolved: the PR moved to the IAP sequence (83a32782), so §4’s four prerequisites are now the PR’s actual prerequisite 1.

Gaps this RFC still carries alone: swarmctl rebalance churn (§6 - zero mentions in the PR diff, and now a live exposure rather than a hypothetical one), the vars-secret naming above, the Phase 0 latency measurement (skipped), and the PHI decision (risk 1, still open for production).

What the first deploy taught us

Staging took five attempts (13231, 13232, 13235, 13237, 13239). All three real failures were in the CIFS mount, and neither document predicted any of them:

• addr= is load-bearing. Docker’s local driver mounts via the kernel directly rather than shelling out to mount.cifs, and the kernel cannot resolve hostnames. Docker resolves addr= and passes the IP through. Without it the mount fails on name resolution.
• iocharset=utf8 must be dropped. The GCP worker image’s kernel ships no nls_utf8 module, so the option inherited from the Azure fstab line fails the mount with “can not access a needed shared library”. SMB3 is UTF-16 on the wire regardless.
• mount.cifs is not required - only the cifs kernel module, which the current worker image already has. This RFC previously called for adding cifs-utils to the worker bootstrap; that was wrong, and it followed from assuming a host mount rather than a driver mount.

And one prediction that did not hold: this RFC rated the SMB dialect as the likeliest first-run failure and recommended vers=3.1.1,seal before any cross-cloud run. Staging mounted successfully on vers=3.0. secureTransferOnly does require an encrypted channel, but SMB 3.0 satisfies it (AES-128-CCM) and the kernel negotiates encryption from the server’s requirement without seal being stated. The 3.1.1 recommendation stands only as a throughput improvement (AES-128-GCM), not a correctness prerequisite - see §5.

Gaps and Proposed Changes

1. Config and Secret Delivery

Problem. Both /root/.vars and the provisioner key arrive as host bind mounts (the key is the host’s /etc/gcp/service-account.json, mounted read-only at /etc/gcp/provisioner-key.json, which is where fetch_gcp_secrets.ps1 looks by default). A bind mount whose host path is missing does not fail the task — Docker creates an empty directory and starts the container. The observable failure is therefore not “mount missing” but a confusing downstream crash: substitute_variables_json.ps1 finds no vars and no secrets, leaves $(SentryDsn) unresolved, and the app dies on Sentry: Invalid URI.

Change 1a — the vars file becomes a Swarm config. Raft distributes configs to whichever node runs the task, which is exactly the property needed. The pipeline step that SCPs .envs/.<tenant>-<env>/ to the manager stays as-is; only the compose changes:

    configs:
      - source: worker_vars
        target: /root/.vars/.interfacetaskworker
        mode: 0400

configs:
  worker_vars:
    # Explicit name: Swarm prefixes stack-scoped configs and the total is capped
    # at 64 chars. Configs are immutable, so the tag makes each deploy a new object.
    name: "itw-${TENANTENVIRONMENT}-vars-${VARS_HASH}"
    file: "./.vars/.${TENANTENVIRONMENT}/.interfacetaskworker"

Version the name on a content hash, not the image tag. Configs and secrets are immutable, so a changed vars file needs a new object name. Keying on ${TAG} is not enough: flipping Worker.OwnedInterfaceTypeIds without a code change redeploys the same tag and collides. The deploy script should compute VARS_HASH as a short digest of the vars file, and prune itw-${TENANTENVIRONMENT}-vars-* objects no longer referenced by a service.

Versioning is also what avoids a stack teardown. With a fixed name, an in-place update whose content changed fails with “secret is in use”, which forces docker stack rm before every deploy - the shape PR 2423 takes, whose driver comments that remove-then-deploy is load-bearing. It is load-bearing for a fixed name. With a versioned name the service simply references a new object and rolls, so the deploy keeps its rollback target and does not tear down the stack’s networks and secrets. For a queue worker the teardown is not free: it kills the in-flight task and orphan-resets its row.

Set the name explicitly regardless, because the stack-scoped default breaks at scale. Swarm caps config and secret names at 64 characters and prefixes stack-scoped ones with <stack>_. That fits for core (56 chars), but the pipelines are per-tenant and six tenant/environment pairs already exceed the cap:

71  healthallianceplan-staging-interfacetask-stack_interfacetaskworker_vars
68  healthallianceplan-prod-interfacetask-stack_interfacetaskworker_vars
67  identityserver-staging-interfacetask-stack_interfacetaskworker_vars
66  networkhealth-staging-interfacetask-stack_interfacetaskworker_vars
65  transamerica-staging-interfacetask-stack_interfacetaskworker_vars
65  abilishealth-staging-interfacetask-stack_interfacetaskworker_vars

Latent today, since the worker’s release pipelines list only core, but it lands the first time the worker follows the legacy service onto another tenant. Same fix as above.

Change 1b — drop the provisioner key mount; authenticate hop 1 from the metadata server. fetch_gcp_secrets.ps1 does a two-hop fetch: a provisioner identity reads ha-<tenant>-vault-key from the vault-keys project, then that vault SA reads ha-<tenant>-vault from the vault-secrets project. Only hop 1 needs the key file, and on a GCP swarm node the node SA already holds that grant. Add a metadata-server path, keeping the key file as the fallback so one image works on both clouds during the parallel run:

function Get-MetadataAccessToken {
    $uri = 'http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token'
    (Invoke-RestMethod -Uri $uri -Headers @{ 'Metadata-Flavor' = 'Google' }).access_token
}

# Prefer a mounted provisioner key (Azure); fall back to the node's own identity
# (GCP swarm), where the compute SA holds secretAccessor on the vault-keys project.
$provisionerToken = if (Test-Path $ProvisionerKeyPath) {
    $key = Get-Content $ProvisionerKeyPath -Raw | ConvertFrom-Json -AsHashtable
    Invoke-WithRetry -Description 'provisioner auth' -ScriptBlock { Get-GcpAccessToken -ServiceAccountKey $key }
} else {
    Invoke-WithRetry -Description 'metadata auth' -ScriptBlock { Get-MetadataAccessToken }
}

Note the direction of the fallback. The current script exit 1s when the key file is absent; preserve a hard failure when both paths fail. There must be no silent degradation to an unauthenticated fetch.

Adopted, and validated by the staging deploy. PR 2423 initially shipped the key as a pre-created cluster secret and moved to this design in 83a32782; the container now authenticates hop 1 as the node SA on GCP and via the mounted key on Azure, so one image serves both clouds. The rejected alternative - a long-lived private key in Raft, materialising on every node that runs the task, for an identity the node already has - is no longer on the table.

One ordering dependency this RFC missed. The fallback lives in the base-app image (haap-interfacetaskworker), not the app image, so the base image has to be rebuilt and pushed before the first app build that a swarm deploy consumes. Otherwise the container looks for a key file that is not mounted and the fetch hard-fails at startup. The PR captures this as its prerequisite 2; it belongs in any similar port of a base-image-mediated contract.

2. Placement and Networking

node.labels.os == linux is an artifact of HA’s mixed-OS two-node swarms; GCP nodes carry only role=manager, set in manager-startup.sh.tftpl. The frontend overlay is likewise created by HA’s bootstrap and does not exist on the GCP clusters.

The worker is headless, so it needs no shared overlay - only egress. Note that dropping the networks block does not leave the service with no network: docker stack deploy still creates a stack-scoped <stack>_default overlay and attaches the service to it. That is fine (it is isolated per stack and costs nothing), but the accurate statement is “no cluster-wide overlay dependency”, not “no overlay”. Declaring a stack-local network explicitly, as PR 2423 does, makes the same outcome legible. Either way, constrain on role:

      placement:
        constraints:
          - node.role == worker

Keep replicas: 1. The single-replica constraint is a correctness requirement, not a capacity choice: PollingWorker.ExecuteAsync calls ResetStartedTaskQueueItems(_owned) on startup, and that reset is scoped to owned InterfaceTypes but not to the current process, so a second replica would clear a row the first is mid-processing.

3. Image Pull

The GCP swarm deliberately avoids --with-registry-auth because of moby/moby#31063: the token baked into the service spec has a roughly 1-hour TTL, and a MIG worker created more than an hour after the last deploy receives an already-expired credential and never pulls. Instead every node runs docker-gcr-proxy on 127.0.0.1:7676 and injects a locally-refreshed token.

The constraint: the proxy’s upstream is hardcoded to https://gcr.io in proxy.go. The worker image lives at us-central1-docker.pkg.dev/the-helper-bees/healthalignpms/interfacetaskworker, which the proxy cannot reach. Every other swarm stack resolves images as ${IMAGE_REGISTRY:-gcr.io}/the-helper-bees/....

Recommendation: also push the worker image to gcr.io/the-helper-bees/healthalignpms/interfacetaskworker by adding a second tag to the Docker@2 step in build.yml, and parameterise the compose reference:

    image: "${IMAGE_REGISTRY:-us-central1-docker.pkg.dev}/the-helper-bees/healthalignpms/interfacetaskworker:${TAG}"

with the swarm deploy exporting IMAGE_REGISTRY=docker-gcr-proxy:7676. This leaves the Azure deploy on its current path unchanged (no IMAGE_REGISTRY set, so the AR default applies), which matters during the parallel run.

The alternative, teaching the proxy a configurable Artifact Registry upstream, is the cleaner long-term fix and probably a small change, but it touches a component every stack on both clusters depends on for image pulls. It should be its own change, not a rider on this migration.

⚠️ Do not simply keep --with-registry-auth. It succeeds on the first deploy and fails silently on the next scale-out, which is the worst possible failure signature.

4. Deploy Path

Three things break at once: the transport (ADO SSH service connections use a static key, while GCP managers set enable-oslogin=TRUE and block-project-ssh-keys=TRUE and are IAP-only), the interpreter (no pwsh on GCP nodes), and parts of the script that only made sense on a private VM.

Recommendation: keep ownership with the HA ADO pipelines, and drive the manager from the agent pool over IAP. The existing deployFromAgentPool: true shape already runs the deploy from ADO Deploy Helper rather than a VM agent, so only the step template changes — replacing deploy-over-ssh.yml’s SSH@0 task with the ephemeral-key plus OS Login plus IAP sequence that gc_deploy_to_swarm.yml already proves out:

gcloud auth activate-service-account --key-file="$SA_KEY"
ssh-keygen -t ed25519 -N '' -f "$KEY" -C "ado-itw-deploy"
gcloud compute os-login ssh-keys add --key-file="$KEY.pub" --ttl=60m

# Stage the driver, the stack file, and the tenant's vars together. The compose
# file resolves the vars secret as ./.vars/.<tenant>-<env>/.interfacetaskworker
# relative to itself, so all three must land in the same directory.
STAGE=$(mktemp -d)
cp compose/deploy_interfacetaskworker_gcpswarm.sh \
   compose/deploy_interfacetaskworker.gcpswarm.yml "$STAGE/"
mkdir -p "$STAGE/.vars"
cp -r "$ARTIFACT/.envs/.${TENANT}-${ENV}" "$STAGE/.vars/"

# --ssh-key-file is required: without it gcloud uses ~/.ssh/google_compute_engine,
# not the ephemeral key just registered with OS Login. Copy the directory itself,
# not a glob -- "$STAGE/*" would silently skip .vars, which is a dotfile.
gcloud compute scp --tunnel-through-iap --ssh-key-file="$KEY" --zone "$ZONE" \
  --recurse "$STAGE" "$MGR:~/interfacetaskworker"
gcloud compute ssh --tunnel-through-iap --ssh-key-file="$KEY" --zone "$ZONE" "$MGR" \
  --command "sudo bash ~/interfacetaskworker/deploy_interfacetaskworker_gcpswarm.sh core prod $SHORT"

gcloud compute ssh --tunnel-through-iap subsumes hb-ansible’s explicit ProxyCommand; the manager is addressed by instance name and zone, not IP.

The staging step is the easiest part to get subtly wrong. The vars file arrives from the build artifact as .envs/.<tenant>-<env>/ but the compose file expects it at .vars/.<tenant>-<env>/, so the transfer both renames the parent directory and has to keep the tenant directory intact. Getting it wrong does not fail the copy - it fails the driver’s own missing ./.vars/... precondition, which is the good outcome. The bad outcome is a partial copy that leaves a stale vars file from an earlier deploy in place, which deploys successfully with the wrong owned-type set.

Port the deploy script to bash, and while porting, delete three single-VM artifacts:

Keep the convergence wait (IsStackRunning plus deadline) — it is the deploy’s only real health signal.

New IAM required (Terraform, hb-infra). The deploying identity needs three grants, not two: roles/iap.tunnelResourceAccessor on the swarm env project, roles/compute.osAdminLogin (not plain osLogin - the driver needs sudo for docker), and roles/iam.serviceAccountUser on the managers’ runtime service account (google_service_account.swarm_manager). The third is easy to miss and is not optional: OS Login checks iam.serviceAccounts.actAs on a VM’s attached service account at every connection, precisely so SSH access cannot be used to escalate into that account. roles/compute.osAdminLogin already carries the instance and project discovery permissions gcloud compute ssh needs. Which identity is an open question (see Risks): the ADO agent already holds a GCP identity for pushing to Artifact Registry, and there is separate in-flight work on keyless ADO-to-GCP auth worth aligning with rather than minting another long-lived key.

Why not a plain ADO SSH service connection to a manager, the shape PR 2423 originally took before moving to this sequence in 83a32782. It is not impossible, but it carries four prerequisites that are easy to under-count - the PR now folds the first two into its own prerequisite 1:

1. The POSIX username is derived, not chosen. With enable-oslogin=TRUE the account is sa_<uniqueId> for a service account (<user>_<domain>_<tld> for a human), so the endpoint’s username has to match what OS Login mints.
2. Two grants, not one. roles/compute.osAdminLogin for passwordless sudo and iam.serviceAccountUser on the manager’s runtime SA - see the comment block in swarm_manager/main.tf.
3. A firewall rule for :22 from the ADO agent’s egress IP to the manager’s external IP. The swarm’s SSH posture is OS Login plus IAP, and the tag-scoped rules live in the separate thehelperbees/gcp-networks repo - a cross-team change, not a portal click. This is the prerequisite most likely to be missed, because everything else can be staged before it and the failure looks like a hung connection.
4. It pins one manager. Any manager can serve docker stack deploy, so leadership is irrelevant, but the endpoint becomes a deploy SPOF and its host key changes whenever the image-upgrade runbook rebuilds that node.

The IAP path avoids 3 and 4 outright and makes 1 and 2 explicit in the pipeline rather than latent in a service-connection definition.

5. The Azure Files Share

This is the only genuinely hard gap, and it is narrower than it first appears. Only two of the worker’s modules touch the media path — X12837FileWriter (type 60) and ChangeHealthcareImportResponseFiles (type 7). Prod currently owns types 60,55,47, so exactly one type is affected there.

The path is a CIFS mount of //hasftp.file.core.windows.net/ha-sftp-share. The storage account is permissive on the network side — publicNetworkAccess: Enabled, networkRuleSet.defaultAction: Allow, no IP rules (verified live 2026-08-31) — so no allowlisting work is required. What is required:

Mechanism: a per-node CIFS Docker volume, not a host mount. This RFC originally proposed adding the mount to worker-startup.sh.tftpl. PR 2423’s approach is better and is adopted here: declare a local-driver volume with CIFS driver_opts in the stack file, with credentials fetched at deploy time from Secret Manager using the manager’s own metadata identity, so no credential rides through ADO.

volumes:
  hasftp:
    driver: local
    driver_opts:
      type: cifs
      device: "//hasftp.file.core.windows.net/ha-sftp-share/HealthAlign/${ENVIRONMENT}"
      # addr= and the absence of iocharset are both load-bearing - see below.
      o: "addr=hasftp.file.core.windows.net,username=${HASFTP_USERNAME},password=${HASFTP_PASSWORD},file_mode=0777,dir_mode=0777,vers=3.0"

Two properties make this the right shape. Docker establishes the mount on whichever node runs the task, so nothing has to be pre-mounted on an autoscaled pool; and a failed mount fails the task rather than silently presenting an empty directory the way a bind mount over a missing path does. That closes the silent-failure hole this RFC previously had to warn about: X12837FileWriter can no longer report success while writing into the void.

What still has to be true:

1. The cifs kernel module - but not cifs-utils. Docker’s local driver issues the mount syscall itself rather than shelling out to mount.cifs, so only the kernel module is needed and the current worker image already ships it. No swarm_worker bootstrap change is required. (An earlier revision of this RFC called for adding cifs-utils; that followed from assuming a host mount and was wrong.)
2. addr= in the options. Because the kernel performs the mount directly, it cannot resolve the hostname in device:. Docker resolves addr= and passes the address through. Omitting it fails the mount on name resolution - one of the three real staging failures.
3. No iocharset=utf8. The GCP worker kernel ships no nls_utf8 module, so the option inherited from hasftp.sh’s in-region fstab line fails the mount with “can not access a needed shared library”. SMB3 is UTF-16 on the wire regardless, and the default codepage covers the ASCII filenames this share exchanges.
4. Encryption is mandatory but vers=3.0 satisfies it. The hasftp account has secureTransferOnly: true (verified live 2026-08-31), so Azure Files refuses an unencrypted channel. An earlier revision of this RFC concluded that the inherited vers=3.0 was therefore the likeliest first-run failure and prescribed vers=3.1.1,seal. That was wrong: staging mounted cleanly on vers=3.0, because SMB 3.0 carries AES-128-CCM and the kernel negotiates encryption from the server’s requirement without seal being stated. Moving to vers=3.1.1 buys AES-128-GCM and better throughput on the internet path, which is worth doing if the 837 volumes grow - as an optimisation, not a prerequisite.
5. Credential and option changes must change the volume name. Docker volumes keep the driver_opts they were created with, so a fixed name means any change - a cred rotation, a mount-option fix - silently keeps mounting with stale options on every node that already materialised it. PR 2423 solves this by content-addressing the volume name from a fingerprint of the compose file plus the creds (14d35e51), so a change forces every node to materialise fresh and orphaned definitions (which hold no data) die with node recycling. This is the same technique §1a asks for on the vars object, and it is strictly better than this RFC’s original advice to remove the volume by hand after a rotation.

⚠️ The CIFS credential is readable from the node. Docker’s local driver has no credentials-file equivalent, so username=/password= are stored verbatim in the volume’s metadata and returned by docker volume inspect. Anyone who can reach the Docker socket on a worker can read the storage-account key. Docker-socket access is already root-equivalent, so this widens exposure from “root on the node” to “root on the node, plus anything that can query the daemon” rather than creating a new class of access - but it is a real difference from the Azure VMs, which keep the key in a 0600 root-owned credentials file outside Docker. The alternative that avoids it is the host-mount-plus-bind-mount shape this RFC originally proposed, which trades the exposure for the autoscaling and silent-failure problems §5 exists to solve. Worth an explicit decision rather than an accident; at minimum, rotate the key on any node compromise and keep it out of any support bundle that captures docker volume inspect.

No storage-firewall work is required, contrary to a common assumption. publicNetworkAccess: Enabled, networkRuleSet.defaultAction: Allow, ipRules: [] — there is nothing to allowlist for the swarm NAT egress IPs, and GCP does not block outbound 445 (it blocks 25). PR 2423 lists this as a prerequisite; it is already satisfied.

That is a statement about what the migration has to do, not an endorsement of the posture. The share is reachable from any network that holds the storage key, and after this migration it is on the PHI path for two clouds instead of one. Restricting it to the swarm Cloud NAT egress IPs plus the existing Azure consumers is a small, well-understood change (defaultAction: Deny plus ipRules), and a private endpoint is the stronger form. Neither is required to ship, and neither is in this RFC’s scope - but shipping without recording the choice would let the migration quietly inherit an open door. Tracked as risk 10.

Even with a good mechanism the ordering question stands for production. Staging settled it by events - it moved all 20 types at once and debugged the mount in place - but prod owns type 60, so the same choice recurs there:

Splitting types across two worker deployments is safe. Two workers with disjoint owned sets never reset each other’s rows, because ResetStartedTaskQueueItems(int[] owned) filters on the owned set. The legacy service’s ExcludedInterfaceTypeIds becomes the union of both workers’ owned sets; its own reset already skips worker-owned types via BuildExcludedTypeClause. The mechanical cost is a second vars file and a distinct stack name, since Worker.OwnedInterfaceTypeIds is one CSV per deployment.

6. Rebalance Churn

New exposure the Azure box does not have: when the autoscaler adds a worker, swarmctl force-updates services so Swarm spreads tasks onto it. For a single-replica order: stop-first service, a force-update is a restart mid-task — the queue row is orphan-reset and the task re-runs. Module side effects are already required to be idempotent (a single service is at-least-once by design), so this is a churn and duplicate-side-effect concern rather than a corruption one.

swarmctl already supports an exclusion list: REBALANCE_EXCLUDED_SERVICES, a CSV of service names read in config/rebalance.go. It is currently unset in src/services/swarmctl/compose.yml. Add the full Swarm service name there - docker stack deploy prefixes the stack, so the value is <tenant>-<env>-interfacetask-stack_worker (for example core-staging-interfacetask-stack_worker), not the bare worker from the compose file. A short name silently matches nothing and the exclusion appears to be configured while every rebalance still restarts the task. Autoscaler scale-in drain and MIG autohealing still apply, which is the behaviour we want.

PR 2423 does not address this — a diff-wide search for rebalance returns nothing — so it is the one item in this RFC with no counterpart in the implementation.

Proposed Compose File

Phase-1 shape: no file share, no frontend, config-based vars, metadata-server secrets.

services:
  worker:
    # IMAGE_REGISTRY is set to docker-gcr-proxy:7676 by the swarm deploy; unset on
    # the Azure path, which keeps pulling directly from Artifact Registry.
    image: "${IMAGE_REGISTRY:-us-central1-docker.pkg.dev}/the-helper-bees/healthalignpms/interfacetaskworker:${TAG}"
    environment:
      - TZ=America/Chicago
      - ENVIRONMENT=${ENVIRONMENT}
      # Reuses each tenant's existing legacy vault: VAULT_APP selects
      # `ha-<tenant>-vault` (+ `-vault-key`); ENVIRONMENT selects the n/p project.
      - VAULT_APP=ha-${TENANT}
      # Ownership is NOT set here — this file is tenant/env-agnostic. Each
      # deployment sets Worker.OwnedInterfaceTypeIds in its vars config below.
    configs:
      - source: worker_vars
        target: /root/.vars/.interfacetaskworker
        mode: 0400
    deploy:
      # Single replica is a correctness requirement, not a capacity choice: the
      # startup reset is scoped to owned types but not to this process.
      replicas: 1
      placement:
        constraints:
          - node.role == worker
      restart_policy:
        condition: any
        max_attempts: 1
        window: 120s
        delay: 5s
      resources:
        limits:
          cpus: "2.0"
          memory: 4G
      update_config:
        parallelism: 1
        delay: 10s
        failure_action: rollback
        order: stop-first

configs:
  worker_vars:
    # VARS_HASH is a short digest of the vars file, computed by the deploy
    # script: the tag alone collides when only the vars content changed (§1a).
    name: "itw-${TENANTENVIRONMENT}-vars-${VARS_HASH}"
    file: "./.vars/.${TENANTENVIRONMENT}/.interfacetaskworker"

Removed relative to today: the /root/.vars and provisioner-key bind mounts, the /data bind mount, the frontend network, and the os == linux constraint.

Phased Rollout

Handoff within each phase follows the established order — deploy legacy-excludes-first, drain in-flight rows to zero, then start the worker — because exclusion stops the legacy service claiming new tasks but does not stop it finishing current ones.

What actually happened. PR 2423 collapsed phases 2 through 4 for staging - deployTarget defaulted to gcp-swarm there at the time, and staging owns both file-touching types - so the first run exercised the new transport, the GCR pull, the metadata-server secret fetch, and cross-internet CIFS at once. It took five attempts (ADO runs 13231 through 13239), and all three real failures were mount options (see What the first deploy taught us). Phase 0 was skipped, so there is still no cross-cloud latency measurement.

Production then did the same thing: run 13460 moved all three owned types at once, including type 60, rather than taking Phase 3’s file-types-last split. Its two failures were both prerequisites rather than mount options - the ADO agent IAM trio (run 13450) and the manager’s access to hasftp-cifs-creds (run 13458) - because staging had already absorbed the mount debugging. Both were hand-applied and are now codified in infrahive#1351 and infrahive#1353.

Read honestly, the collapsed rollout argues for neither position outright. It cost four failed staging runs on an environment where failed runs are cheap, and it surfaced three findings a phased approach would have hit later and one at a time. What it did cost is the Phase 0 measurement: production shipped without a cross-cloud latency number, so the assumption this RFC most wanted evidence for in advance is now something to measure after the fact rather than a gate that was passed. Taking it against either environment is still close to free and worth doing.

Rollback at every phase is a vars change plus a redeploy: hand the types back to the Azure deployment and shrink the GCP deployment’s owned set. No data migration is involved at any point, because the database never moves.

Risks and Open Questions

| 8 | Swarm config/secret names exceed the 64-char cap on six tenant/environment pairs once the worker moves past core | Explicit name: on the object (§1a), which is also what enables the rolling update | | 9 | Cross-internet SMB has never been exercised. Closed. Exercised by ADO run 13239 on vers=3.0; the dialect concern was a throughput question, not a correctness one (§5 item 4) | Retained as a record of a wrong call. The real mount hazards were addr= and iocharset, which this RFC did not anticipate | | 10 | The hasftp share is reachable from any network holding the storage key, and this migration puts it on a second cloud’s PHI path | Restrict to the swarm NAT egress IPs plus existing Azure consumers (defaultAction: Deny + ipRules), or a private endpoint. Not required to ship; record the decision either way | | 11 | The CIFS credential is exposed via docker volume inspect on any worker that materialises the volume (§5) | Docker-socket access is already root-equivalent, so treat as exposure widening rather than a new class. Rotate on node compromise; keep docker volume inspect out of support bundles |

Effort

Compare against the HA consolidation RFC’s Path 2 at 5 to 6 weeks, which buys the Windows tier this workload does not need.

Alternatives Considered

• Leave it on Azure. Zero cost, and legitimate if the answer to risk 1 is “no HA workloads on the shared swarm”. The worker keeps a statically-sized VM with no autohealing and no graceful drain — the same gap the consolidation RFC describes for the whole HA tier.
• Wait for the dedicated HA GCP swarm (Path 2b). Correct if the PHI-isolation answer forces it; this work then becomes the first stack on that cluster rather than an addition to the shared one. Most of §1 through §4 is reusable either way.
• Deploy via AWX and gc_deploy_to_swarm.yml. Reuses a proven IAP path, but that playbook is heavily Django-shaped (migration detection, celery scaling, Caddy, canary) and would need real surgery. A short bash script driven from the existing ADO pipeline is smaller and keeps HA deploys in one place.
• Ship the provisioner SA key to GCP workers as a Swarm secret. Works, but puts a long-lived private key in Raft to obtain an identity the node already has.

Verification Checklist

Per phase, before declaring done:

• The startup log reads InterfaceTaskWorker starting. Owned InterfaceType ids: [...] with exactly the intended set
• The legacy service’s startup Excluded [...] equals the union of all worker deployments’ owned sets
• docker service ps shows the task on the intended node class, 1/1, with no restart loop
• Papertrail: the Dispatching count reconciles with Completed S plus F after a settle window, using absolute --min-time and --max-time
• No secret placeholder survives substitution (grep the running container’s appsettings.json for $()
• A deliberate MIG scale-out does not restart the worker task (the swarmctl exclusion holds)
• A worker created more than one hour after the deploy still pulls the image (the --with-registry-auth regression test)
• Phase 4 only: a deliberately broken CIFS mount fails the task rather than writing into an empty directory
• The stack’s config and secret object names are under 64 characters for the tenant being deployed
• The CIFS mount is present on the node running the task (findmnt -t cifs -o TARGET,SOURCE,OPTIONS) and the session is actually encrypted - findmnt reports the options that were requested, not what was negotiated, so confirm against /proc/fs/cifs/DebugData on that node

References

• healthAlignPMS#2423 — the pipeline and compose implementation of this design (HB-8825)
• ADO run 13239 — the first successful core-staging deploy onto the GCP swarm
• HomeAlign Swarm Consolidation: Azure-Native vs GCP Migration — the parent decision and its cost model
• Docker Swarm Consolidation — HB-8200, the GCP cluster this targets
• Swarm architecture overview
• docker-gcr-proxy — why --with-registry-auth is avoided
• swarmctl — rebalance and drain
• App Service Accounts — the no-key-file secret-fetch pattern
• Docker Swarm Outbound IP Audit — NAT egress IPs and the Azure SQL firewall inventory
• InterfaceTaskService/RFC/0001-dotnet-linux-migration.md (healthAlignPMS) — the worker’s own design, module status, and the at-least-once/orphan-recovery follow-up