RFC: Generic WebApp Release Pipeline

Table of Contents

• Summary
• Motivation
• Current Architecture
  • Tenant-to-VM Mapping
  • SSH Service Connections
  • ADO Agents
• Proposed Changes
  • Change 1: Replace SSH tasks with bash scp
  • Change 2: Deployment group tags
  • Change 3: Template variables
• Implementation Plan
  • Phase 1: Validate SSH approach
  • Phase 2: Set up deployment groups
  • Phase 3: Create UAT template
  • Phase 4: Test with TransAmerica
  • Phase 5: Test different VM cluster
  • Phase 6: Create Production template
  • Phase 7: Roll out
• Risks and Mitigations
• Phase 2: YAML Pipeline Migration
  • Target Architecture
  • Entry-Point Pipeline
  • Per-Tenant Variable Files
  • Reusable Stage Template
  • Step Templates
  • Key Advantages Over Classic
  • YAML Migration Implementation Plan
  • YAML Migration Risks
• Future Considerations

Summary

Convert the tenant-specific WebApp Docker release pipelines (e.g., WebApp-TransAmerica-UAT-Secrets20) into generic templates — one for UAT and one for Production — that can deploy any tenant to its correct VM by overriding variables at release time.

Motivation

Today, each tenant (TransAmerica, Aetna, StateFarm, etc.) requires its own cloned release pipeline because two infrastructure targets are hardcoded:

1. SSH service connections (sshEndpoint GUIDs) in the “Copy files over SSH” tasks bind to a specific VM at design time. Classic Release pipelines do not support variable substitution for connected-service inputs.
2. Deployment group (queueId) in the deploy phase is a fixed reference to a single deployment group tied to one VM.

This means every new tenant onboarding requires cloning and manually editing a pipeline, and any pipeline structure change must be replicated across 30+ definitions per environment.

Current Architecture

Pipeline: WebApp-TransAmerica-UAT-Secrets20
  Phase 1 - "Transfer .env files" (runs on ADO Deploy Helper agent, queueId: 49)
    Task: Replace .webapp environment variables
    Task: Copy .env variables to Linux VM      -> sshEndpoint: 03211b98-... (THB-N-WEB3-LINUX SSH)
    Task: Copy .env variables to Windows VM     -> sshEndpoint: 8cf5492c-... (THB-N-WEB3-WIN SSH)
    Task: Copy .caddy files to Linux VM         -> sshEndpoint: 03211b98-... (THB-N-WEB3-LINUX SSH)
    Task: Copy compose files to Linux VM        -> sshEndpoint: 03211b98-... (THB-N-WEB3-LINUX SSH)
  Phase 2 - "Configure Cloudflared" (runs on ADO Deploy Helper agent, queueId: 49)
    Task: Pull cloudflared certificate
    Task: Copy cloudflared certificate to Linux VM  -> sshEndpoint: 03211b98-...
    Task: Copy cloudflared config.yml to Linux VM   -> sshEndpoint: 03211b98-...
    Task: Pull cloudflared tunnel
    Task: Copy cloudflare tunnel to Linux VM        -> sshEndpoint: 03211b98-...
    Task: Cleanup certificate
    Task: Cleanup tunnel
  Phase 3 - "Deployment group job" (deployment group queueId: 116 = THB-N-WEB3-LINUX)
    Task: Get Short Commit ID
    Task: PowerShell - deploy_webapp_all.ps1

The production pipelines follow the same structure but target production VMs (THB-P-WEB*) and use the production ADO agent.

Tenant-to-VM Mapping

UAT (non-production)

4 VM clusters. Domain: myhaapp.com

Production

8 VM clusters (no WEB8). Primary domain: myhomealign.com. Exception: Moo uses betterlivinglonger.com.

Note: Tenants are distributed differently across VMs in UAT vs Production (e.g., TransAmerica is on WEB3 in UAT but WEB6 in Production). The template must not assume any fixed tenant-to-VM mapping.

SSH Service Connections (managed by Terraform)

Terraform creates SSH service connections via azuredevops_serviceendpoint_ssh.cluster_ssh with the naming convention {VM_NAME} SSH (e.g., THB-N-WEB3-LINUX SSH). Each connection has a unique GUID. This is defined identically in both the non-production and production Terraform configurations.

ADO Agents

Each environment has its own self-hosted ADO agent on a GCP VM:

• UAT: ado-agent-n-vm (non-production GCP project)
• Production: ado-agent-p-vm (production GCP project)

Both agents have GCP service accounts with access to ado_agent_ssh_key in their respective Secret Manager projects, and their IPs are allowlisted in the Azure NSG for SSH access (ports 22 and 2290).

Proposed Changes

Change 1: Replace “Copy files over SSH” tasks with bash scp commands

Why: The built-in “Copy Files Over SSH” task (67cec91b-...) requires a sshEndpoint connected-service input, which is bound at design time in Classic pipelines and does not support $(variable) substitution.

How: Replace each “Copy Files Over SSH” task with a Bash task that runs scp directly, using pipeline variables for the target hostname.

New variables (all allowOverride: true):

SSH authentication: The self-hosted ADO agents already have GCP service accounts with access to ado_agent_ssh_key in Secret Manager. The bash tasks will:

1. Pull the SSH private key from GCP Secret Manager (or use the key already on the agent)
2. Use scp -P 2290 with the key to transfer files

Example replacement for “Copy .env variables to Linux VM”:

#!/bin/bash
set -euo pipefail

SSH_KEY="/home/azagent/.ssh/id_rsa"           # Key deployed during agent provisioning
KNOWN_HOSTS="/home/azagent/.ssh/known_hosts"  # Pre-populated during Phase 1
SSH_OPTS="-o StrictHostKeyChecking=yes -o UserKnownHostsFile=${KNOWN_HOSTS} -P 2290"
SRC="$(System.DefaultWorkingDirectory)/_WebApp-Docker/environment/.envs/$(envDir)/"
DEST="HALocalAdmin@$(linuxVmHost):/DevOps/.vars/$(envDir)/"

scp ${SSH_OPTS} -i "${SSH_KEY}" -r "${SRC}"* "${DEST}"

Full task mapping (same for both UAT and Production templates):

Change 2: Use deployment group tags for the deploy phase

Why: The deployment group job’s queueId is hardcoded to a specific deployment group (e.g., 116 = a group containing only THB-N-WEB3-LINUX).

How: Two approaches, in order of preference:

Option A: Single deployment group with tag filtering (preferred)

For each environment, create one deployment group containing agents on ALL Linux VMs, with each agent tagged by its VM name:

• UAT: WebApp-UAT-Deploy — agents on THB-N-WEB{1-4}-LINUX, tagged accordingly
• Production: WebApp-Prod-Deploy — agents on THB-P-WEB{1-7,9}-LINUX, tagged accordingly

Add a deploymentTag variable with allowOverride: true. Set the deployment group phase tags to ["$(deploymentTag)"]. This routes the deploy script to the correct VM at release time.

Terraform change required: In both business_unit_1/non-production/azure_devops.tf and business_unit_1/production/azure_devops.tf, add a new deployment group resource that registers agents on all Linux VMs, each tagged with its hostname.

Option B: Per-VM deployment groups with variable queue ID

If Classic pipelines support variable substitution in the deployment group phase queueId (needs verification), add a deployQueueId variable. Each tenant release would override this with the numeric ID of the appropriate deployment group.

Change 3: Template variables

Two templates will be created: WebApp-UAT-Template and WebApp-Prod-Template. They share the same pipeline structure but differ in their default variable values and agent pool.

WebApp-UAT-Template

WebApp-Prod-Template

Key differences between the two templates:

• ENVIRONMENT default: staging vs production
• domain default: myhaapp.com vs myhomealign.com
• Agent pool queueId for phases 1 and 2: UAT ADO agent vs Production ADO agent
• Deployment group queueId for phase 3: WebApp-UAT-Deploy vs WebApp-Prod-Deploy

Note: domain is allowOverride: true in both templates because Production has an exception (Moo uses betterlivinglonger.com), and future UAT tenants could also deviate.

Implementation Plan

Phase 1: Validate SSH approach on existing agents

1. SSH into the UAT ADO agent VM (ado-agent-n-vm) and verify the SSH key path and connectivity to target VMs on port 2290
2. Scan and store host keys for all target VMs: ssh-keyscan -p 2290 <vm-ip> >> /home/azagent/.ssh/known_hosts for each Linux and Windows VM
3. Test scp -P 2290 -i <key> <file> HALocalAdmin@<linux-vm-ip>:<path> manually from the agent against a Linux VM
4. Test scp -P 2290 -i <key> <file> HALocalAdmin@<windows-vm-ip>:<path> manually from the agent against a Windows VM (confirms Win32-OpenSSH Server is running)
5. Repeat steps 2-4 for the Production ADO agent (ado-agent-p-vm) against production VMs

Phase 2: Set up unified deployment groups

1. In Terraform (business_unit_1/non-production/azure_devops.tf), create a WebApp-UAT-Deploy deployment group with agents on all 4 UAT Linux VMs, each tagged with its hostname
2. In Terraform (business_unit_1/production/azure_devops.tf), create a WebApp-Prod-Deploy deployment group with agents on all 8 Production Linux VMs, each tagged with its hostname
3. Apply via the infrahive pipeline

Phase 3: Create the UAT template pipeline

1. Clone the existing WebApp-UAT-Template (definition 1005)
2. Replace all “Copy Files Over SSH” tasks with Bash scp tasks
3. Update the deployment group phase to use the new WebApp-UAT-Deploy group with tag filtering
4. Add the new variables (linuxVmHost, windowsVmHost, deploymentTag)
5. Set domain to allowOverride: true
6. Remove unused clientSecret variable

Phase 4: Test UAT template with TransAmerica

1. Create a release from the template, overriding variables for TransAmerica:
   • TENANTS = transamerica
   • TENANT_URL = https://transamerica.myhaapp.com
   • envDir = .transamerica-staging
   • tunnelSecret = tunnel_myhaapp_com_to_transamerica_admin_portal
   • linuxVmHost = <THB-N-WEB3-LINUX IP>
   • windowsVmHost = <THB-N-WEB3-WIN IP>
   • deploymentTag = THB-N-WEB3-LINUX
2. Verify all file transfers and deployment succeed
3. Compare results against the existing WebApp-TransAmerica-UAT-Secrets20 pipeline

Phase 5: Test UAT template with a different VM cluster

1. Test with a tenant on a different VM cluster (e.g., Aetna on WEB2)
2. Verify the variable overrides correctly route to the different VM

Phase 6: Create and test the Production template pipeline

1. Clone the validated UAT template
2. Update defaults: ENVIRONMENT = production, domain = myhomealign.com
3. Switch agent pool queueId to the Production ADO agent
4. Switch deployment group to WebApp-Prod-Deploy
5. Test with a low-risk production tenant (e.g., Demo)
6. Verify all file transfers and deployment succeed

Phase 7: Roll out

1. Gradually migrate remaining tenants to the templates (UAT first, then Production)
2. Retire the per-tenant pipeline definitions
3. Document variable values for each tenant, or create variable groups per tenant (see Future Considerations)

Risks and Mitigations

Phase 2: YAML Pipeline Migration

After the Classic pipeline template (Phases 1-7 above) is validated, migrate to Azure DevOps YAML pipelines. YAML pipelines provide compile-time parameterization of service connections, version-controlled pipeline definitions, and reusable templates — eliminating the scp workaround and all hardcoded references.

Target Architecture

healthAlignPMS repo:
  .azuredevops/
    pipelines/
      webapp-release.yml              # Entry-point pipeline (triggered per-environment)
      templates/
        webapp-release-stages.yml     # Reusable stage template
        steps/
          transfer-env-files.yml      # File transfer steps
          configure-cloudflared.yml   # Cloudflare tunnel/cert steps
          deploy-webapp.yml           # Deployment group deploy steps
    tenant-configs/
      staging/
        transamerica.yml              # Per-tenant variable file
        aetna.yml
        ...
      production/
        transamerica.yml
        aetna.yml
        ...

Entry-Point Pipeline

One pipeline definition per environment, parameterized by tenant:

# .azuredevops/pipelines/webapp-release.yml
trigger: none  # Manual or triggered by build completion

parameters:
  - name: tenant
    displayName: Tenant
    type: string
    values:
      - transamerica
      - aetna
      - statefarm
      # ... all tenants
  - name: environment
    displayName: Environment
    type: string
    default: staging
    values:
      - staging
      - production

variables:
  - template: tenant-configs/${{ parameters.environment }}/${{ parameters.tenant }}.yml

stages:
  - template: templates/webapp-release-stages.yml
    parameters:
      environment: ${{ parameters.environment }}
      tenant: ${{ parameters.tenant }}
      tenants: ${{ variables.TENANTS }}
      tenantUrl: ${{ variables.TENANT_URL }}
      domain: ${{ variables.domain }}
      envDir: ${{ variables.envDir }}
      tunnelSecret: ${{ variables.tunnelSecret }}
      linuxSshConnection: ${{ variables.linuxSshConnection }}
      windowsSshConnection: ${{ variables.windowsSshConnection }}
      agentPool: ${{ variables.agentPool }}
      deploymentEnvironment: ${{ variables.deploymentEnvironment }}

Per-Tenant Variable Files

Each tenant gets a small YAML file with its configuration, replacing the allowOverride variables:

# .azuredevops/tenant-configs/staging/transamerica.yml
variables:
  TENANTS: transamerica
  TENANT_URL: https://transamerica.myhaapp.com
  domain: myhaapp.com
  envDir: .transamerica-staging
  tunnelSecret: tunnel_myhaapp_com_to_transamerica_admin_portal
  linuxSshConnection: THB-N-WEB3-LINUX SSH
  windowsSshConnection: THB-N-WEB3-WIN SSH
  agentPool: ADO Deploy Helper UAT
  deploymentEnvironment: WebApp-UAT-WEB3

Reusable Stage Template

# .azuredevops/pipelines/templates/webapp-release-stages.yml
parameters:
  - name: environment
    type: string
  - name: tenant
    type: string
  - name: tenants
    type: string
  - name: tenantUrl
    type: string
  - name: domain
    type: string
  - name: envDir
    type: string
  - name: tunnelSecret
    type: string
  - name: linuxSshConnection
    type: string
  - name: windowsSshConnection
    type: string
  - name: agentPool
    type: string
  - name: deploymentEnvironment
    type: string

stages:
  - stage: TransferFiles
    displayName: Transfer .env files
    jobs:
      - job: TransferEnvFiles
        pool: ${{ parameters.agentPool }}
        steps:
          - template: steps/transfer-env-files.yml
            parameters:
              envDir: ${{ parameters.envDir }}
              linuxSshConnection: ${{ parameters.linuxSshConnection }}
              windowsSshConnection: ${{ parameters.windowsSshConnection }}

  - stage: ConfigureCloudflared
    displayName: Configure Cloudflared
    dependsOn: TransferFiles
    jobs:
      - job: ConfigureCloudflared
        pool: ${{ parameters.agentPool }}
        steps:
          - template: steps/configure-cloudflared.yml
            parameters:
              domain: ${{ parameters.domain }}
              envDir: ${{ parameters.envDir }}
              tunnelSecret: ${{ parameters.tunnelSecret }}
              environment: ${{ parameters.environment }}
              linuxSshConnection: ${{ parameters.linuxSshConnection }}

  - stage: Deploy
    displayName: Deploy WebApp
    dependsOn: ConfigureCloudflared
    jobs:
      - deployment: DeployWebApp
        environment: ${{ parameters.deploymentEnvironment }}
        strategy:
          runOnce:
            deploy:
              steps:
                - template: steps/deploy-webapp.yml
                  parameters:
                    tenants: ${{ parameters.tenants }}
                    environment: ${{ parameters.environment }}
                    tenantUrl: ${{ parameters.tenantUrl }}

Step Templates

# .azuredevops/pipelines/templates/steps/transfer-env-files.yml
parameters:
  - name: envDir
    type: string
  - name: linuxSshConnection
    type: string
  - name: windowsSshConnection
    type: string

steps:
  - task: replacetokens@6
    displayName: Replace .webapp environment variables
    inputs:
      rootDirectory: $(Pipeline.Workspace)/_WebApp-Docker/environment/.envs/${{ parameters.envDir }}
      targetFiles: .webapp
      tokenPattern: azpipelines

  - task: CopyFilesOverSSH@0
    displayName: Copy .env variables to Linux VM
    inputs:
      sshEndpoint: ${{ parameters.linuxSshConnection }}
      sourceFolder: $(Pipeline.Workspace)/_WebApp-Docker/environment/.envs/${{ parameters.envDir }}
      contents: '**'
      targetFolder: /DevOps/.vars/${{ parameters.envDir }}
      overwrite: true

  - task: CopyFilesOverSSH@0
    displayName: Copy .env variables to Windows VM
    inputs:
      sshEndpoint: ${{ parameters.windowsSshConnection }}
      sourceFolder: $(Pipeline.Workspace)/_WebApp-Docker/environment/.envs/${{ parameters.envDir }}
      contents: '**'
      targetFolder: /DevOps/.vars/${{ parameters.envDir }}
      overwrite: true

  - task: CopyFilesOverSSH@0
    displayName: Copy .caddy files to Linux VM
    inputs:
      sshEndpoint: ${{ parameters.linuxSshConnection }}
      sourceFolder: $(Pipeline.Workspace)/_WebApp-Docker/environment/.caddy/${{ parameters.envDir }}
      contents: '**'
      targetFolder: /DevOps/.caddy/${{ parameters.envDir }}
      overwrite: true

  - task: CopyFilesOverSSH@0
    displayName: Copy compose files to Linux VM
    inputs:
      sshEndpoint: ${{ parameters.linuxSshConnection }}
      sourceFolder: $(Pipeline.Workspace)/_WebApp-Docker/environment/compose/
      contents: '**'
      targetFolder: /DevOps
      overwrite: true

Key Advantages Over Classic

YAML Migration Implementation Plan

Phase 8: Scaffold YAML pipeline structure

1. Create .azuredevops/pipelines/ directory structure in the healthAlignPMS repo
2. Write the entry-point pipeline (webapp-release.yml) with tenant/environment parameters
3. Write the stage template (webapp-release-stages.yml)
4. Write the step templates (transfer-env-files.yml, configure-cloudflared.yml, deploy-webapp.yml)

Phase 9: Create per-tenant config files

1. Generate tenant config YAML files from the infrahive tfvars apps list (script or manual)
2. Each file maps the tenant to its service connection names and deployment environment
3. Create files for UAT first, then production

Phase 10: Set up Azure DevOps environments

1. Create YAML environments in Azure DevOps (e.g., WebApp-UAT-WEB1, WebApp-UAT-WEB2, etc.)
2. Register the target Linux VM as a resource in each environment
3. Optionally add approval gates on production environments

Phase 11: Validate YAML pipeline on UAT

1. Register the YAML pipeline in Azure DevOps pointing to webapp-release.yml
2. Run for TransAmerica (UAT) and compare results against the Classic template
3. Run for a tenant on a different VM cluster (e.g., Aetna on WEB2)
4. Verify service connection parameterization works correctly — the CopyFilesOverSSH task should use the named connection from the tenant config

Phase 12: Validate YAML pipeline on Production

1. Create production tenant config files
2. Set up production YAML environments with approval gates
3. Test with a low-risk production tenant (e.g., Demo)

Phase 13: Full migration and cleanup

1. Migrate all remaining tenants to YAML pipeline
2. Retire the Classic template pipelines (UAT and Production)
3. Delete the interim scp-based Classic templates from Phase 3/6
4. Update onboarding documentation: new tenant = new YAML config file + PR

YAML Migration Risks

Future Considerations

• Variable groups per tenant: Once the templates are validated, create an Azure DevOps variable group per tenant per environment (e.g., WebApp-UAT-TransAmerica, WebApp-Prod-TransAmerica) pre-loaded with all override values. Link the appropriate group at release time instead of manually entering values.
• Automation from infrahive: The infrahive Terraform already knows the tenant-to-VM mapping in the apps list. A script could generate or update the per-tenant YAML config files and YAML environments automatically from the tfvars, ensuring the pipeline config always matches the infrastructure source of truth.
• Multi-tenant batch deploys: The YAML pipeline could be extended with a matrix strategy to deploy multiple tenants on the same VM in parallel, reducing total deployment time for full-environment rollouts.
• Pipeline triggers: Once stable, wire the YAML release pipeline to trigger automatically on successful WebApp-Docker build completions, replacing the current manual release creation.