RFC: YAML Pipeline Migration for Release Pipelines

Table of Contents

• Summary
• Motivation
• Current Architecture
  • Tenant-to-VM Mapping
  • SSH Service Connections
  • ADO Agents
• Implemented Architecture
  • Key Design Decisions
  • File Structure
  • Pipelines Overview
  • Shared Templates
  • Per-Tenant Variable Files
  • Key Advantages Over Classic
• Implementation Plan
• Lessons Learned
• Future Considerations

Summary

Replaced ~200+ per-tenant Classic Release pipelines across 5 services (WebApp, IdentityServer, InterfaceTaskService, Database, SyncTenant) with 21 parameterized YAML pipelines checked into the healthAlignPMS repo. Per-tenant configuration is stored in YAML variable files, and all pipeline logic is version-controlled via reusable templates.

Motivation

Each tenant required its own cloned Classic Release pipeline because:

1. SSH service connections (sshEndpoint GUIDs) bind to a specific VM at design time. Classic Release pipelines do not support variable substitution for connected-service inputs.
2. Deployment group (queueId) is a fixed reference to a single deployment group tied to one VM.

This meant every new tenant onboarding required cloning and manually editing a pipeline, and any pipeline structure change had to be replicated across 30+ definitions per environment.

YAML pipelines solve both problems: service connections are parameterized via compile-time template expressions (${{ }}), and deployment groups are replaced by YAML environment: resources with VM tags.

Current Architecture

Pipeline: WebApp-TransAmerica-Docker-UAT (Classic Release)
  Phase 1 - "Transfer .env files" (runs on ADO Deploy Helper agent, queueId: 49)
    Task: Replace .webapp environment variables
    Task: Copy .env variables to Linux VM      -> sshEndpoint: 03211b98-... (THB-N-WEB3-LINUX SSH)
    Task: Copy .env variables to Windows VM     -> sshEndpoint: 8cf5492c-... (THB-N-WEB3-WIN SSH)
    Task: Copy .caddy files to Linux VM         -> sshEndpoint: 03211b98-...
    Task: Copy compose files to Linux VM        -> sshEndpoint: 03211b98-...
  Phase 2 - "Configure Cloudflared" (runs on ADO Deploy Helper agent, queueId: 49)
    Task: Pull cloudflared certificate / tunnel from GCP
    Task: Copy cert/tunnel/config to Linux VM
    Task: Cleanup temp files
  Phase 3 - "Deployment group job" (deployment group queueId: 116 = THB-N-WEB3-LINUX)
    Task: Get Short Commit ID
    Task: PowerShell - deploy_webapp_all.ps1

Other services (IdentityServer, InterfaceTaskService, Database, SyncTenant) follow similar patterns with variations in env files, vault secrets, and deploy scripts.

Tenant-to-VM Mapping

UAT (non-production)

4 VM clusters. Domain: myhaapp.com

Production

8 VM clusters (no WEB8). Primary domain: myhomealign.com. Exception: Moo uses betterlivinglonger.com.

SSH Service Connections (managed by Terraform)

Terraform creates SSH service connections via azuredevops_serviceendpoint_ssh.cluster_ssh with the naming convention {VM_NAME} SSH (e.g., THB-N-WEB3-LINUX SSH). YAML pipelines reference these by name instead of GUID, resolved at compile time via ${{ }} expressions.

ADO Agents

• Staging: ADO Deploy Helper agent pool (non-production GCP agent)
• Production: ADO Deploy Helper - Prod agent pool (production GCP agent)

Implemented Architecture

Key Design Decisions

1. Phases 1+2 merged into a single job. Vault secret variables and Cloudflare output variables are naturally available to subsequent steps without cross-job plumbing.

2. Artifact via resources.pipelines. Downloaded with - download: {pipeline} at $(Pipeline.Workspace)/{pipeline}/. Only environment artifact is downloaded (not drop) except for Database which needs both.

3. Deploy stage uses environment: with resourceType: virtualMachine and tags. Replaces Classic deployment groups. VMs are re-registered in YAML environments (separate from Classic deployment groups). 2 shared environments (Staging, Production) with all VMs tagged by hostname. Production has approval gates.

4. Artifact source commit passed via file on VM. Pipeline resource variables aren’t available on environment VM agents, and ADO keeps hyphens in env var names. The workaround: env | grep -i 'RESOURCES_PIPELINE_.*_SOURCECOMMIT' extracts the commit, writes to /DevOps/.artifact_commit_$(Build.BuildId) via SSH, deploy step reads it.

5. Separate staging and production entry-point files. Staging uses latest build from any branch; production restricts to release branch via resources.pipelines.branch filter.

6. Separate agent pools per environment. Staging uses ADO Deploy Helper (queueId 49), production uses ADO Deploy Helper - Prod (queueId 72).

7. Token replacement preserves unresolved tokens. replacetokens@6 with tokenPattern: 'azpipelines' and missingVarAction: 'keep' — only vault secrets are replaced, container-consumed tokens like $(server) are left intact.

File Structure

healthAlignPMS/.azuredevops/
  pipelines/
    webapp/
      release-staging.yml, release-production.yml
      templates/release-stages.yml, steps/{transfer-env-files,deploy}.yml
    identityserver/
      release-staging.yml, release-production.yml
      templates/release-stages.yml, steps/{transfer-env-files,deploy}.yml
    interfacetaskservice/
      release-staging.yml, release-production.yml
      templates/release-stages.yml, steps/{transfer-env-files,deploy}.yml
    database/
      release-staging.yml, release-production.yml
      templates/release-stages.yml, steps/{transfer-files,deploy}.yml
    synctenant/
      release-staging.yml, release-production.yml
      enable-staging.yml, enable-production.yml
      disable-staging.yml, disable-production.yml
      update-client-staging.yml, update-client-production.yml
      update-server-client-staging.yml, update-server-client-production.yml
      one-time-sync-production.yml
      templates/release-stages.yml, steps/{transfer-files,deploy}.yml
    shared/
      steps/configure-cloudflared.yml
  tenant-configs/
    staging/    # 39 WebApp + 1 IdentityServer = 40 configs
    production/ # 39 WebApp + 1 IdentityServer = 40 configs

Pipelines Overview

21 YAML pipelines registered in ADO:

Service comparison:

Shared Templates

shared/steps/configure-cloudflared.yml — Used by WebApp and IdentityServer. Takes an artifactName parameter. Steps: pull cert from GCP, copy to VM, pull tunnel, copy to VM, cleanup.

Per-Tenant Variable Files

Each per-tenant service (WebApp, InterfaceTaskService, Database) shares the same tenant config files:

# .azuredevops/tenant-configs/staging/transamerica.yml
variables:
  envDir: '.transamerica-staging'
  domain: 'myhaapp.com'
  tenantUrl: 'https://transamerica.myhaapp.com'
  tunnelSecret: 'tunnel_myhaapp_com_to_transamerica_admin_portal'
  linuxSshEndpoint: 'THB-N-WEB3-LINUX SSH'
  windowsSshEndpoint: 'THB-N-WEB3-WIN SSH'
  deploymentEnvironment: 'Staging'
  deploymentTag: 'THB-N-WEB3-LINUX'
  agentPool: 'ADO Deploy Helper'

Production configs use agentPool: 'ADO Deploy Helper - Prod'.

Single-service pipelines (IdentityServer, SyncTenant) have config values inlined in their entry-point files.

Key Advantages Over Classic

Implementation Plan

All phases complete.

Phase 1: Scaffold YAML pipeline structure — DONE

Created .azuredevops/pipelines/ directory structure for all 5 services with staging/production entry-points, stage templates, step templates, and shared Cloudflare template.

Phase 2: Create per-tenant config files — DONE

Generated 80 tenant config files (39+39 WebApp, 1+1 IdentityServer). All tunnelSecret values verified against Classic pipeline variables. Removed defunct tenants (humanabenefits, uch).

Phase 3: Set up Azure DevOps environments — DONE

Created 2 shared YAML environments (Staging, Production) with all VMs registered and tagged by hostname. Production environment has approval gates. All SSH service connections authorized.

Phase 4: Validate on staging — DONE

Validated all services on staging: WebApp (transamerica on WEB3, unum on WEB1), IdentityServer (WEB4), InterfaceTaskService (transamerica), Database (transamerica), SyncTenant (WEB4).

Phase 5: Validate on production — DONE

Validated all services on production with approval gates and release branch enforcement.

Phase 6: Cleanup — REMAINING

1. Retire Classic per-tenant pipeline definitions
2. Update onboarding documentation: new tenant = new YAML config file + PR
3. Deprecate LaunchBot Classic pipeline generation

Lessons Learned

• YAML environments and Classic deployment groups are separate systems. VMs must be re-registered. Both agents coexist on the same VM during migration.
• resourceType: virtualMachine is required. Without it, ADO defaults to a hosted agent.
• Pipeline resource variables with hyphens don’t resolve via $() macro. ADO keeps hyphens in env var names. Use env | grep to extract.
• replacetokens@6 defaults replace missing tokens with empty strings. Must set missingVarAction: 'keep' and tokenPattern: 'azpipelines'.
• Staging and production need separate agent pools. ADO Deploy Helper (staging) vs ADO Deploy Helper - Prod (production).
• Disable CI/PR triggers in ADO UI settings — YAML trigger: none and pr: none aren’t always sufficient.
• Download only needed artifacts — artifact: environment plus download: none + checkout: none prevents redundant downloads.

Future Considerations

• Automate environment setup: Manage via azuredevops Terraform provider or ADO REST API.
• Automation from infrahive: Generate tenant config YAML files from tfvars automatically.
• Multi-tenant batch deploys: Matrix strategy for parallel deployment of multiple tenants on the same VM.
• Pipeline triggers: Auto-trigger on build completions once stable.
• LaunchBot deprecation: Replace Classic pipeline generation with YAML config generation.