Launching an HA Admin Portal Tenant

Step-by-step guide to launch a new HA Admin Portal tenant from scratch.

There are two ways to open the tenant’s PRs: the Launch HA Tenant form (recommended) or launchbot by hand.

With the form, you need nothing installed to start: go straight to Step A1. Do Getting Started before Phase 3, which uses launchbot.
With launchbot by hand, do Getting Started first; it opens Option B.

Prerequisites

Before starting a launch, confirm the following:

Partner number assigned (check partner index, auto-increment from last)
Tenant key chosen, 25 characters or fewer (see Choosing the tenant key)
Visual Studio Code with the SQL Server (mssql) extension for database verification
Access to Azure DevOps release pipelines

sqlpackage installed (required for database seeding):

sudo apt-get install -y dotnet-sdk-8.0 unzip
dotnet tool install -g microsoft.sqlpackage
# The seed script expects sqlpackage at ~/sqlpackage/sqlpackage
mkdir -p ~/sqlpackage
ln -s ~/.dotnet/tools/sqlpackage ~/sqlpackage/sqlpackage

Convention: Run each phase for staging first, verify, then repeat for production. Never deploy staging and production simultaneously.

The tenant key is the tenant’s permanent identifier: it becomes the subdomain (<key>.myhomealign.com), the tenant_id in ha-infra, the deploy config directories (.envs/.<key>-staging), the Cloudflare tunnel and its secret, and the Docker Swarm stack and service names. Renaming it later means redoing all of those, so pick it before launching.

The form, the workflow and launchbot all accept only keys that:

are 2-25 characters long;
use lowercase letters, digits and hyphens only;
start with a letter and don’t end with a hyphen.

Why 25 characters. Docker Swarm refuses service names longer than 63 characters, and a stack’s service names are <stack>_<service>. The longest one a tenant gets is its staging webapp stack’s cloudflared (or webapp_next) service:

<key>-staging-webappcaddy-stack_cloudflared
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  38 characters besides the key

63 − 38 = 25. A longer key would launch cleanly, with all PRs opened and ha-infra applied, then fail on the first webapp deploy (Step 10), when Swarm rejects the service name. The other names built from the key have more room: the tunnel secret tunnel_myhomealign_com_to_<key>_admin_portal has up to 255 characters, and ha-infra shortens the vault service account ID to fit GCP’s 30-character limit on its own.

Abbreviate long partner names. Existing keys are short forms of the partner’s name, e.g. jh, bcbs-az, healthallianceplan. The longest today is healthallianceplan (18). Prefer a short key the team will recognise over a full name that only just fits.

Phase 1–2: Open the Tenant PRs and Build Infrastructure

Choose one of the two paths below. Both end with the same merged PRs and applied infrastructure; from Phase 3 on, the steps are the same either way.

	Option A: Launch HA Tenant form	Option B: launchbot by hand
Use it for	Every normal launch	When the form or workflow can’t be used, or to regenerate a PR
Client record	Created through each Core WebApp’s `/api/launch/clients`; with Both, one client ID on both sides	Written over SQL as `launchbot_client_creator`, from your machine; each database picks its own next ID
Deploy config PRs	Opened at once as drafts; you fill in the tunnel ID after the apply	Generated after the apply
Who can run it	Anyone with a THB email (form); `dev-infra-admins` to dispatch the workflow directly	Anyone set up per Getting Started, with access to the Core SQL servers
Setup before you start	None; Getting Started can wait until Phase 3	Getting Started

All steps are sequential dependencies. Each depends on the output of the one before it; skipping ahead will cause failures.

Option A: Launch HA Tenant form

Step A1: Submit the form

Open launch-tenant.thb.nu and sign in through Cloudflare Access.
Enter the tenant key: 2-25 lowercase letters, digits and hyphens, starting with a letter and not ending with a hyphen. Abbreviate long names; see Choosing the tenant key.
Choose the environment: Both for a real launch, so staging and production get the same client ID. Staging only or Production only configure and write just that side, for testing.
Pick the clusters. The suggestion is the cluster with the fewest tenants; VM sizes differ, so check before accepting it.
Click Open tenant PRs. Keep the page open: it follows the run, shows the step it is on, and when the run ends shows its summary in place, with the PR links or the reason it failed and how to retry.

The form runs the Launch HA Tenant workflow as you. Members of dev-infra-admins can also dispatch that workflow directly from GitHub Actions with the same inputs.

What the run does:

Creates the client record in the Core database of each environment, through that Core WebApp’s /api/launch/clients (core.myhaapp.com for staging, www.myhomealign.com for production). The ID is the tenant’s existing one if it already has a row, otherwise one past the highest ID on either side. Production is written first, then staging.
Opens the infrahive tenant PR with launchbot generate tenant --client-id <id>, on branch feat/tenant-<tenant>-<staging|production|multi>-<suffix>. The suffix is 4 hex digits, new each run, so a retry never collides with an earlier run’s branch; the healthAlignPMS branches get one too. It holds the same files as Option B’s Step B1.
Opens one draft healthAlignPMS deploy config PR per environment, against release. The tunnel ID is the placeholder SET_AFTER_HA_INFRA_APPLY, because the tunnel only exists once ha-infra is applied.

The run summary links every PR it opened. The form shows the same summary; it is also on the run’s page in GitHub Actions.

If the run stops at the client record step, its error says why, and no PR is opened:

Different client IDs on each side: the tenant already exists in both Core databases with different IDs. Fix the rows by hand, then re-run.
The ID belongs to another client: the chosen ID is taken on one side. A re-run reads the IDs again; if it keeps failing, fix the conflicting row by hand.

Re-running is safe: a row that already matches is reused, never duplicated.

Verify:

The summary on the form (or the run’s page) lists the infrahive PR and the healthAlignPMS draft PR(s).
In each Core database (hAUAT_Clean for staging, hACore for production), SELECT * FROM clients ORDER BY PkClientID DESC shows the tenant. For Both, the two IDs match.

Step A2: Build Infrastructure

Merge the infrahive tenant PR into plan.
Deploy a release.

What gets provisioned:

Azure SQL database in elastic pool
Cloudflare tunnel
Cloudflare Access app on https://<tenant>.<domain>/admin
App configuration entries

Verify: Open Cloud Build History and confirm the apply completed for each target environment.

Step A3: Finish and Merge the Deploy Config PRs

After the apply in Step A2, for each draft healthAlignPMS PR:

Get the tenant’s tunnel ID from cloudflare.cloudflare_tunnel_ids in ./zig/zig build output -- ha-infra <n|p>.
Replace SET_AFTER_HA_INFRA_APPLY in .cloudflared/.<tenant>-<env>/config.yml with it.
Mark the PR ready, then review and merge it into release.

Then continue with Phase 3.

Option B: launchbot by hand

Getting Started

Phase 3 onwards uses launchbot on either path, so form users do these two steps before Phase 3.

Step 1: Complete the Setup Guide

Follow the setup guide to install the Zig toolchain, authenticate with Google Cloud, and verify group membership. If you’ve already done this, skip to Step 2.

Step 2: Build LaunchBot

./zig/zig build launchbot

The compiled binary will be available as ./launchbot in the project root.

Step B1: Generate Tenant

Run LaunchBot to generate the Terraform configuration and the client records:

./launchbot generate tenant -t <tenant-key> -e staging --cluster-name <THB-N-WEBx>
./launchbot generate tenant -t <tenant-key> -e production --cluster-name <THB-P-WEBx>

Or both environments in a single PR, without prompts:

./launchbot generate tenant -t <tenant-key> --staging-cluster <THB-N-WEBx> --production-cluster <THB-P-WEBx> --create-pr

Or run in interactive mode (recommended for first-time operators):

./launchbot generate tenant

The client record: launchbot writes it itself through create_client_records.sh, in both Core databases by default (--client-record-env narrows that). It reuses the tenant’s existing row, or inserts that database’s MAX(PkClientID) + 1, so staging and production can get different IDs. It logs in as launchbot_client_creator, a user limited to dbo.Clients, with the password from launchbot_client_creator_password in prj-n-secrets-817e / prj-p-secrets-3c7d, read with your own gcloud login. Your machine needs network access to the Core SQL servers, the same as for VS Code. To use an ID that already exists instead, pass --client-id <id>; launchbot then touches no database.

What this creates:

Terraform .tfvars app setting entries for the tenant, with /admin gated by Cloudflare Access (thb_email)
databases.tf module blocks for Azure SQL elastic pool databases
Client record in the clients table
Cloudflare tunnel configuration
Optionally, a PR on branch feat/tenant-<tenant>-<env>-<suffix>, with a new 4-hex-digit suffix each run (LaunchBot will prompt you to confirm)

Serving WebAppNext on /admin is a separate manual rollout; its prerequisites are listed next to $webappNextTenants in healthAlignPMS compose/deploy_webapp_caddy.ps1.

Verify:

If you confirmed PR creation, check the PR in GitHub. If you declined, create it manually with gh pr create.
Connect via VS Code with the SQL Server (mssql) extension to the Core databases (hAUAT_Clean for staging, hACore for production) and run: SELECT * FROM clients ORDER BY PkClientID DESC
Confirm the new tenant appears in each database launchbot wrote. For a real launch, the two client IDs must match; if they differ, fix the rows by hand so both use one ID, then make sure the tenant PR uses that ID too (rerun with --client-id <id> if not) before merging it.

Step B2: Build Infrastructure

Same as Step A2: merge the tenant PR into plan, deploy a release, and confirm the apply in Cloud Build.

Do not continue until Cloud Build has finished applying the infrastructure. Step B3 reads Terraform output values that only exist after the apply.

Step B3: Generate Deploy Config

./launchbot generate deploy_config -t <tenant-key> -e staging
./launchbot generate deploy_config -t <tenant-key> -e production

What this creates:

.azuredevops/pipelines/../*.yml - tenant configs for AzureDevOps pipelines
.azuredevops/pipelines/all/release-<env>.yml — the full-fleet deploy list, so a full run deploys the tenant
.envs/.<tenant>-<env>/ — application environment variables
.cloudflared/.<tenant>-<env>/config.yml — Cloudflare tunnel config
.envs/.identityserver-<env>/ — IdentityServer client config
Optionally, a PR on branch feat/deploy_config-<tenant>-<env>-<suffix> targeting the release branch in healthAlignPMS (LaunchBot will prompt you to confirm; --create-pr opens it without asking)

Auto-extraction mode pulls URL, database name, tunnel ID, and client ID from Terraform output, so only --tenant and --environment are needed.

Before ha-infra is applied: pass --url, --database, --client-id and --cluster-name with --tunnel-pending. The tunnel ID becomes the placeholder SET_AFTER_HA_INFRA_APPLY and the PR opens as a draft; finish it as in Step A3. This is what Option A does.

If PR creation fails: LaunchBot reports git’s own error and prints the commands to finish by hand. The generated config is already committed on its feat/deploy_config-<tenant>-<env>-<suffix> branch in the synced clone, which is only removed after a successful PR, so nothing is lost — push the branch and run gh pr create --base release --head <that branch> (add --draft for a placeholder tunnel); launchbot prints both commands with the branch filled in. --head is needed because the synced clone only tracks release.

Step B4: Merge Deploy Config PR

Review and merge the deploy config PR in healthAlignPMS, then continue with Phase 3.

Phase 3: Create Vault Service Account Keys and Secrets

Step 3: Create Vault Service Account Keys and Secrets

Launched with the form? From here on you need launchbot. If you haven’t yet, complete Getting Started (Steps 1–2) first.

Create GCP vault service account keys and secrets so the tenant’s containers can fetch secrets at deploy time:

./launchbot generate vault_key -t <tenant-key> -e staging
./launchbot generate vault_key -t <tenant-key> -e production

./launchbot generate vault_secret -t <tenant-key> -e staging
./launchbot generate vault_secret -t <tenant-key> -e production

What this does:

vault_key — Runs vault_post_apply to create a service account key and store it in the vault-keys secret for each environment.
vault_secret — Populates vault secrets from ADO pipeline variables, generates a new clientSecret UUID and stores it in both the tenant vault and the ha-identity-server vault. Also generates the tenant’s machineKeyValidationKey and machineKeyDecryptionKey, one pair per environment, so the WebApp pins its <machineKey> instead of ASP.NET generating a fresh one inside every container. Key material is never printed; the command logs each key’s fingerprint, which the WebApp repeats at startup.

Verify:

vault_key — Check GCP Console > Secret Manager in the vault-keys project. Confirm ha-<tenant>-vault-key secret exists for each environment.
vault_secret — Check GCP Console > Secret Manager in the vault-secrets project. Confirm ha-<tenant>-vault secret contains the tenant’s app secrets (including clientSecret, machineKeyValidationKey and machineKeyDecryptionKey), and ha-identity-server-vault secret contains <tenant>ClientSecret. The two machineKey values must differ between staging and production — a shared pair would make a staging cookie valid in production.

Phase 4: Database Deployment and Seeding

Depends on the infrastructure release (Step A2 or B2). The ADO database pipeline deploys using a Docker image built by that release. Confirm the Docker image build has completed in ADO before creating a release. If the image is not ready, the pipeline will fail.

Step 4: Deploy Database Schema via Azure DevOps

Go to Azure DevOps > Pipelines > Releases > Database for the tenant
Create a new release using the latest build
Deploy to staging first, then production

Database names:

Staging: hA<Tenant>UAT
Production: hA<Tenant>

Verify: Connect via VS Code with the SQL Server (mssql) extension. The database should have tables but no data yet.

Step 5: Seed Database

./launchbot seed databases -e staging -d hA<Tenant>UAT
./launchbot seed databases -e production -d hA<Tenant>

Verify: Query the seeded database — tables should now contain baseline data.

Step 6: Seed App Records (Identity Server)

./launchbot seed app_records -t <tenant-key> -e staging
./launchbot seed app_records -t <tenant-key> -e production

What this does: Creates a record in the tenants.apps table in the core database, which Identity Server uses to recognize the tenant.

Verify:

SELECT * FROM Tenants.Apps ORDER BY ClientId DESC

The new tenant should appear.

Phase 5: Application Deployment

Strategy: Deploy staging completely and verify before starting production.

Step 7: Verify Vault Secrets

The client secret and vault secrets were already created in Step 3 by generate vault_secret. No manual script run is needed.

If you need to add or update secrets later, see the Secret Management guide.

Verify: Open GCP Console > Secret Manager and confirm the client secret exists for the tenant and in the ha-identity-server vault for each environment.

Step 8: Deploy Identity Server

Deploy via the ADO Identity Server release pipeline.

Create a new release with the latest build number
Deploy to staging, verify, then deploy to production

Verify: After deployment, the tenant should appear in the Identity Server clients list.

Step 9: Deploy Sync Tenants

Important: Disable Sync Tenants before deploying to avoid conflicts. Disabling has a 15-minute timeout waiting for .NET processes to exit.

Disable the Sync Tenants service on the target VM
Deploy via the ADO Sync Tenants release pipeline
Re-enable the service

Known UAT issue: .NET processes hang and don’t exit cleanly during the disable step. Workaround — SSH into the VM and kill them:

Get-Process -Name dotnet | Stop-Process

Verify: Check the Sync Sign-in Status dashboard. The new tenant’s client ID should appear within 5 minutes (one sync cycle).

Step 10: Deploy Web App

Deploy via the ADO Web App release pipeline.

GCP vault secrets must be in place before deploying (Step 3)
The fetch-secrets script runs at container startup to pull secrets from GCP
Caddy shows “bad gateway” briefly while the container starts — this is normal
First cold launch is slow (loading database connections)

Verify: Navigate to the tenant’s admin URL. The login page should appear. After Sync Tenants completes a cycle, users should be visible.

Step 11: Deploy Interface Tasks

Deploy via the ADO Interface Tasks release pipeline.

Verify: Confirm the container is running on the target VM.

Phase 6: Post-Launch Verification

Login page loads at the tenant’s admin URL (both staging and production)
Users are visible after Sync Tenants completes a cycle
Identity Server shows the tenant in its clients list
Sync Sign-in Status dashboard shows the tenant’s client ID
Interface Tasks container is running
Database contains seeded baseline data

Post-Launch Tasks

Step 12: Add the Production URL to Pingdom

Register an uptime check for the tenant’s production admin URL so an outage reaches the triage rotation instead of going unnoticed.

Open the Pingdom uptime checks list, signing in through SolarWinds
Add an HTTPS check for the health endpoint: https://<tenant>.myhomealign.com/api/healthcheck
Name it to match the rest of the fleet: <Partner Name> hA Admin Portal - <check URL>
Apply the same tags the other hA Admin Portal checks carry, so the new check inherits the existing alert routing

Verify: The check appears under Up in the checks list and reports a response time within a few minutes.

Step 13: Logo Setup (Optional)

Upload the partner’s logo to Azure Storage (SFTP): HealthAlign/prod/identity-server/logos/<tenant>/
Update the tenants.apps table with the logo filename

Edit this page