Launching an HA Admin Portal Tenant

Step-by-step guide to launch a new HA Admin Portal tenant from scratch.

Getting Started

Step 1: Complete the Setup Guide

Follow the setup guide to install the Zig toolchain, authenticate with Google Cloud, and verify group membership. If you’ve already done this, skip to Step 2.

Step 2: Build LaunchBot

./zig/zig build launchbot

The compiled binary will be available as ./launchbot in the project root.

Prerequisites

Before starting a launch, confirm the following:

Partner number assigned (check partner index, auto-increment from last)
Visual Studio Code with the SQL Server (mssql) extension for database verification
Access to Azure DevOps release pipelines

sqlcmd installed (required for client record creation):

curl -fsSL https://packages.microsoft.com/keys/microsoft.asc | sudo gpg --dearmor -o /usr/share/keyrings/microsoft-prod.gpg
curl -fsSL https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/prod.list | sudo tee /etc/apt/sources.list.d/mssql-release.list
sudo apt-get update && sudo ACCEPT_EULA=Y apt-get install -y mssql-tools18 unixodbc-dev
echo 'export PATH="$PATH:/opt/mssql-tools18/bin"' >> ~/.bashrc && source ~/.bashrc

sqlpackage installed (required for database seeding):

sudo apt-get install -y dotnet-sdk-8.0 unzip
dotnet tool install -g microsoft.sqlpackage
# The seed script expects sqlpackage at ~/sqlpackage/sqlpackage
mkdir -p ~/sqlpackage
ln -s ~/.dotnet/tools/sqlpackage ~/sqlpackage/sqlpackage

Convention: Run each phase for staging first, verify, then repeat for production. Never deploy staging and production simultaneously.

Phase 1: Generate Tenant & Build Infrastructure

All phases are sequential dependencies. Each phase depends on the output of the previous one. They must be run in order — skipping ahead will cause failures.

Step 3: Generate Tenant

Run LaunchBot to generate Terraform configuration and client records:

./launchbot generate tenant -t <tenant-key> -e staging --cluster-name <THB-N-WEBx>
./launchbot generate tenant -t <tenant-key> -e production --cluster-name <THB-P-WEBx>

Or run in interactive mode (recommended for first-time operators):

./launchbot generate tenant

What this creates:

Terraform .tfvars app setting entries for the tenant
databases.tf module blocks for Azure SQL elastic pool databases
Client record in the clients table (auto-incremented client ID)
Cloudflare tunnel configuration
Optionally, a PR on branch feat/tenant-<tenant>-<env> (LaunchBot will prompt you to confirm)

Verify:

If you confirmed PR creation, check the PR in GitHub. If you declined, create it manually with gh pr create.
Connect via VS Code with the SQL Server (mssql) extension to the Core databases (hAUAT_Clean for staging, hACore for production) and run: SELECT * FROM clients ORDER BY PkClientID DESC
Confirm the new tenant appears with the correct client ID

Step 4: Build Infrastructure

Merge the tenant configuration PR (from Step 3) into the plan branch, then deploy via a release.

Merge the PR into plan
Deploy a release

What gets provisioned:

Azure SQL database in elastic pool
Cloudflare tunnel
App configuration entries

Verify: Open Cloud Build History and confirm the apply completed successfully for each target environment. The build logs should show the expected resources created.

Do not proceed to Phase 2 until Cloud Build has finished applying the infrastructure. Phase 2 reads Terraform output values that only exist after apply completes.

Phase 2: Generate Deployment Configuration

Depends on Phase 1. The deploy config command reads Terraform output values (URL, database name, tunnel ID, client ID) that are created when Cloud Build applies the infrastructure in Step 4. Do not run this before Phase 1 is complete.

Step 5: Generate Deploy Config

./launchbot generate deploy_config -t <tenant-key> -e staging
./launchbot generate deploy_config -t <tenant-key> -e production

What this creates:

.azuredevops/pipelines/../*.yml - tenant configs for AzureDevOps pipelines
.envs/.<tenant>-<env>/ — application environment variables
.cloudflared/.<tenant>-<env>/config.yml — Cloudflare tunnel config
.envs/.identityserver-<env>/ — IdentityServer client config
Optionally, a PR on branch feat/deploy_config-<tenant>-<env> targeting the release branch in healthAlignPMS (LaunchBot will prompt you to confirm)

Auto-extraction mode pulls URL, database name, tunnel ID, and client ID from Terraform output, so only --tenant and --environment are needed.

Known issues:

PR auto-creation can fail silently. Check GitHub; if no PR exists, push the branch manually and create the PR with gh pr create --base release.

Step 6: Merge Deploy Config PR

Review and merge the deploy config PR in healthAlignPMS before proceeding to Phase 3.

Phase 3: Create Vault Service Account Keys and Secrets

Step 7: Create Vault Service Account Keys and Secrets

Create GCP vault service account keys and secrets so the tenant’s containers can fetch secrets at deploy time:

./launchbot generate vault_key -t <tenant-key> -e staging
./launchbot generate vault_key -t <tenant-key> -e production

./launchbot generate vault_secret -t <tenant-key> -e staging
./launchbot generate vault_secret -t <tenant-key> -e production

What this does:

vault_key — Runs vault_post_apply to create a service account key and store it in the vault-keys secret for each environment.
vault_secret — Populates vault secrets from ADO pipeline variables, generates a new clientSecret UUID and stores it in both the tenant vault and the ha-identity-server vault.

Verify:

vault_key — Check GCP Console > Secret Manager in the vault-keys project. Confirm ha-<tenant>-vault-key secret exists for each environment.
vault_secret — Check GCP Console > Secret Manager in the vault-secrets project. Confirm ha-<tenant>-vault secret contains the tenant’s app secrets (including clientSecret), and ha-identity-server-vault secret contains <tenant>ClientSecret.

Phase 4: Database Deployment and Seeding

Depends on Phase 1 release. The ADO database pipeline deploys using a Docker image built by the release in Phase 1 (Step 4). Confirm the Docker image build has completed in ADO before creating a release. If the image is not ready, the pipeline will fail.

Step 8: Deploy Database Schema via Azure DevOps

Go to Azure DevOps > Pipelines > Releases > Database for the tenant
Create a new release using the latest build
Deploy to staging first, then production

Database names:

Staging: hA<Tenant>UAT
Production: hA<Tenant>

Verify: Connect via VS Code with the SQL Server (mssql) extension. The database should have tables but no data yet.

Step 9: Seed Database

./launchbot seed databases -e staging -d hA<Tenant>UAT
./launchbot seed databases -e production -d hA<Tenant>

Verify: Query the seeded database — tables should now contain baseline data.

Step 10: Seed App Records (Identity Server)

./launchbot seed app_records -t <tenant-key> -e staging
./launchbot seed app_records -t <tenant-key> -e production

What this does: Creates a record in the tenants.apps table in the core database, which Identity Server uses to recognize the tenant.

Verify:

SELECT * FROM Tenants.Apps ORDER BY ClientId DESC

The new tenant should appear.

Phase 5: Application Deployment

Strategy: Deploy staging completely and verify before starting production.

Step 11: Verify Vault Secrets

The client secret and vault secrets were already created in Step 7 by generate vault_secret. No manual script run is needed.

If you need to add or update secrets later, see the Secret Management guide.

Verify: Open GCP Console > Secret Manager and confirm the client secret exists for the tenant and in the ha-identity-server vault for each environment.

Step 12: Deploy Identity Server

Deploy via the ADO Identity Server release pipeline.

Create a new release with the latest build number
Deploy to staging, verify, then deploy to production

Verify: After deployment, the tenant should appear in the Identity Server clients list.

Step 13: Deploy Sync Tenants

Important: Disable Sync Tenants before deploying to avoid conflicts. Disabling has a 15-minute timeout waiting for .NET processes to exit.

Disable the Sync Tenants service on the target VM
Deploy via the ADO Sync Tenants release pipeline
Re-enable the service

Known UAT issue: .NET processes hang and don’t exit cleanly during the disable step. Workaround — SSH into the VM and kill them:

Get-Process -Name dotnet | Stop-Process

Verify: Check the Sync Sign-in Status dashboard. The new tenant’s client ID should appear within 5 minutes (one sync cycle).

Step 14: Deploy Web App

Deploy via the ADO Web App release pipeline.

GCP vault secrets must be in place before deploying (Step 7)
The fetch-secrets script runs at container startup to pull secrets from GCP
Caddy shows “bad gateway” briefly while the container starts — this is normal
First cold launch is slow (loading database connections)

Verify: Navigate to the tenant’s admin URL. The login page should appear. After Sync Tenants completes a cycle, users should be visible.

Step 15: Deploy Interface Tasks

Deploy via the ADO Interface Tasks release pipeline.

Verify: Confirm the container is running on the target VM.

Phase 6: Post-Launch Verification

Login page loads at the tenant’s admin URL (both staging and production)
Users are visible after Sync Tenants completes a cycle
Identity Server shows the tenant in its clients list
Sync Sign-in Status dashboard shows the tenant’s client ID
Interface Tasks container is running
Database contains seeded baseline data

Post-Launch Tasks (Optional)

Logo Setup

Upload the partner’s logo to Azure Storage (SFTP): HealthAlign/prod/identity-server/logos/<tenant>/
Update the tenants.apps table with the logo filename

SSO Configuration

If the partner requires SSO, configure Keycloak with their client credentials. This requires coordination with the partner and cannot be fully tested without their environment.

Edit this page