Launching a Benefits Hub Tenant

Step-by-step guide to launch a new Benefits Hub tenant on the shared Benefits Hub VM.

Prerequisites

Before starting a launch, confirm the following:

• The partner’s HA Admin Portal tenant is already launched and operational
• Tenant name confirmed (lowercase identifier used in resource naming, e.g., simpra)
• Tenant display name confirmed (e.g., Simpra)
• Subdomain decided (pattern: <tenant>benefits)
• Access to the Pingdom dashboard (needed in Step 9)

The HA platform and tenant must already be launched. Benefits Hub depends on the Django HomeAlign backend for its API and authentication. If the HA tenant doesn’t exist yet, follow the HA Admin Portal launch runbook first.

Phase 1: Setup Partner in Django HomeAlign

Find <ALIAS> off the selector:

grep -oE '<MenuItem value="[^"]+">[^<]+<' DatabaseSelect.tsx

<PREFIX> is <ALIAS> all uppercase.

1. Run the Tenant Database Script

   • ./add_new_tenant_database.sh

     UAT database name         e.g. hAScanUAT
     Production database name  e.g. hAScan
     database alias            e.g. scan     <- becomes <ALIAS>
     display name              e.g. Scan     <- the dropdown label
     database user             default haadmin is correct
     prod / staging DB host    defaults are correct

   • Verification: Review the diff PR 861. Expect exactly 4 files.

     NOTE: If doing this for <ENV> = production, most likey this step has already been done.

2. Add <PREFIX>_WEB_DB_PASS

   • <ENV> = staging ==> project prj-bu1-n-vault-secrets-9d03
   • <ENV> = production ==> project prj-bu1-p-vault-secrets-8d4a

   Two vaults per environment: django-ha-vault and eligibility-service-vault.

   • Add the key <PREFIX>_WEB_DB_PASS to both.
   • Verification
     • Four places: {django-ha-vault, eligibility-service-vault} x {staging, production}
       • <ENV> = staging ==> project prj-bu1-n-vault-secrets-9d03
       • <ENV> = production ==> project prj-bu1-p-vault-secrets-8d4a
     • Each vault is ONE secret holding all tenants’ keys, so check for the KEY <PREFIX>_WEB_DB_PASS inside the payload.
     • PASS = the key exists and is non-empty in all four.

3. Add the tenant to config/client_config.json

   • Add the following key

     "<KEY_NAME>" : {
         "ha_db_name_key": "<ALIAS>",
         "client_id": <ID>
     }

   • If Sanity is required feature

     "cp_tenants": {
         "<cp_key>": {
             "sanity_client": {
                 "uat":        { "sanity_client_name": "<name>" },
                 "production": { "sanity_client_name": "<name>" }
             }
         }
     }

     • <cp_key>: find in consumer_portal/.envs/.<TENANT>-<ENV>/.web. No .envs dir for the tenant means Consumer Portal does not have it, which also means it has no Sanity content.

     • <name>: the tenant’s value in Sanity’s client field. Must match EXACTLY.

       1. List every client Sanity knows about:
       2. Find <name> in the list.

       Not in the list -> no Sanity client exists yet. Exit and ask ME.

   • Verficiation: Check if key + fields are in the file.

4. If <ENV> == staging, PR and Deploy. Between staging and production, this only needs to be done once.

   • Verification

     frontend/src/layout/DatabaseSelect.tsx — one line:

     <MenuItem value="<ALIAS>">

   If <ENV> == production, deploy master to release. Only needs to be done once per batch.

Phase 2: Add App Config in Infrahive

Benefits Hub tenants share a single VM. Launching a new tenant means adding an app entry to the hb-infra tfvars files, which provisions Cloudflare DNS records, tunnels, and Ansible deployment configuration.

Step 3: Add App Entry to Non-Production

Edit infra/hb-infra/non-production.auto.tfvars and add a new entry to the apps list:

{
  name                   = "<TENANT>-benefits-platform"
  ansible_name           = "<TENANT>-benefits-platform"
  vm_resource_name       = "benefits-hub"
  db_resource_name       = ""
  public_container_name  = "ui"
  admin_container_name   = "ui"
  github_repo            = "git@github.com:thehelperbees/benefits-platform.git"
  github_short_repo      = "benefits-platform"
  docker_container_abbrv = "benefits-platform"
  docker_image           = "benefits-platform/ui"
  docker_stack_name      = "<TENANT>_benefits_platform"
  docker_compose_version = "3.9"
  docker_secret_file     = ""
  ping_route             = "/api/ping"
  app_env                = "<TENANT>-staging"
  dashboard_name         = "<TENANT_DISPLAY> Benefits Platform"
  domains = [
    {
      name               = "thb.sh"
      cloudflare_zone_id = "<NON_PROD_ZONE_ID>"
      subdomains = [
        {
          "type" : "admin"
          "name" : "<TENANT>benefits"
          "access_policies" : ["thb_email", "vpn_bypass"]
        },
      ]
    }
  ]
  public_site_urls = []
  admin_site_urls  = ["<TENANT>benefits.thb.sh"]
  enable_ws : false
  enable_gql : false
  enable_flower : false
  enable_camunda : false
},

Step 4: Add App Entry to Production

Edit infra/hb-infra/production.auto.tfvars with the production variant. Key differences from non-production:

{
  name                   = "<TENANT>-benefits-platform"
  ansible_name           = "<TENANT>-benefits-platform"
  vm_resource_name       = "benefits-hub"
  db_resource_name       = ""
  public_container_name  = "ui"
  admin_container_name   = "ui"
  github_repo            = "git@github.com:thehelperbees/benefits-platform.git"
  github_short_repo      = "benefits-platform"
  docker_container_abbrv = "benefits-platform"
  docker_image           = "benefits-platform/ui"
  docker_stack_name      = "<TENANT>_benefits_platform"
  docker_compose_version = "3.9"
  ping_route             = "/api/ping"
  app_env                = "<TENANT>-production"
  dashboard_name         = "<TENANT_DISPLAY> Benefits Platform"
  domains = [
    {
      name               = "thb.nu"
      cloudflare_zone_id = "<PROD_ZONE_ID_THB_NU>"
      subdomains = [
        {
          "type" : "public"
          "name" : "<TENANT>benefits"
          "access_policies" : []
        }
      ]
    },
    {
      name               = "thehelperbees.com"
      cloudflare_zone_id = "<PROD_ZONE_ID_THB_COM>"
      subdomains = [
        {
          "type" : "public"
          "name" : "<TENANT>benefits"
          "access_policies" : []
        }
      ]
    }
  ]
  public_site_urls = ["<TENANT>benefits.thehelperbees.com"]
  admin_site_urls  = ["<TENANT>benefits.thb.nu"]
  enable_ws : false
  enable_gql : false
  enable_flower : false
  enable_camunda : false
},

Remember to ./zig/zig build fmt to ensure the tfvars file is properly formatted.

Check the length of docker_stack_name.

Docker caps config and secret names at 64 characters. The deploy builds each name as:

<docker_stack_name>_cf_<domain with . as _>_config_<5-char hash>

Production’s longest domain is thehelperbees.com, which becomes thehelperbees_com — 17 characters. With the template’s docker_stack_name = "<tenant>_benefits_platform":

len(<tenant>) + 18 + 4 + 17 + 5 + 8  must be <= 64
         ==>  len(<tenant>) must be <= 12

If <tenant> is longer than 12 characters, choose a shorter docker_stack_name (e.g. hapcaresrc_benefits_platform) and use it in both staging and production.

Fields that change per tenant:

Non-prod vs prod differences:

Convention: Copy the Cloudflare zone IDs from an existing benefits-platform entry in the same tfvars file.

Phase 3: Release and Build Infrastructure

Step 5: Create a PR and Deploy

1. Create a branch and commit the tfvars changes
2. Open a PR targeting the plan branch
3. After PR approval, merge into plan
4. Ask a dev-infra-admins member to deploy a release.

If you are deploying a whole batch of tenants, do so in one big staging batch, and then later, one big production batch.

What gets provisioned:

• Cloudflare tunnel routes and DNS records for the new subdomain(s)
• Cloudflare Access policies (non-production only)
• GCP Secret Manager vault for the app
• Ansible config YAML uploaded to GCS bucket (consumed by AWX for deployment)
• GCP uptime monitoring (production only)

Verify: use Main branch.

Read the hosts from the tenant’s entry in tfvars: public_site_urls[] and admin_site_urls[].

These are complete FQDNs. Do not assemble them from domains[] / subdomains[]. Production has two zones and staging has one, so the number of hosts differs by environment — check every URL listed.

For each host:

curl -sS -o /dev/null -m 15 -w '%{http_code}\n' https://<host>

• 000, or DNS does not resolve — Error, Terraform never applied
• any HTTP status — pass, DNS and the tunnel config exist

If That Fails: Open Cloud Build History and confirm the hb-infra apply completed successfully for each target environment.

Phase 4: Setup Benefits-Platform Environment

Step 6: Add Environment Variables

In the benefits-platform repo, create .env files for the new tenant. Each tenant needs two files:

• .envs/.<tenant>-staging/.web
• .envs/.<tenant>-production/.web

As a reminder, ALIAS may or may not be the same as TENANT

Staging template (.envs/.<tenant>-staging/.web):

API_BASE_URL=https://homealign-dj.thb.sh/core-api
TENANT_LOGIN_URL=https://homealign-dj.thb.sh/tenant-login/
TENANT_REFRESH_URL=https://homealign-dj.thb.sh/tenant-refresh/
IDENTITY_SERVER_URL=https://auth.myhaapp.com
TENANT=<ALIAS>
POSTHOG_DEBUG=false
POSTHOG_HOST=https://us.i.posthog.com
POSTHOG_KEY=<copy from existing tenant>
SUPPORT_URL=<SUPPORT_URL>

Production template (.envs/.<tenant>-production/.web):

API_BASE_URL=https://homealign-dj.thehelperbees.com/core-api
TENANT_LOGIN_URL=https://homealign-dj.thehelperbees.com/tenant-login/
TENANT_REFRESH_URL=https://homealign-dj.thehelperbees.com/tenant-refresh/
IDENTITY_SERVER_URL=https://auth.myhomealign.com
TENANT=<ALIAS>
POSTHOG_DEBUG=false
POSTHOG_HOST=https://us.i.posthog.com
POSTHOG_KEY=<copy from existing tenant>
SUPPORT_URL=<SUPPORT_URL>

IF Sanity is in tenant.md:Features, add the following fields to production:

ENABLED_FEATURES=["service_photos"]
SANITY_PROJECT_ID=dqoma4hi
SANITY_DATASET=cp_dataset_prod
SANITY_API_VERSION=v1
SANITY_USE_CDN=true
SANITY_TOKEN=<copy from an existing Sanity tenant>
SANITY_CLIENT_NAMES=<CSV, from client_config.json cp_tenants[].sanity_client>

IF “Encounter Notes” is in tenant.md:Features:

ENABLED_FEATURES=["encounter_notes"]

IF ENABLED_FEATURES already exists, append to list, rather than replace

    ENABLED_FEATURES=["service_photos", "encounter_notes"]

SANITY_CLIENT_NAMES is the only per-tenant value; humana carries three (humana,humana-d2c,humana-longevity). Read it from django_homealign config/client_config.json under cp_tenants -> <name> -> sanity_client -> <ENV> -> sanity_client_name. If that is empty, STOP and ask ME.

Reference tenant: .envs/.humana-production/.web is the only one configured.

Only TENANT and SUPPORT_URL are tenant-specific. Copy POSTHOG_KEY from any existing tenant’s .env file. All other values are constant per environment.

SUPPORT_URL may not be correct. To verify:

curl -sS -m 15 -o /dev/null -w '%{http_code}\n' "https://<subdomain>.zendesk.com/api/v2/help_center/en-us/articles.json?per_page=1"

<subdomain> is the host from SUPPORT_URL, minus .zendesk.com.

• 200 -> pass
• 404 -> Error, that help center does not exist

Two naming patterns:

• legacy: <TENANT>-thb.zendesk.com
• current: thb<TENANT>.zendesk.com

Do NOT curl SUPPORT_URL itself. Zendesk answers 403 to any non-browser request, so every tenant looks identical — working or not. The API path is what discriminates. Confirm by running it against a name you invent: it also returns 404.

To make the new tenant accessible on the admin benefits hub, add it to AVAILABLE_TENANTS in

• .envs/.admin-production/.web
• .envs/.admin-staging/.web

Append: {"id":"<ALIAS>","name":"<TENANT_DISPLAY>"}

Add the tenant to the GitHub Actions deploy lists:

• .github/tenants.json, in the tenants array: { "id": "<TENANT>" },
• .github/workflows/deploy-staging.yml, the tenants dropdown under workflow_dispatch: - "<TENANT>"

Create a PR in the benefits-platform repo. See this PR for a reference example. If deploying a batch of tenants, make 1 PR for the whole batch per ENV.

Verification: sanity check only — that both files landed on benefits-platform:main.

git fetch -q origin
git show origin/main:.envs/.<TENANT>-<ENV>/.web | grep -E '^TENANT='
git show origin/main:.envs/.admin-<ENV>/.web | grep -o '"id":"<ALIAS>"'
git show origin/main:.github/tenants.json | grep -c '"id": "<TENANT>"'
git show origin/main:.github/workflows/deploy-staging.yml | grep -c -- '- "<TENANT>"'

TENANT= must be <ALIAS>, the admin file must list <ALIAS>, and both list counts must be 1. Any missing -> Error.

Phase 5: Add App to AWX Deployment

Step 7: Update AWX Job Template

VPN Must be On. Must get an AWX Write token

• <ENV> = staging ==> template 94 (Deployment Stage [Benefits Platform])
• <ENV> = production ==> template 98 (Deployment Prod [Benefits Platform])

Add "<TENANT>-benefits-platform" to the template’s survey

1. Back up the current survey, and confirm the fetch actually returned it:

   T=$(tr -d ' \r\n' < ~/.awx_api_token)
   curl -sS -m 30 -o ~/awx<id>-survey-backup.json -w '%{http_code}\n' \
       -H "Authorization: Bearer $T" \
       https://tools2.thb.dev/api/v2/job_templates/<id>/survey_spec/

   Not 200, or the file has no "spec" array -> STOP. Never POST without a backup.

2. Edit a COPY, not the backup. Read the app list off the question whose variable is app_name; choices may be a newline-joined string or a list, so handle both. Keep "benefits-platform" (the admin) FIRST — it is first by convention — then add <TENANT>-benefits-platform to the rest and sort.

3. POST the whole edited spec back, with the WRITE token:

   curl -sS -m 40 -w '%{http_code}\n' -X POST \
       -H "Authorization: Bearer $W" -H "Content-Type: application/json" \
       --data-binary @<edited file> \
       https://tools2.thb.dev/api/v2/job_templates/<id>/survey_spec/

Can always do this manually too.

Verification

Re-READ the survey and diff it against the backup. A 200 from the POST is not proof; the only proof is what the server hands back afterwards.

• app count went up by exactly 1
• <TENANT>-benefits-platform is present
• NOTHING was lost: the set difference backup-minus-new must be empty
• the question COUNT is unchanged (94 has 3, 98 has 2 — do not assume 3)

Any of those failing -> restore from the backup by POSTing it back, then Error.

Do not verify with grep -c on the raw response. A 401 or a Cloudflare 403 returns a body that greps to 0, which reads identically to “the entry is missing” — parse the JSON and compare sets instead.

Step 7b: Add the App to the GitHub->AWX Proxy Allowlist

Needed once per tenant (the allowlist is not per environment). The GitHub Actions deploys in benefits-platform (“Deploy to Staging” and “Release to Production”) reach AWX through the Cloud Run proxy gh-actions-proxy (infra/common-infra/jobs/gh_actions_proxy), which validates extra_vars.app_name against its own allowlist: a JSON parameter in GCP Parameter Manager that the container loads once at startup. A passing AWX survey is not enough. A missing entry fails the tenant’s leg with HTTP 400 extra_vars.app_name is not allowed, and in “Release to Production” that skips the production deploy for every stack.

The canonical procedure is section “6. Register the app_name in the proxy allowlist” of the shared-actions README; the steps below are the same procedure for a Benefits Hub tenant.

Access: roles/parametermanager.editor is granted to sg-developers@thehelperbees.com on the common-infra projects by gcp-projects business_unit_1/shared/security_groups/sg-developers.tf (resource sg_developer_parameter_manager_common_infra). Editor covers versions list, describe, and create. Do not use versions render; that needs the admin role.

SDK: gcloud parametermanager is GA only in recent SDKs. If gcloud answers Invalid choice: 'parametermanager', use gcloud beta parametermanager with the same arguments, or update the SDK.

1. Read the current list (needs gcloud auth login as an sg-developers member):

   P=prj-bu1-c-common-infra-c4aa
   gcloud parametermanager parameters versions list \
       --parameter=gh-actions-proxy-allowed-apps --location=global --project=$P \
       --sort-by=~createTime --limit=1 --format='value(name.basename())'
   gcloud parametermanager parameters versions describe <that version> \
       --parameter=gh-actions-proxy-allowed-apps --location=global --project=$P \
       --format='value(payload.data)' | base64 -d > allowlist-current.json

   Not valid JSON with an allowed_app_names array -> STOP.

2. Write a new file that is the current list plus "<TENANT>-benefits-platform", with nothing removed (never retype the list), then create the next version (<N> = highest existing v<number> + 1, from the versions list output):

   gcloud parametermanager parameters versions create v<N>-<TENANT>-benefits-platform \
       --parameter=gh-actions-proxy-allowed-apps --location=global --project=$P \
       --payload-data-from-file=allowlist-new.json

3. Redeploy the proxy so new instances load the new version. Dispatch the infrahive workflow Build and Deploy gh-actions-proxy on branch production and wait for it to finish:

   gh workflow run build-gh-actions-proxy.yml --repo thehelperbees/infrahive --ref production

Verification

Describe and decode the new version: the entry count went up by exactly 1, <TENANT>-benefits-platform is present, nothing was lost. Then confirm a new Cloud Run revision is ready and carries all traffic:

gcloud run services describe gh-actions-proxy --region=us-central1 --project=$P \
    --format='value(status.latestReadyRevisionName,status.traffic[0].revisionName,status.traffic[0].percent)'

The ready revision must be new (compare with the value from before the redeploy) and the same revision must show 100 percent of traffic. Do this before dispatching any GitHub Actions deploy for the tenant.

Step 8: Deploy via GitHub Actions

Input: <ENV> = {staging, production}

thehelperbees/benefits-platform

Wait for the Docker image of main’s latest commit to finish building first. Cloud Build starts it a few minutes after a merge, and AWX deploys only commits that have an image. Deploying too early fails every stack at AWX’s first check with Trying to deploy to a commit (<sha>) that does not have a docker image built, before anything is deployed. Done when this prints completed success:

gh api repos/thehelperbees/benefits-platform/commits/main/check-runs \
    --jq '.check_runs[] | select(.name | startswith("benefits-platform-docker-images")) | "\(.status) \(.conclusion)"'

If <ENV> = staging, deploy the tenant, then the admin:

gh workflow run deploy-staging.yml --repo thehelperbees/benefits-platform -f branch=main -f tenants=<TENANT>
gh workflow run deploy-staging.yml --repo thehelperbees/benefits-platform -f branch=main -f tenants=benefits-platform

Verification — watch Deploy to Staging. Done when both runs conclude success.

If <ENV> = production, deploy through the benefits-platform release. It merges main into release, then deploys every stack in .github/tenants.json to staging and then production.

1. Dry run, from benefits-platform, and review the commits

   ./merge_main_into_release.sh

2. Release

   ./merge_main_into_release.sh --run

   This ships main to production for every tenant, not just <TENANT>.

Verification — watch Release to Production. Done when the Slack release thread says live on all N stacks.

Phase 6: Uptime Monitoring

Step 9: Add the Production URL to Pingdom

Production only. Without a check, a Benefits Hub outage reaches us only when the partner reports it.

Open the Pingdom uptime checks list, signing in through SolarWinds, then add a check matching the rest of the fleet:

For <TENANT> = hap: name Benefits Hub Prod HAP, URL https://hapbenefits.thehelperbees.com/login, tags benefits-hub prod hap.

The tags are what route an alert, so a check saved without them is a check nobody hears.

Verification — the check appears under Up in the checks list and plots a response time within a few minutes.

Post-Launch Verification

Input: <ENV> = {staging, production}

benefits-platform:main

scripts/check-deployments.sh

<ENV> = staging ==> --staging, production ==> --prod. VPN ON for staging.

• Reachable
  • unreachable — no response at all; DNS, Cloudflare, or the container is down
  • (no version) — reachable but no commit in the payload
  • wrong version — serving a DIFFERENT commit than expected, i.e. the deploy ran but shipped stale code. A bare curl cannot catch this one.
• /api/ping
  • compose/ui/Caddyfile. 200 proves the container is up and serving.

Tenant appears in the admin instance’s dropdown

• <ENV> = staging ==> admin-benefits.thb.sh
• <ENV> = production ==> admin-benefits.thehelperbees.com

curl -sS -m 15 https://<admin host>/config.js | grep -c '"id":"<ALIAS>"'

• 1 -> pass
• 0 -> Error.

Application loads

Read the hosts from the tenant’s entry in infra/hb-infra/<ENV>.auto.tfvars: public_site_urls[] and admin_site_urls[].

• <ENV> = staging ==> one host (thb.sh)
• <ENV> = production ==> two hosts (thb.nu + thehelperbees.com)

For every URL listed:

curl -sS -m 15 -w '\nHTTP:%{http_code} TYPE:%{content_type}' https://<host>

Pass needs all of:

• HTTP 200
• content_type text/html
• <title>Benefits Hub</title>
• a hashed /assets/index-*.js bundle

Not 200 -> Error. “Bad gateway” means the container is still starting; retry.

IF <ENV> == production: the bundle hash must MATCH across both hosts.

Live tenant key is one Django accepts

Read it off the DEPLOYED site, not the repo.

1. Read the live value. Any one of the tenant’s hosts will do:

   • <ENV> = staging ==> <TENANT>benefits.thb.sh
   • <ENV> = production ==> <TENANT>benefits.thehelperbees.com

   curl -sS -m 15 https://<host>/config.js | grep -oE 'tenant: *"[^"]*"'

   It must be <ALIAS>. If it is <TENANT>, the container is serving stale config — redeploy, do not edit anything.

2. Hand that exact value to Django. Junk credentials are fine: the tenant key is checked BEFORE the password, so the status code alone answers it.

   • <ENV> = staging ==> homealign-dj.thb.sh
   • <ENV> = production ==> homealign-dj.thehelperbees.com

   curl -sS -o /dev/null -w '%{http_code}\n' -X POST https://<django host>/tenant-login/ \
       -H 'Content-Type: application/json' \
       -d '{"email":"probe@example.invalid","password":"x","tenant":"<value from config.js>"}'

   • 401 = Django accepts the key. Login is possible.
   • 400 = “Invalid tenant specified”. Login is IMPOSSIBLE for everyone, and no account will fix it. -> Error.

Use the value from step 1 verbatim, not <ALIAS> from # Info — the point is to test what the live site actually sends.

Visual Check, Human or Playwright

<ENV> == staging:

• <HUB_HOST> = <TENANT>benefits.thb.sh
• <DJANGO_HOST> = homealign-dj.thb.sh
• <TOKEN_FILE> = ~/.bh_refresh_token.staging

<ENV> == production:

• <HUB_HOST> = <TENANT>benefits.thehelperbees.com
• <DJANGO_HOST> = homealign-dj.thehelperbees.com
• <TOKEN_FILE> = ~/.bh_refresh_token.production

playwright test login_check.ts