Decommissioning a Legacy CoPo Partner

Step-by-step guide to retire a legacy Consumer Portal (CoPo) partner stack. The mirror of launching a legacy CoPo partner.

Status: draft. The generated-resource map in Appendix A is verified against this repo. The archive phase is newer and its commands have not all been run end to end. Correct this document as you work through it.

Why this is slower than it looks

Two properties make a tenant decommission riskier than deleting some config.

Resources are generated from a list, not declared. Every stack is one entry in the apps list in infra/pd-infra/<env>.auto.tfvars. Unrelated Terraform roots then build resources by comprehension over var.apps:

# pdp<N>/main.tf — the tenant's Postgres database
dbs_to_create = [for app in var.apps : app.name if contains(app.db_resources, local.base_name)]

# production/mssql.tf — two MSSQL databases and four users per stack
mssql_apps = [for app in var.apps : app.name if contains(app.db_resources, "mssql")]

# pdp<N>/main.tf — the stack's Okta app
okta_apps = flatten([for app in var.apps : app.okta_configuration if app.vm_resource_name == local.base_name])

None of these carry the stack name in a Terraform address you would grep for. Deleting the tfvars entry destroys all of them in one plan, databases included.

Names do not indicate ownership. A legacy hbcp_<tenant> stack and its CoPo3 hbtcp_<tenant> replacement share a vm_resource_name, so they share a root directory and a Cloud SQL instance. Resources get named after whichever stack created them first, which is often the one being removed.

Verified example: module.hbcp_unum_cloudsql_alerts in pdp20/monitoring.tf monitors pdp20-p-psql — the instance CoPo3 runs on — and pages PagerDuty. Its name says legacy; its job is CoPo3’s alerting. Deleting it by name match removes production alerting silently.

Decide disposition per resource by reading what it points at, never by name.

Prerequisites

• Traffic cut over and the replacement soaked in production
• Member of ssh-p-env@thehelperbees.com (IAP tunnel) and sg-developers@thehelperbees.com (compute.osAdminLogin) — both are needed to reach a swarm manager, and missing the second looks like a broken SSH key
• gcloud, Terraform plan access, AWX access, Okta admin
• A reviewer lined up for each terraform plan

Phase 1: Inventory

Nothing changes in this phase. Replace assumptions with observations — stacks are frequently half-decommissioned already by earlier undocumented work.

Step 1: Confirm the stack is running

Managers are swarm-mgr-1/2/3 in us-central1-a/b/c. Note they carry no environment infix, unlike swarm-p-worker-*.

gcloud compute ssh swarm-mgr-1 \
  --tunnel-through-iap \
  --zone=us-central1-a \
  --project=prj-bu1-p-hb-infra-1da6

# on the manager
sudo docker stack ls | grep hbcp_<tenant>

Verify: you know whether the stack is live or already gone.

Step 2: Confirm nobody is using it

Check 30 days of access logs for every host in the stack’s public_site_urls and admin_site_urls. Include non-browser traffic — partner callbacks and health checks break loudly but do not look like user sessions.

Step 3: Enumerate what Terraform generates

Read the tfvars entry and note each field that feeds a comprehension. See Appendix A. Then grep the stack’s root directory for modules named after it, and read each one to find out what it actually points at.

Step 4: Inventory the data

gcloud sql databases list --instance=<pdpN-p-psql> --project=<env-project>
gcloud sql databases list --instance=<pd-p-mssql>  --project=<env-project>

Record row counts per table. Phase 2 verifies the archive against these numbers.

Watch for databases that exist in Terraform but hold nothing because the app was pointed elsewhere. Unum’s .envs/.unum-production/.web sets WEB_DB_HOST to an Azure server, which makes its GCP MSSQL databases potentially vestigial.

Step 5: Enumerate inbound links

• ConsumerPortalConfigs.URL for the tenant’s client ID in the hA production DB
• CONSUMER_PORTAL_URLS in django_homealign
• IdentityServer ConsumerPortalUrl entries in healthAlignPMS
• Okta app assignments
• AWX job template surveys (staging 83, production 84)
• Data-team or BI pipelines reading the tenant’s databases

Gate: no live traffic, row counts recorded, inbound links enumerated.

Phase 2: Archive

Retiring a stack destroys data that cannot be recovered. This phase is what makes Phase 5 survivable.

The archive bucket retains for six years, and objects are immutable once written. Under the retention policy an object cannot be deleted or replaced until its age exceeds the retention period — including a replacement upload to the same path under versioning. A bad export cannot be corrected in place.

So this phase exports to a scratch bucket, verifies there, and only then copies into the archive. Nothing reaches the archive until it has been restored and checked.

The policy is deliberately unlocked, so a project admin can shorten or remove it if something does get frozen by mistake. Treat that as an incident recovery path, not a routine step.

Use any regional bucket you can write to and delete from for the scratch space; it holds member data, so it must not be public and should be emptied afterwards.

Step 6: Grant write access by CLI

Exports run as the source instance’s service agent, not as you. That agent needs roles/storage.objectAdmin on the scratch bucket, or the export fails with an error that does not name the account it was using.

This grant is deliberately not in the archive bucket’s Terraform: binding an instance service agent there ties the bucket to the lifecycle of the instances it archives, and a removed instance’s agent breaks every later apply.

SA=$(gcloud sql instances describe <instance> \
  --project=<env-project> \
  --format='value(serviceAccountEmailAddress)')

gcloud storage buckets add-iam-policy-binding \
  gs://<scratch-bucket> \
  --member="serviceAccount:$SA" \
  --role=roles/storage.objectAdmin

Step 7: Export to the scratch bucket

gcloud sql export sql <pdpN-p-psql> \
  gs://<scratch-bucket>/<tenant>/postgres/hbcp_<tenant>.sql.gz \
  --database=hbcp_<tenant> --project=<env-project>

gcloud sql export bak <pd-p-mssql> \
  gs://<scratch-bucket>/<tenant>/mssql/hbcp_<tenant>_web_db.bak \
  --database=hbcp_<tenant>_web_db --project=<env-project>

Also stage the tenant’s prefix from the shared media bucket, and the tenant config with secret values stripped. An archive is not a secret store — record secret names so they can be traced, and rotate the values instead.

Step 8: Verify, while the data can still be replaced

Verify every export by restoring it. A successful export proves the file was written, not that it can be read back — and after step 9 it cannot be corrected.

• Restore the Postgres dump into a scratch database and compare row counts against step 4.
• Restore each MSSQL .bak into a scratch SQL Server database and do the same. If step 4 found those databases empty — which happens when the tenant’s live web database is hosted outside GCP — record that instead, and skip the export rather than archiving an empty file that implies coverage it does not have.

A green exit code is not verification. If a restore fails, re-export into the scratch bucket and try again; nothing is frozen yet.

Step 9: Copy the verified archive in

Only verified artifacts move into the archive bucket. This is the point of no correction.

gcloud storage cp -r \
  gs://<scratch-bucket>/<tenant>/ \
  gs://bkt-pd-p-decommission-archive/<tenant>/<date>/

Step 10: Revoke, then write the manifest last

Revoke the export grant from the scratch bucket, then empty the scratch bucket.

gcloud storage buckets remove-iam-policy-binding \
  gs://<scratch-bucket> \
  --member="serviceAccount:$SA" \
  --role=roles/storage.objectAdmin

Write MANIFEST.md after the revocation, so it can record it, and upload it into the archive prefix last. It records: date, operator, every artifact with size and checksum, the source resource for each, the restore procedure, the grant made and revoked in steps 6 and 10, and an explicit list of what was deliberately not archived and why.

The manifest is immutable once uploaded, like everything else here. Later corrections go in a dated file alongside it — CORRECTION-<date>.md — rather than an edit. Say so in the manifest itself so the next reader knows to look for them.

Gate: every export verified by restore, archive copied in, manifest written.

Phase 3: Defuse the traps

Separate PRs, each a no-op in effect, so that whoever performs Phase 5 cannot hit a trap by grep.

Step 12: Rename misleading shared modules

moved {
  from = module.hbcp_<tenant>_cloudsql_alerts
  to   = module.pdp<N>_cloudsql_alerts
}

Add a one-line comment to each surviving resource in the stack’s directory naming which stack it serves.

Verify: terraform plan reports no changes. If it wants to destroy and recreate, the moved block is wrong — stop.

Phase 4: Stop serving and soak

Step 13: Remove the stack

sudo docker stack rm hbcp_<tenant>

Remove the stack from the AWX survey choices on templates 83 and 84 so nobody redeploys it underneath you. Leave Terraform, data, DNS and secrets alone — rollback at this point is a redeploy through AWX.

Step 14: Soak

One to two weeks minimum, spanning at least one full eligibility cycle. Watch the replacement portal’s error rates, hA workflows touching the tenant, eligibility ingestion, and scheduled jobs against the tenant’s databases.

Gate: a clean soak. Nothing has been destroyed yet, so a rollback is free.

Phase 5: Retire infrastructure

One change per PR. The point is that each destroy lands in a plan small enough to read.

Reject a plan that touches anything prefixed tcp_ or hbtcp_, the shared Cloud SQL instances, the shared cp-media bucket, partner-level eligibility ingestion in common-infra, or the alerts module renamed in step 12.

Step d also shrinks local.mssql_app_regex in production/mssql.tf, which builds the MSSQL instance’s authorized-networks list from compute address names. If this stack is the only one contributing its VM name, every matching external IP drops off that allowlist. Read that diff specifically.

This phase is not reversible. Phase 2 is what makes it survivable.

Phase 6: Clean up the trail

• Delete the tenant’s .envs/ and compose/ directories from consumer_portal
• Rotate every secret that was in those files — deleting a file does not revoke a token, and git history keeps it
• Mark the tenant retired in the consumer portal architecture docs
• Remove or repoint the inbound links from step 5
• Record anything deliberately retained — external databases, partner integrations, the archive itself — so a later pass does not assume this decommission handled it
• Correct this runbook with whatever it got wrong

Appendix A: Generated resource map

Never removed during a tenant decommission

• The shared Cloud SQL instances (pdp<N>-p-psql, pd-p-mssql)
• The shared media bucket — only the tenant’s prefix is in scope
• Partner eligibility SFTP jobs and their config secrets, in common-infra
• Anything tcp_* / bkt-*-tcp-* — that is CoPo3
• Cloud SQL alerting modules, whatever they are named