Recreate an Azure DevOps Environment

Rebuild a corrupt Azure DevOps Environment (e.g. Staging, Production, Development in the HealthAlign PMS project) by renaming the broken record aside, creating a fresh one with the same name, and re-registering every swarm-manager agent into it.

Use this runbook when:

• A new Linux swarm manager fails to provision with its SetupVM extension reporting VMExtensionProvisioningError, and /var/log/setup-vm.log on the VM ends with:

  Failed to add virtual machine resource. Linked environment pool is null.

• More generally, when adding an agent to an existing Environment fails with “Linked environment pool is null” even though existing resources in that Environment still deploy fine.

Why this happens

Swarm managers register an on-VM Azure DevOps agent as an Environment → Virtual machines resource during bootstrap (infra/ha-infra/scripts/linux/bootstrap/azagent.sh, run only when is_swarm_manager = true). It authenticates with the thb-ado-terraform-sp service principal and targets the Environment name mapped from the deployment environment:

Legacy Environment records (created early, low environment ID) can end up with a null backing deployment-pool link. Existing resources keep working, but the server rejects new VM-resource registrations with “Linked environment pool is null.” A freshly-created Environment usually gets a correctly-linked pool, so recreating it is the durable fix.

The service principal is not necessarily the problem. thb-ado-terraform-sp registers environment VM resources fine against a cleanly-provisioned environment (verified on Development). “Linked environment pool is null” is a property of a specific environment’s broken pool — and in practice even a freshly-recreated environment may still reject SP registration while a PAT succeeds. So treat SP registration as best-effort and fall back to a PAT (Stage 3c). The bootstrap (azagent.sh) treats a failed registration as non-fatal — the VM provisions cleanly and logs a warning to /var/log/setup-vm.log — so a manager whose agent didn’t register still comes up healthy, and you register it out-of-band afterward (Stage 4).

This is manager-only. Windows workers (*-WIN) skip agent registration, so they are unaffected — do not run this for a worker failure.

Prerequisites

• ADO Administrator on the target Environment (Environment → ⋮ → Security) and on its deployment pool (Org Settings → Deployment pools → Security) for the thb-ado-terraform-sp service principal
• SSH access to each affected Linux manager (see SSH Into Azure VMs)
• The thb-ado-terraform-sp credentials:
  • client_id af82b357-0019-49ec-859c-c7dc13cc6c76, tenant_id 00ff0ed4-7e61-48b7-aac1-14848f6a3574 (both in infra/ha-infra/common.auto.tfvars)
  • client secret in GCP Secret Manager: secret ado_sp_client_secret in project prj-c-secrets-a7cc
• A maintenance window for that environment — deployments to it are down from the moment you rename until every manager is re-registered
• Terraform apply access (only if you need to re-run a manager’s IaC bootstrap in Stage 4)

Never paste the client secret into a terminal literal, a chat, or a commit. Always load it into $SECRET via the gcloud secrets fetch shown below. If it is ever exposed, rotate it (Entra → app registration thb-ado-terraform-sp → Certificates & secrets; add a new version to ado_sp_client_secret).

Procedure

Substitute <ENV> (e.g. Staging), <DEPLOY_ENV> (e.g. non-production), and each manager <VM_NAME> (e.g. THB-N-WEB2-LINUX) throughout.

Stage 0 — Rule out the cheap fix first

Recreating is disruptive. First confirm it is actually the corrupt-record case and not a plain permission gap:

• Environment → ⋮ → Security: is thb-ado-terraform-sp present as Administrator?
• Org Settings → Deployment pools → the pool behind <ENV> → Security: is thb-ado-terraform-sp an Administrator there too?

If the SP is missing/under-privileged on the pool, grant it and re-run the manager bootstrap — that fixes it with no recreate and no downtime. Only proceed with the recreate if the SP already has Administrator on both and registration still fails.

Stage 1 — Rename the broken environment

In ADO → Pipelines → Environments → <ENV> → ⋮ → Edit:

• Rename <ENV> → Old-<ENV> (e.g. Old-Staging)
• Add a description noting why (e.g. “Obsolete legacy record; backing pool link broken for new VM-resource registrations.”)

Renaming frees the name for a fresh environment while preserving the old record (and its existing registrations) as a rollback. Names are unique per project, so the rename must happen before the new one exists.

Back up first: screenshot the Environment’s Approvals and checks, Security roster, and Pipeline permissions (which pipelines are authorized to use it). A recreated environment gets a new ID and starts with none of these.

Stage 2 — Create the fresh environment

Either create it in the UI (New environment, name it exactly <ENV>, resource type None), or simply let the first agent registration in Stage 3 auto-create it (config.sh --environmentname "<ENV>" creates the environment and its pool if absent). Re-add everything you captured: the Security roster (thb-ado-terraform-sp as Administrator), approvals/checks, and pipeline permissions.

Pipelines resolve the environment by name (environment: <ENV>), so no YAML edits are needed — but the recreated environment has a new ID, so pipeline authorizations do not carry over. Re-authorize each pipeline that deploys to <ENV> (Environment → Security → Pipeline permissions → add the pipelines, or enable open access), and confirm a real deployment before deleting Old-<ENV> (Stage 5).

Stage 3 — Re-register each manager agent

Run this on each affected manager, one at a time, verifying each before moving on. The agent must live in /azagent (the IaC/bootstrap location) — do not create a second copy under ~.

3a — Inventory every agent first (do not assume there’s only one)

A manager can carry more than one agent registration — a live Environment agent plus a stale Deployment Group agent left over from an earlier bootstrap (deployment groups are deprecated; all pipelines moved to YAML/environments). They can live in different directories (/azagent vs ~/azagent), so removing “the” agent blindly leaves a second one running.

Confirm the box, then list every agent service and trace each one:

hostname                                                   # confirm the intended VM
systemctl list-units --type=service --all | grep -i vsts   # may show 1, 2, or more

For each service listed, find its backing directory and what it’s registered to:

systemctl cat '<full-service-name>' | grep -iE 'ExecStart|WorkingDirectory'   # → the agent dir
cat <that-dir>/.agent                                                          # agentName / poolName / serverUrl

Read the .agent JSON: a deploymentGroupId field (and an old visualstudio.com serverUrl) marks a legacy Deployment Group agent — remove it, don’t keep it. An Environment agent has no deploymentGroupId. You must remove all of them before re-registering, or the box ends up with duplicate/competing agents.

3b — Remove every agent found, then re-register once

For each agent directory identified above (commonly /azagent, and ~/azagent if a stray exists):

SECRET=$(gcloud secrets versions access latest --secret=ado_sp_client_secret --project=prj-c-secrets-a7cc)

cd <agent-dir>                       # repeat this block for every dir from 3a
sudo ./svc.sh stop && sudo ./svc.sh uninstall
# Pass the secret via env var (VSTS_AGENT_INPUT_CLIENTSECRET), not argv, so it
# never appears in `ps`.
VSTS_AGENT_INPUT_CLIENTSECRET="$SECRET" ./config.sh remove --auth SP \
  --clientid af82b357-0019-49ec-859c-c7dc13cc6c76 \
  --tenantid 00ff0ed4-7e61-48b7-aac1-14848f6a3574

A legacy deployment-group agent (old visualstudio.com server) may fail config.sh remove with SP auth — that’s fine, it still clears the local .agent/.credentials; if it errors hard, rm -f <dir>/.agent <dir>/.credentials. Then confirm you’re at zero before re-registering:

systemctl list-units --type=service --all | grep -i vsts    # MUST be empty

Now register a single fresh agent from /azagent, and delete any stray non-/azagent dir afterward (rm -rf ~/azagent):

cd /azagent

# Register into the fresh environment (secret via env var, not argv)
VSTS_AGENT_INPUT_CLIENTSECRET="$SECRET" ./config.sh --unattended --environment --environmentname "<ENV>" \
  --acceptteeeula --agent "$HOSTNAME" \
  --url https://dev.azure.com/thehelperbees/ \
  --work _work --projectname "HealthAlign PMS" \
  --runasservice --replace \
  --addvirtualmachineresourcetags --virtualmachineresourcetags "$HOSTNAME" \
  --auth SP \
  --clientid af82b357-0019-49ec-859c-c7dc13cc6c76 \
  --tenantid 00ff0ed4-7e61-48b7-aac1-14848f6a3574

sudo ./svc.sh install HALocalAdmin && sudo ./svc.sh start && sudo ./svc.sh status
unset SECRET   # done with the secret

Expected: config.sh completes with no “Linked environment pool is null”; svc.sh status shows “Listening for Jobs”; and the manager appears online in <ENV> → Resources. (--virtualmachineresourcetags "$HOSTNAME" tags the resource with the box name so pipelines can target it.)

3c — If SP registration fails: fall back to a PAT

If config.sh above fails with Failed to add virtual machine resource. Linked environment pool is null (HTTP 500), the environment’s pool won’t accept the SP. This is environment-specific (the SP works elsewhere), and the reliable fallback is a PAT:

1. In ADO, open the environment → Resources → Add resource → Virtual machines. ADO generates a registration script containing a short-lived PAT (--token …) — it mints this on the fly; it is not the SP secret and expires in a few hours.
2. Re-run the registration from /azagent with that token, keeping the rest of the flags (do not run the raw UI script — it does a relative mkdir azagent that recreates the ~/azagent split-brain and pulls a different agent version):

cd /azagent
# Token via env var (VSTS_AGENT_INPUT_TOKEN), not argv, so it stays out of `ps`.
VSTS_AGENT_INPUT_TOKEN='<UI-generated-token>' ./config.sh --unattended --environment --environmentname "<ENV>" \
  --acceptteeeula --agent "$HOSTNAME" \
  --url https://dev.azure.com/thehelperbees/ \
  --work _work --projectname "HealthAlign PMS" \
  --runasservice --replace \
  --addvirtualmachineresourcetags --virtualmachineresourcetags "$HOSTNAME" \
  --auth PAT
sudo ./svc.sh install HALocalAdmin && sudo ./svc.sh start && sudo ./svc.sh status

Treat the token as a secret — don’t paste it into chat or commits. (A personal PAT with Agent Pools (Read & manage) scope works too — that’s the documented scope for agent registration; if VM-environment registration still fails, Microsoft suggests scoping the PAT to All accessible organizations.)

Confirm exactly one agent service is present after each box:

systemctl list-units --type=service --all | grep -i vsts   # one entry, active (running)

Stage 4 — Finish a manager whose bootstrap couldn’t register its agent

azagent.sh treats a failed registration as non-fatal: if the environment rejects the agent during bootstrap, it logs a warning to /var/log/setup-vm.log and exits 0, so setup_vm still succeeds and the VM comes up as a healthy swarm node — just not registered for deployments. Look for:

### WARNING: Azure DevOps agent registration FAILED (non-fatal)

To finish it, just register the agent out-of-band — no taint or re-apply needed:

• SSH to the manager and follow Stage 3b (SP) or Stage 3c (PAT, if the environment rejects the SP).
• Confirm it shows online in <ENV> → Resources and systemctl list-units --type=service --all | grep -i vsts shows one running vsts.agent.* service.

Recovering a genuinely failed/tainted setup_vm

Only needed if setup_vm itself is in a failed/tainted state (e.g. a manager provisioned before the non-fatal change, or a non-registration failure). Register the agent first (Stage 3b/3c) so azagent.sh’s skip-logic passes on the re-run, then re-create the extension.

Development — local apply is allowed. zig build apply applies the whole environment (it can’t target one resource), so gate it on the plan:

./zig/zig build plan -- ha-infra development

Review the plan — it should contain only:

module.managers["<VM_NAME>"].azurerm_virtual_machine_extension.setup_vm   (create/replace)

If any other resource appears (unrelated drift, another manager, a destroy), stop and reconcile first — don’t run a broad apply that sweeps in unintended changes. Only when the plan is scoped to that one setup_vm:

./zig/zig build apply -- ha-infra development

Non-production / production — local apply is blocked (these applies run in Cloud Build). Delete the failed SetupVM extension out-of-band so Terraform recreates it on the next apply:

# Resource group: non-prod = THB-UAT-RG, prod = THB-PROD-RG
az vm extension delete \
  --resource-group <THB-UAT-RG|THB-PROD-RG> \
  --vm-name <VM_NAME> \
  --name SetupVM

Then re-trigger the environment’s apply in Cloud Build. Terraform sees SetupVM missing and recreates it; because the agent is already registered, azagent.sh skips and the extension goes green. (Deleting the extension only removes the CustomScript resource — swarm/docker set up by the earlier run persist.)

Verify: setup_vm shows Creation complete; the manager is online in <ENV> → Resources; and systemctl list-units --type=service --all | grep -i vsts shows one running vsts.agent.* service.

Stage 5 — Verify, then delete the old environment

Once every manager is online in the fresh <ENV>:

• Verify a real deployment to <ENV> succeeds end-to-end (e.g. a WebApp release) — this confirms both agent registration and pipeline authorization (Stage 2) are in place.
• Inventory Old-<ENV> before deleting it. Its Resources tab should list only the migrated swarm-manager agents (now offline). If anything else is registered there — a manually-added agent, a non-manager resource — account for it first; deletion is irreversible and destroys your rollback record.
• Delete Old-<ENV> (⋮ → Delete) once both checks are clean.

Troubleshooting

SP registration fails with “Linked environment pool is null”

The target environment’s backing pool won’t accept the service principal. It’s environment-specific — the same SP works against cleanly-provisioned environments — and it can happen even on a freshly-recreated environment. Register the agent with a PAT instead (Stage 3c). Rotating the SP secret does not help (it’s not an auth failure — the request connects, then 500s on the pool).

config.sh remove says “Does not exist. Skipping”

Expected for an Environment VM resource — the CLI’s pool-based removal doesn’t delete the environment-side entry, so the resource stays listed in the UI. Remove it with <ENV> → Resources → ⋮ → Delete. (Local .agent/.credentials are still cleaned by the command.)

systemctl disable says the unit “does not exist” but list-units shows it

Unit names escape literal - to \x2d, so systemctl disable <plain-dashes> doesn’t match the on-disk file. Remove the unit directory-independently with a glob:

sudo systemctl stop 'vsts.agent.thehelperbees.<ENV>.*<VMshort>*LINUX.service' 2>/dev/null || true
sudo rm -f /etc/systemd/system/vsts.agent.thehelperbees.<ENV>.*<VMshort>*LINUX.service
sudo rm -f /etc/systemd/system/multi-user.target.wants/vsts.agent.thehelperbees.<ENV>.*<VMshort>*LINUX.service
sudo systemctl daemon-reload && sudo systemctl reset-failed

The shell glob on rm matches the escaped filename on disk, which is why this works where systemctl disable didn’t. If you prefer an exact target, copy the full escaped unit name verbatim from systemctl list-units … | grep -i vsts and pass that quoted name to systemctl stop/disable and rm instead of the glob.

svc.sh keeps acting on a stale/old unit, or an agent appears in two places

svc.sh reads the service name from the .service file in its current directory. If you ran config.sh from ~ you created a second agent under ~/azagent (split-brain) and svc.sh in /azagent targets the wrong (often already-removed) unit. Fix: uninstall the real unit by file (see above), rm -rf ~/azagent, and re-register strictly from /azagent.

A manager has two (or more) agents

Seen in the wild: an Environment agent in ~/azagent and a stale Deployment Group agent in /azagent (or vice-versa), both running. Don’t migrate just one — see Stage 3a to inventory every vsts.* service, trace each to its directory (systemctl cat … | grep ExecStart) and its .agent (a deploymentGroupId = legacy DG agent), remove all of them to zero, then register a single fresh agent from /azagent. Leftover deployment groups can be deleted from ADO afterward (Org/Project Settings → Deployment groups).

“Cannot configure because it is already configured”

A prior registration succeeded. Run ./config.sh remove … first (as HALocalAdmin, not sudo), then re-register.

Agent registers but stays offline / “inactive dead”

The service isn’t started. cd /azagent && sudo ./svc.sh start && sudo ./svc.sh status and look for “Listening for Jobs.”

Validation checklist

• SP confirmed Administrator on both the Environment and its deployment pool (Stage 0)
• Legacy environment renamed to Old-<ENV>; approvals/checks + security captured beforehand
• Fresh <ENV> exists with thb-ado-terraform-sp as Administrator
• Every manager re-registered: config.sh clean, svc.sh status “Listening for Jobs”, one vsts service running
• Every manager shows online under <ENV> → Resources
• Any bootstrap-failed manager re-applied via taint (Stage 4)
• One deployment to <ENV> succeeds end-to-end
• Old-<ENV> deleted
• Client secret never exposed; rotated if it was

The agent must live in /azagent. Running config.sh or the ADO UI-generated command from a home directory creates ~/azagent, which diverges from what the IaC bootstrap manages and causes svc.sh to operate on the wrong systemd unit. Always cd /azagent first.