GitHub

build_metrics

build_metrics collects CI build outcomes into BigQuery so build duration, queue time, and failure rate can be tracked over time, the same way dora tracks deploy frequency and lead time.

Source: src/scripts/build_metrics/ (collector), infra/common-infra/business_unit_1/shared/build_metrics.tf (dataset, views, Cloud Run job, IAM).

What it is

build-metrics-collect is a daily Cloud Run job (prj-bu1-c-common-infra-c4aa, us-central1) that polls two build platforms and writes terminal build outcomes into build_metrics.build_events:

  • Cloud Build, in the app-infra pipeline project (prj-bu1-c-app-infra-98a6).
  • Azure DevOps Builds API, for the HealthAlign PMS project (ADO_PROJECT = "HealthAlign PMS", org thehelperbees).

build_events is the raw append-only ledger, keyed by event_id (<platform>:<native build id>). Two BigQuery views derive from it, same naming convention as dora:

  • builds (one row per build, deduped on event_id, latest collected_at wins): adds duration_seconds, queue_seconds, and pipeline_category, computed in the view. Categories: image_build (all Cloud Build, plus ADO docker/database build definitions), release (ADO *-Release-* deploy pipelines), and ops (SyncTenant tenant automation and anything unclassified, including NULL definition names). SyncTenant pipelines are forced into ops even when their names contain Docker or Release.
  • build_times (rolling weekly, ~12 trailing weeks, windows anchored to today): p50/p85 duration, p50 queue time, build count (n), n_timed (rows with a measurable duration; quantiles skip builds canceled before start), and failed_count (status != 'success'), grouped by window_end, platform, app_name, pipeline_category.

build_events also carries the ADO run name (build_number) and the raw pipeline parameters (template_parameters). ha_deploys and ha_deploy_times derive HA full-orchestration deploy timing from those fields.

template_parameters is stored raw because ADO’s values need defensive reading, which is why ha_deploys wraps every lookup in LOWER(IFNULL(JSON_VALUE(...), 'false')):

  • An absent key means the pipeline default, not false. Runs predating a parameter omit it entirely (warmOnly is missing on the earliest ALL-Release-* runs, and run 11716 carries only skipDatabase), and a run queued with every parameter left at its default can omit them all.
  • Booleans arrive inconsistently. Across the first 50 ALL-Release-* runs the same key appears as "true", "True", "false", "False" and as real JSON booleans, so a bare = 'true' comparison silently misclassifies.

The Grafana Build Metrics dashboard (src/services/observability/grafana/dashboards/BuildMetrics/build_metrics.json) reads the builds view (build_times only feeds the Platform/App dropdowns), same cross-project pattern as the DORA dashboards: the swarm manager SA has bigquery.dataViewer on the build_metrics dataset and queries it directly.

Auth

  • GCP: the job runs as build-metrics-sa, with bigquery.dataEditor on the build_metrics dataset, bigquery.jobUser on its own project, and cloudbuild.builds.viewer on the app-infra project (cross-project, so it can list/describe Cloud Build history there).
  • ADO: the collector authenticates as the existing thb-ado-terraform-sp service principal (client ID af82b357-0019-49ec-859c-c7dc13cc6c76, Helper Bees tenant), the same identity ha-infra’s azuredevops Terraform provider uses. The client secret is ado_sp_client_secret in the common secrets project (prj-c-secrets-a7cc); build-metrics-sa only has secretAccessor on it. There’s no separate PAT, and rotation rides whatever process already rotates that secret for ha-infra.

Operations

Manual run:

gcloud run jobs execute build-metrics-collect --region us-central1 --project prj-bu1-c-common-infra-c4aa --wait

Deploying collector changes: merging to plan rebuilds and pushes gcr.io/the-helper-bees/build-metrics:latest (workflow build-build-metrics-image.yml), but Cloud Run resolves the tag to a digest when the job is created or updated, not at each execution. After an image push, refresh the job so the next run picks it up:

gcloud run jobs update build-metrics-collect --region us-central1 --project prj-bu1-c-common-infra-c4aa \
  --image gcr.io/the-helper-bees/build-metrics:latest

Backfill (recent window): pass --backfill-since as a one-shot execution override to collect from that date forward without mutating the job spec:

gcloud run jobs execute build-metrics-collect --region us-central1 --project prj-bu1-c-common-infra-c4aa \
  --args=--backfill-since=2024-06-01 --wait

This only works for recent windows: BigQuery’s streaming insertAll (what the job uses) cannot write into partitions older than roughly a year. Since the override is passed per-execution (not persisted to the job spec), the next scheduled run automatically returns to the normal daily watermark; there’s nothing to unset.

Backfill (deep history): for anything older than roughly a year, --backfill-since job runs will fail on the old partitions. Instead, run the collector locally with --dump-file to write mapped rows as NDJSON, then bq load that file into build_events directly (bq load writes partitions of any age; only streaming inserts have the ~1-year floor):

./zig/zig build scripts -- build_metrics --dump-file /tmp/build_events.ndjson --backfill-since 2024-06-01
bq load --source_format=NEWLINE_DELIMITED_JSON \
  prj-bu1-c-common-infra-c4aa:build_metrics.build_events /tmp/build_events.ndjson

Cloud Build history only reaches back to roughly May 2024; ADO’s Builds API only retains roughly the last 41 days regardless of the requested start.

Re-ingesting rows that already exist: a plain --backfill-since job run will NOT update rows already in the table. Insert mode filters out rows whose event_id already exists, so it reports duplicates skipped and changes nothing. To add newly-collected fields to existing rows, use --dump-file (which exits before that dedup) plus bq load, which appends unconditionally; the builds view then surfaces the newer row via its ROW_NUMBER() ... ORDER BY collected_at DESC dedupe:

./zig/zig build scripts -- build_metrics --dump-file /tmp/backfill.ndjson --backfill-since 2026-08-24
grep '"platform": "ado"' /tmp/backfill.ndjson > /tmp/backfill_ado.ndjson
bq load --source_format=NEWLINE_DELIMITED_JSON \
  prj-bu1-c-common-infra-c4aa:build_metrics.build_events /tmp/backfill_ado.ndjson

The grep keeps the load from also duplicating untouched Cloud Build rows. Run this away from the 06:00 America/Chicago collector window to avoid the streaming buffer.

Local dry run (no BigQuery insert, prints the first few mapped rows per platform to stdout):

./zig/zig build scripts -- build_metrics --dry-run

Needs the same required env vars as the job (GCP_PROJECT_ID, CLOUD_BUILD_PROJECT, ADO_ORG, ADO_PROJECT, ADO_CLIENT_ID, ADO_TENANT_ID, ADO_CLIENT_SECRET) plus application default credentials for the GCP side.

Local dump (no BigQuery insert, writes mapped rows as NDJSON instead):

./zig/zig build scripts -- build_metrics --dump-file /tmp/build_events.ndjson

--dry-run and --dump-file are mutually exclusive; the collector exits with an argument error if both are set.

Weekly HA deploy times report: src/scripts/ha_deploy_times_report.py posts a per-deploy summary of the last completed week’s full-fleet HA release deploys (from build_metrics.ha_deploys) to #devops, on the same Monday Cloud Scheduler cadence as the build times report (infra/common-infra/business_unit_1/shared/ha_deploy_report.tf). It shares the build_metrics.report_posts ledger with build_times_report.py, keyed by the report column (ha_deploy_times vs build_times), so the two reports coexist in one idempotency table without colliding. Run it manually:

./zig/zig build scripts -- ha_deploy_times_report --dry-run
./zig/zig build scripts -- ha_deploy_times_report --week-ending 2026-08-23
./zig/zig build scripts -- ha_deploy_times_report --force

--dry-run prints the report without posting to Slack, --week-ending re-runs a specific past week (must be a Sunday), and --force reposts even if that week’s report already exists in the ledger for the target channel.

Monitoring

Dead Man’s Snitch checks in once per successful run (SNITCH_URL, set after the job’s snitch is created in the deadmanssnitch.com UI, same manual-setup pattern as swarmctl). The check-in is disabled until SNITCH_URL is set: an empty/unset value is a no-op, not an error, so the job runs fine before the snitch exists. No ping within a day (once SNITCH_URL is set) means the collector failed or didn’t run: check Cloud Run job execution logs first, since a Cloud Build or ADO fetch error still exits non-zero and skips the check-in.

Boundaries

  • No pipeline YAML changes. This only reads build history after the fact; it doesn’t touch Cloud Build trigger config or ADO pipeline definitions.
  • Infra-pipeline Terraform builds (the Cloud Build jobs that plan/apply infrahive itself) are out of scope here; devhive_metrics.py covers those separately.
  • dora.deploy_events remains the source of truth for deploy frequency and lead time (one event per deployed service per tenant). build_metrics.ha_deploys measures something different: the wall-clock duration of the single ALL-Release-* orchestration run. They are not substitutes and should not be joined for counting.
  • DORA dashboards: the sibling metrics pipeline this mirrors (raw ledger + curated views + metric views).
  • infra/common-infra/business_unit_1/shared/dora.tf: the naming convention build_metrics.tf follows.
  • Secrets Management: how ado_sp_client_secret and other GCP Secret Manager secrets are stored and rotated.
Edit this page