GitHub

build_metrics

build_metrics collects CI build outcomes into BigQuery so build duration, queue time, and failure rate can be tracked over time, the same way dora tracks deploy frequency and lead time.

Source: src/scripts/build_metrics/ (collector), infra/common-infra/business_unit_1/shared/build_metrics.tf (dataset, views, Cloud Run job, IAM).

What it is

build-metrics-collect is a daily Cloud Run job (prj-bu1-c-common-infra-c4aa, us-central1) that polls two build platforms and writes terminal build outcomes into build_metrics.build_events:

  • Cloud Build, in the app-infra pipeline project (prj-bu1-c-app-infra-98a6).
  • Azure DevOps Builds API, for the HealthAlign PMS project (ADO_PROJECT = "HealthAlign PMS", org thehelperbees).

build_events is the raw append-only ledger, keyed by event_id (<platform>:<native build id>). Two BigQuery views derive from it, same naming convention as dora:

  • builds (one row per build, deduped on event_id, latest collected_at wins): adds duration_seconds, queue_seconds, and pipeline_category, computed in the view. Categories: image_build (all Cloud Build, plus ADO docker/database build definitions), release (ADO *-Release-* deploy pipelines), and ops (SyncTenant tenant automation and anything unclassified, including NULL definition names). SyncTenant pipelines are forced into ops even when their names contain Docker or Release.
  • build_times (rolling weekly, ~12 trailing weeks, windows anchored to today): p50/p85 duration, p50 queue time, build count (n), n_timed (rows with a measurable duration; quantiles skip builds canceled before start), and failed_count (status != 'success'), grouped by window_end, platform, app_name, pipeline_category.

The Grafana Build Metrics dashboard (src/services/observability/grafana/dashboards/BuildMetrics/build_metrics.json) reads the builds view (build_times only feeds the Platform/App dropdowns), same cross-project pattern as the DORA dashboards: the swarm manager SA has bigquery.dataViewer on the build_metrics dataset and queries it directly.

Auth

  • GCP: the job runs as build-metrics-sa, with bigquery.dataEditor on the build_metrics dataset, bigquery.jobUser on its own project, and cloudbuild.builds.viewer on the app-infra project (cross-project, so it can list/describe Cloud Build history there).
  • ADO: the collector authenticates as the existing thb-ado-terraform-sp service principal (client ID af82b357-0019-49ec-859c-c7dc13cc6c76, Helper Bees tenant), the same identity ha-infra’s azuredevops Terraform provider uses. The client secret is ado_sp_client_secret in the common secrets project (prj-c-secrets-a7cc); build-metrics-sa only has secretAccessor on it. There’s no separate PAT, and rotation rides whatever process already rotates that secret for ha-infra.

Operations

Manual run:

gcloud run jobs execute build-metrics-collect --region us-central1 --project prj-bu1-c-common-infra-c4aa --wait

Deploying collector changes: merging to plan rebuilds and pushes gcr.io/the-helper-bees/build-metrics:latest (workflow build-build-metrics-image.yml), but Cloud Run resolves the tag to a digest when the job is created or updated, not at each execution. After an image push, refresh the job so the next run picks it up:

gcloud run jobs update build-metrics-collect --region us-central1 --project prj-bu1-c-common-infra-c4aa \
  --image gcr.io/the-helper-bees/build-metrics:latest

Backfill (recent window): pass --backfill-since as a one-shot execution override to collect from that date forward without mutating the job spec:

gcloud run jobs execute build-metrics-collect --region us-central1 --project prj-bu1-c-common-infra-c4aa \
  --args=--backfill-since=2024-06-01 --wait

This only works for recent windows: BigQuery’s streaming insertAll (what the job uses) cannot write into partitions older than roughly a year. Since the override is passed per-execution (not persisted to the job spec), the next scheduled run automatically returns to the normal daily watermark; there’s nothing to unset.

Backfill (deep history): for anything older than roughly a year, --backfill-since job runs will fail on the old partitions. Instead, run the collector locally with --dump-file to write mapped rows as NDJSON, then bq load that file into build_events directly (bq load writes partitions of any age; only streaming inserts have the ~1-year floor):

./zig/zig build scripts -- build_metrics --dump-file /tmp/build_events.ndjson --backfill-since 2024-06-01
bq load --source_format=NEWLINE_DELIMITED_JSON \
  prj-bu1-c-common-infra-c4aa:build_metrics.build_events /tmp/build_events.ndjson

Cloud Build history only reaches back to roughly May 2024; ADO’s Builds API only retains roughly the last 41 days regardless of the requested start.

Local dry run (no BigQuery insert, prints the first few mapped rows per platform to stdout):

./zig/zig build scripts -- build_metrics --dry-run

Needs the same required env vars as the job (GCP_PROJECT_ID, CLOUD_BUILD_PROJECT, ADO_ORG, ADO_PROJECT, ADO_CLIENT_ID, ADO_TENANT_ID, ADO_CLIENT_SECRET) plus application default credentials for the GCP side.

Local dump (no BigQuery insert, writes mapped rows as NDJSON instead):

./zig/zig build scripts -- build_metrics --dump-file /tmp/build_events.ndjson

--dry-run and --dump-file are mutually exclusive; the collector exits with an argument error if both are set.

Monitoring

Dead Man’s Snitch checks in once per successful run (SNITCH_URL, set after the job’s snitch is created in the deadmanssnitch.com UI, same manual-setup pattern as swarmctl). The check-in is disabled until SNITCH_URL is set: an empty/unset value is a no-op, not an error, so the job runs fine before the snitch exists. No ping within a day (once SNITCH_URL is set) means the collector failed or didn’t run: check Cloud Run job execution logs first, since a Cloud Build or ADO fetch error still exits non-zero and skips the check-in.

Boundaries

  • No pipeline YAML changes. This only reads build history after the fact; it doesn’t touch Cloud Build trigger config or ADO pipeline definitions.
  • Infra-pipeline Terraform builds (the Cloud Build jobs that plan/apply infrahive itself) are out of scope here; devhive_metrics.py covers those separately.
  • dora.deploy_events remains the source of truth for deploy frequency and lead time. build_metrics only measures the build step, not the deploy.
  • DORA dashboards: the sibling metrics pipeline this mirrors (raw ledger + curated views + metric views).
  • infra/common-infra/business_unit_1/shared/dora.tf: the naming convention build_metrics.tf follows.
  • Secrets Management: how ado_sp_client_secret and other GCP Secret Manager secrets are stored and rotated.
Edit this page