Infrastructure Drift

The infrastructure drift fuzzer detects when deployed cloud resources have diverged from the Terraform code that defines them. It runs terraform plan against every discovered module and reports any with a non-zero change count as drift. CFO invokes it on a schedule across all environment branches.

Running

Discover all modules:

./zig/zig build fuzz -- infra_drift --discover

Run a single module:

./zig/zig build fuzz -- infra_drift --infra=hb-infra --branch=production --module=vault

Dry run (print the plan command without executing):

./zig/zig build fuzz -- infra_drift --infra=hb-infra --branch=production --module=vault --dry-run

How It Works

CFO invokes the infra_drift binary in two stages. First, discovery mode enumerates all modules across all tracked branches and emits them as a JSON array. Second, CFO fans out — invoking infra_drift once per module in single-module mode, running plans concurrently across workers. Each single-module invocation runs the full detection pipeline and exits 0 (no drift) or 1 (drift found or execution error).

The development branch is excluded. Only production, shared, and non-production are checked.

Module Discovery

Discovery scans infra/ for directories with a -infra suffix, then walks each project directory looking for backend.tf files. Every directory containing backend.tf is a Terraform module. The path structure encodes three pieces of information:

infra/<project>/<business_unit>/<branch>/<module>/backend.tf

Discovery extracts the project, branch, and module name from the path, filters to the target branch, and returns the full list. Discovery always prints a JSON array. CFO treats a non-array, empty, or duplicated list as a broken discovery and stops, because it uses the discovered list to decide which modules were deleted.

Across all five infrastructure projects and three environment branches, discovery yields approximately 96 modules.

Deleted Modules

When a module is deleted, discovery stops producing a task for it, so its last failure would never be superseded by a pass. On every run that includes infra_drift, CFO drops stored infra_drift failures whose (project, branch, module) is not in the discovered list. The prune only persists when the run passes --upload. Only failures are dropped: passes are merged per commit and branch, so their command names an arbitrary module. Discovery walks the plan checkout, so a module deleted on plan but not yet promoted is treated as deleted on the env branches too; CFO has already stopped planning it there.

Infrastructure Projects

The Detection Pipeline

Each module runs through a three-stage pipeline:

flowchart LR
    A["tfo plan<project> <branch> <module>"] --> B["tfparse"]
    B --> C["jq '.[0].change_count'"]
    C --> D{"> 0?"}
    D -->|yes| E["DRIFT"]
    D -->|no| F["OK"]

tfo plan runs terraform plan through TFO with service account impersonation. tfparse parses the plan output into structured JSON. jq extracts the change_count field. Any value greater than zero means the deployed infrastructure no longer matches the code.

The pipeline runs as a bash one-liner with pipefail set so any stage failure propagates as an error:

tfo plan <project> <branch> <module> | tfparse | jq '.[0].change_count'

Transient API failures (gateway timeouts, TLS errors, provider download failures) are retried up to three times with exponential backoff before the module is recorded as an error.

Source

• src/fuzz_tests/infra_drift/main.go — entry point, CLI flags, discovery mode
• src/fuzz_tests/infra_drift/runner.go — single-plan execution and retry logic
• src/fuzz_tests/infra_drift/discovery.go — module discovery and path parsing