GitHub

Build System

The repository uses a custom Zig build system that serves as the single interface for all operations: ./zig/zig build It downloads, pins, and cryptographically verifies every toolchain dependency (Terraform, Go, Pandoc, uv, jq, gh) — no system-installed tools are required beyond Zig itself and gcloud for authentication.

Table of Contents

  1. Introduction & Rationale
  2. Hermeticity
  3. Core Components
  4. Dependency Management

Introduction & Rationale

Why Zig?

The choice of Zig for orchestrating this infrastructure project was driven by several key requirements:

Hermetic Builds: Zig’s build system enables hermeticity - complete isolation from host environments with deterministic, reproducible outputs. It uses pinned, cryptographically verified toolchains to ensure identical builds across all environments.

Cross-Platform Reproducibility: Unlike shell scripts or Makefiles, Zig provides deterministic builds across Linux, macOS, and Windows without platform-specific quirks.

Zero Dependencies: Zig eliminates the need for external interpreters, package managers, or runtime dependencies - the build system is completely self-contained.

Hermeticity

What is Hermeticity?

Hermeticity (from “hermetic” meaning airtight or sealed) refers to builds that are completely isolated from the host environment and produce deterministic, reproducible outputs regardless of who runs the build, when it’s run, or where it’s executed.

A hermetic build system has these fundamental properties:

  • Complete Isolation: No dependencies on host system state, installed software, or external services
  • Deterministic Results: Identical inputs always produce bit-for-bit identical outputs
  • Self-Contained: All dependencies are explicitly controlled and included
  • Reproducible: Builds can be recreated months or years later with identical results

Traditional Problems with Non-Hermetic Builds:

  • “Works on my machine” syndrome from different tool versions
  • Security vulnerabilities from compromised system tools
  • Compliance failures due to untracked dependencies
  • Difficult incident reproduction and debugging

How we achieve hermeticity

This build system achieves hermeticity through multiple layers of isolation and control:

1. Tools as Source Code

The system treats external tools (Terraform, Go) as source code by pinning exact versions with cryptographic verification:

const terraform_version = std.SemanticVersion{ .major = 1, .minor = 11, .patch = 3 };
const golang_version = std.SemanticVersion{ .major = 1, .minor = 24, .patch = 5 };

fn resolve_terraform_checksum(os: std.Target.Os.Tag, cpu: std.Target.Cpu.Arch) []const u8 {
    switch (os) {
        .linux => switch (cpu) {
            .x86_64 => "N-V-__8AANIzZgVtY0M64MsUfL8H8Y8wyYq6m0NhUhVcUY_F",
            .aarch64 => "N-V-__8AANITIgUZmOWcweYJtkC62Z_XPI8bi-EmQTwYsglt",
            // ... platform-specific checksums
        },
    }
}

Hermetic Properties:

  • Version Immutability: No “latest” versions or auto-updates
  • Cryptographic Verification: Checksums hash checks prevent tampering
  • Platform Isolation: Different checksums for each OS/architecture
  • Zero Host Dependencies: Never uses system-installed tools

2. Content-Addressable Binary Storage

The system uses Zig’s fetch mechanism with content-addressable storage:

fn fetch(b: *std.Build, options: struct {
    url: []const u8,
    file_name: []const u8,
    hash: ?[]const u8,
}) std.Build.LazyPath {
    copy_from_cache.addFileArg(
        b.addSystemCommand(&.{ b.graph.zig_exe, "fetch", options.url }).captureStdOut(),
    );
    // Verifies hash matches expected value or panics
    if (options.hash) |hash| {
        copy_from_cache.addArg(hash);
    }
}

Hermetic Properties:

  • Global Cache: ~/.cache/zig/p/{hash}/ provides content-addressable storage
  • Hash Verification: Automatic verification prevents corruption
  • Lazy Download: Tools downloaded only when needed
  • Deduplication: Identical binaries shared across projects

3. Deterministic Service Account Permissions

Every developer and CI system uses identical permissions through service account impersonation:

fn impersonate_service_account(terraform: *Self) !void {
    const token = terraform.shell.exec_stdout(
        "{gcloud} auth print-access-token --impersonate-service-account={sa} --lifetime={lifetime}",
        .{
            .sa = config.service_account,
            .lifetime = config.token_lifetime_seconds,
        },
    );
}

Hermetic Properties:

  • Permission Consistency: Everyone has identical IAM permissions
  • No Personal Credentials: Personal accounts never used for infrastructure
  • Deterministic Access: Same service account across all environments
  • Audit Clarity: All operations traced to specific service accounts

Hermeticity Guarantees

The build system provides these concrete guarantees:

Guarantee 1: Identical Build Interface Across All Environments

The exact same commands work identically on every platform and environment without modification.

# These commands work IDENTICALLY everywhere:
./zig/zig build plan -- ha-infra development     # macOS M1 developer laptop
./zig/zig build plan -- ha-infra development     # Linux x86_64 CI runner
./zig/zig build plan -- ha-infra development     # Windows x64 workstation

Technical Implementation:

  • Unified Entry Point: Single ./zig/zig build command across all platforms
  • No Platform Scripts: No separate .sh, .bat, Dockerfile or platform-specific makefiles
  • Cross-Platform Zig: Zig compiler itself provides platform abstraction

Guarantee 2: Complete Dependency Isolation

The build system never conflicts with or depends on any system-installed tools.

Technical Guarantees:

  • Never Uses System PATH: Tools sourced exclusively from Zig cache
  • No Global Dependencies: Zero reliance on homebrew, apt, chocolatey, etc.
  • Concurrent Safety: Multiple tool versions coexist without conflicts
  • Clean Uninstall: rm -rf ~/.cache/zig/p/ removes all managed tools

Guarantee 3: Deterministic Builds

Each commit produces identical build system behavior and tool execution across all platforms and time.

Benefits:

  • Perfect Build Recreation: Debug any historical build exactly
  • Tool Consistency: Cryptographic proof of exact binaries used
  • Cross-Platform Verification: Same behavior regardless of OS
  • Supply Chain Security: Immutable audit trail of tool versions and build logic

Benefits of Hermetic Infrastructure Builds

Security Benefits

  • Supply Chain Security: Cryptographic verification prevents tampered tools
  • Consistent Attack Surface: Identical tools mean predictable security properties
  • Audit Trail: Complete traceability of tool versions and permissions

Reliability Benefits

  • Eliminates “Works on My Machine”: Identical behavior across all environments
  • Predictable Failures: Issues reproduce reliably for debugging
  • Safe Rollbacks: Can recreate any historical build exactly

Operational Benefits

  • Zero Setup Time: New developers productive immediately
  • Incident Response: Perfect reproduction of any deployment
  • Compliance: Immutable audit trail of tool usage

Development Benefits

  • Faster CI: No tool installation or version checking needed
  • Confident Deployments: High trust in build reproducibility
  • Easy Testing: Deterministic behavior enables thorough testing

Technical Implementation: How It Works

Content-Addressable Storage

# Example: Terraform binary storage
~/.cache/zig/p/N-V-__8AANIzZgVtY0M64MsUfL8H8Y8wyYq6m0NhUhVcUY_F/terraform

Each binary is stored by its content hash, making tampering impossible.

Compile-Time Enforcement

comptime {
    if (!zig_version_match) {
        @compileError(std.fmt.comptimePrint(
            "unsupported zig version: expected 0.15.0-0.15.{}, found {}",
            .{ zig_version.patch, builtin.zig_version },
        ));
    }
}

Version mismatches are caught at compile time, not runtime.

Hash Verification

if (args.len == 6 and !std.mem.eql(u8, args[5], hash)) {
    std.debug.panic(
        \\bad hash
        \\specified:  {s}
        \\downloaded: {s}
    , .{ args[5], hash });
}

The system panics if downloaded content doesn’t match expected hash.

Core Components

build.zig - Main Orchestrator

Location: /build.zig

The main build file serves as the entry point and orchestrates the entire system:

pub fn build(b: *std.Build) !void {
    const target = try resolve_target(b);
    const terraform_exe = build_terraform_exe(b, target);
    const go_exe = build_go_exe(b, target);

    // Setup all build steps
    const build_steps = .{
        .plan = b.step("plan", "Run Terraform Plan"),
        .apply = b.step("apply", "Run Terraform Apply"),
        // ... other steps
    };
}

Key Responsibilities:

  • Version Validation: Ensures Zig 0.15.x compatibility at compile time
  • Binary Management: Downloads and caches Terraform/Go with checksum verification
  • Build Step Registration: Defines all available commands (plan, apply, clean, etc.)
  • Cross-Platform Support: Handles Windows, macOS, and Linux differences
  • Mode Configuration: Sets up local, CI, or CloudBuild execution modes

Architecture Decisions:

  • Uses std.Build.LazyPath for lazy evaluation of downloads
  • Implements wrapper scripts for Go to handle GOROOT properly
  • Leverages compile-time evaluation for platform detection

Dependency Management

The build system manages external dependencies through two complementary approaches:

  1. Pre-built Binary Downloads: Tools like Terraform, Go, and uv are downloaded as pre-compiled binaries with cryptographic verification
  2. Source Code Dependencies: Tools like jq, gh, and swo_cli are pulled from Git repositories and built from source

Both approaches share the same hermetic properties: version pinning, hash verification, and content-addressable caching.

Pre-built Binary Downloads

Pre-built binaries are used when official releases are available and building from source offers no advantage.

Download Process

  1. Checksum Resolution: Platform-specific checksums are hardcoded for security

    fn resolve_terraform_checksum(os: std.Target.Os.Tag, cpu: std.Target.Cpu.Arch) []const u8 {
    switch (os) {
        .linux => switch (cpu) {
            .x86_64 => "N-V-__8AANIzZgVtY0M64MsUfL8H8Y8wyYq6m0NhUhVcUY_F",
            .aarch64 => "N-V-__8AANITIgUZmOWcweYJtkC62Z_XPI8bi-EmQTwYsglt",
        },
        // ... other platforms
    }
}

  2. Zig Fetch Integration: Uses zig fetch for downloading and verification

  3. Cache Management: Leverages Zig’s global cache (~/.cache/zig/p/) for persistence

  4. Lazy Evaluation: Downloads only occur when binaries are actually needed

Go Runtime Wrapper

Unlike Terraform (a single binary), Go distributions are complete toolchains containing the compiler, standard library source, and auxiliary tools (gofmt, vet, doc, etc.). Go requires special handling due to its dependency on GOROOT to locate these components:

const wrapper_content = b.fmt(
    "#!/bin/bash\n" ++
    "export GOROOT=\"$HOME/.cache/zig/p/{s}\"\n" ++
    "exec \"$GOROOT/bin/go\" \"$@\"\n",
    .{go_hash},
);

This wrapper script:

  • Sets GOROOT to the cached Go distribution root
  • Enables Go to locate its standard library and all auxiliary tools
  • Preserves all command-line arguments for seamless execution

Version Pinning Strategy

All tool versions are pinned in build.zig:

const terraform_version = std.SemanticVersion{ .major = 1, .minor = 11, .patch = 3 };
const golang_version = std.SemanticVersion{ .major = 1, .minor = 24, .patch = 5 };

This ensures:

  • Reproducible Builds: Every developer uses identical tool versions
  • Security: Prevents supply chain attacks through version drift
  • Stability: Controlled updates with testing

See Also: For instructions on upgrading these dependencies, see Upgrade Build Dependencies

Source Code Dependencies (build.zig.zon)

Dependencies declared in build.zig.zon are Git repositories that Zig pulls and builds from source. This approach is used when:

  • Official pre-built binaries aren’t available for all platforms
  • Building from source provides better integration with the build system
  • We need to customize the build process

Current Source Dependencies:

Tool Language Purpose
jq C JSON parsing and manipulation
gh Go GitHub CLI for PR/workflow operations
swo_cli Go SolarWinds Observability CLI

How It Works

  1. Declaration: Dependencies are pinned to specific Git commits in build.zig.zon:

    .jq = .{
    .url = "git+https://github.com/jqlang/jq?ref=v1.8.1#b29504b4e191a4b1028bd1c252aaab0c62fbdbfa",
    .hash = "N-V-__8AAA_fRwDh8z1F7t5Sjbpp-RTQuke1LgqopB00GLps",
},

  2. Fetching: Zig’s package manager downloads the repository to ~/.cache/zig/p/{hash}/

  3. Building: The build system compiles from source using language-appropriate toolchains

C Projects (jq)

Zig includes a full C compiler via LLVM integration, allowing us to build C projects without requiring system-installed compilers:

const jq = b.dependency("jq", .{ .target = target });
const jq_exe = b.addExecutable(.{
    .name = "jq",
    .root_module = b.createModule(.{
        .target = target,
        .optimize = .Debug,
    }),
});

// Add C source files from the jq repository
for (sources) |src| {
    jq_exe.addCSourceFile(.{
        .file = jq.path(src),
        .flags = flags.items,
    });
}
jq_exe.linkLibC();

Go Projects (gh, swo_cli)

Go projects use the Go toolchain that the build system downloads as a pre-built binary:

const gh = b.dependency("gh", .{ .target = target });

const build_cmd = b.addSystemCommand(&.{
    go_exe,   // Pre-built Go binary
    "-C",
});
build_cmd.addDirectoryArg(gh.path("."));
build_cmd.addArgs(&.{ "build", "-trimpath", "-o" });
const built_gh = build_cmd.addOutputFileArg("gh");
build_cmd.addArg("./cmd/gh");

Hermetic Properties

Source code dependencies maintain the same hermetic guarantees as pre-built binaries:

  • Commit Pinning: Exact Git commits (not tags or branches) ensure reproducible builds
  • Hash Verification: Content hashes prevent tampering or unexpected changes
  • No System Dependencies: C compiler (Zig) and Go toolchain are both managed by the build system
  • Cross-Platform: Same source builds identically on macOS, Linux, and Windows
Edit this page