RFC: Docker Swarm VM Consolidation

Status: Draft | Author: DevOps Team | Created: 2026-02-03 | Updated: 2026-02-03

Overview

Consolidate 98 isolated Docker Swarm VMs across staging and production environments into two unified swarms (one per environment) with MIG-based auto-scaling. Current infrastructure runs at ~7% CPU and ~40% memory utilization, representing massive over-provisioning.

Total Organizational Impact:

• **92, 000 + annualsavings * *(54k infrastructure + $38k operational time)
• 80% reduction in management overhead (98 VMs → 20 nodes)
• 8-month payback, $214k+ savings over 3 years

---

Executive Summary

---

Background

Staging Environment (Non-Production)

Analysis covers 3 GCP projects:

• prj-bu1-n-hb-infra-5381 - HB Infrastructure (2 running VMs)
• prj-bu1-n-pd-infra-fee5 - PD Infrastructure (43 running VMs)
• prj-bu1-n-bees-infra - Bees Infrastructure (0 running VMs)

Production Environment

Analysis covers 3 GCP projects:

• prj-bu1-p-hb-infra-1da6 - HB Infrastructure (10 running VMs)
• prj-bu1-p-pd-infra-b355 - PD Infrastructure (42 running VMs)
• prj-bu1-p-bees-infra-8bed - Bees Infrastructure (1 running VM)

*Estimated based on staging utilization patterns; architectures mirror each other.

Combined Current State

The Problem

The current architecture deploys each Docker workload on a dedicated VM, creating 98 separate single-node swarms. This approach suffers from:

• Resource fragmentation: 223 vCPUs provisioned but only ~16 used; 869 GB memory provisioned but only ~350 GB used
• Per-VM overhead: ~98 GB wasted on OS and monitoring agents across all VMs
• Operational complexity: 98 separate management targets, monitoring configs, and patching cycles
• No resource sharing: Underutilized VMs cannot lend resources to busy ones
• Duplicated effort: Same problems solved twice (staging and production managed separately)

---

Proposed Architecture

Two separate consolidated swarms (staging and production remain isolated):

graph TB
    subgraph "Staging Swarm (Non-Production)"
        subgraph "Staging Managers"
            SM1[Manager 1
e2-medium]
            SM2[Manager 2
e2-medium]
            SM3[Manager 3
e2-medium]
        end
        subgraph "Staging Workers - MIG (6-9)"
            SW1[Worker 1-6
n2-highmem-4]
            SWN[Workers 7-9
Auto-scale]
        end
        SM1 <--> SM2 <--> SM3 <--> SM1
        SM1 --> SW1
        SM2 -.-> SWN
    end

    subgraph "Production Swarm"
        subgraph "Production Managers"
            PM1[Manager 1
e2-medium]
            PM2[Manager 2
e2-medium]
            PM3[Manager 3
e2-medium]
        end
        subgraph "Production Workers - MIG (8-12)"
            PW1[Worker 1-8
n2-highmem-4]
            PWN[Workers 9-12
Auto-scale]
        end
        PM1 <--> PM2 <--> PM3 <--> PM1
        PM1 --> PW1
        PM2 -.-> PWN
    end

Node Specifications

Staging Swarm

Production Swarm

Why n2-highmem-4: Memory is the binding constraint (~40% utilization vs ~7% CPU). The n2-highmem-4 provides 8 GB/vCPU at $5.97/GB, the best cost ratio for memory-constrained workloads.

Auto-scaling Configuration

---

Cost Analysis

Staging Environment

Production Environment

Combined Infrastructure Savings

---

ROI Analysis

Team Context: 3 DevOps engineers at 150k/yearaveragesalary( 72/hr, or ~$108/hr fully loaded with benefits/overhead).

One-Time Investment

Timeline optimized to 8 weeks by leveraging shared learnings between staging and production.

Annual Savings

Operational savings based on 80% reduction in maintenance burden:

• Staging: 45 VMs → 9 nodes, ~160 hrs/year saved
• Production: 53 VMs → 11 nodes, ~192 hrs/year saved
• Combined: 98 VMs → 20 nodes, ~352 hrs/year saved at $108/hr

Payback and 3-Year ROI

---

Implementation Plan

Timeline: 8-10 weeks (3 DevOps engineers at ~50% allocation = 480 person-hours)

Phase 1: Staging Infrastructure (Week 1-2)

• Provision 3 manager nodes (e2-medium) in separate zones
• Configure worker MIG with n2-highmem-4 template
• Set up internal load balancer for swarm ingress
• Deploy centralized Prometheus/Grafana stack

Phase 2: Staging Validation & Migration (Week 3-5)

• Deploy test workloads to validate networking and storage
• Verify auto-scaling triggers (memory-based)
• Migrate staging workloads (low risk, fast iteration)
• Document runbooks based on learnings
• Keep source VMs stopped for rollback

Phase 3: Production Infrastructure (Week 6-7)

• Apply learnings from staging
• Provision production swarm (3 managers, 8 workers)
• Configure with production-grade monitoring
• Establish separate MIG auto-scaling policies

Phase 4: Production Migration (Week 8-10)

• Migrate non-critical production workloads first
• Migrate standard workloads
• Migrate high-memory workloads with dedicated node labels
• Validate application behavior under production load

Phase 5: Decommission & Optimization (Week 11-12+)

• Decommission source VMs after 30-day validation
• Analyze actual utilization in consolidated environments
• Adjust MIG sizing based on observed patterns
• Evaluate 3-year CUD after 6 months of stable operation

---

Risks and Mitigations

Rollback Strategy

• Keep old VMs stopped for 30 days post-migration
• DNS-based traffic shifting enables instant rollback
• Maintain deployment scripts for old architecture
• Staging serves as canary for production changes

---

Decision

Recommendation

Proceed with full consolidation of both environments. The data is unambiguous:

• Paying for 223 vCPUs, using ~16
• Paying for 869 GB memory, using ~350 GB
• Managing 98 VMs when 20 nodes would suffice
• **92, 000 + annualsavings * *(54k infrastructure + $38k operational time)
• Payback in ~8 months, $214k+ savings over 3 years

Decision Matrix

3-Year Total Cost Comparison

Infra: 36 months × monthly cost.

Success Criteria

Key Decision Points

Edit this page