THB-P-SFTP-WIN Cerberus FTP Server Upgrade

Upgrade Cerberus FTP Server on THB-P-SFTP-WIN, the Health Align production SFTP server, with a rollback ready and partner traffic checked afterwards.

Use this runbook when:

• The Cerberus dashboard’s Vulnerability Assessment shows Vulnerabilities: Detected
• Help → Check for Update offers a new release
• The licence has been renewed and a newer version is covered

The upgrade takes about 2 minutes of SFTP downtime: the installer stops the service, replaces the program files and starts it again. Users and settings are kept. It rarely needs a reboot.

The server at a glance

See THB-P-SFTP-WIN Storage Cleanup for the VM, drives and scheduled tasks, and Access for RDP and Run Command. The upgrade itself needs RDP; the checks below work through Run Command.

1. Prepare (days before)

Read the release notes

Read the release notes for every version between the installed one and the target. Look for:

• CVEs and their exposure. Only port 22 is reachable from the internet. TLS fixes matter less than SSH or local-privilege fixes.
• Removed SSH algorithms (ciphers, MACs, host keys, key exchange). See Old SSH clients before agreeing to any removal.
• “Installs older than X must update manually”, a licence agreement (EULA or MSA) change, or other notes about the upgrade path. These mean the in-app updater stops partway (see Upgrade).
• Reporting, FIPS or log-format changes. This server doesn’t use reporting or FIPS mode.

Check the licence

The dashboard’s Current license expiration must be in the future, and the licence must cover the target version. Renewals go through the Renew button, which opens the Cerberus customer portal (portal.cerberusftp.com). The portal can be down (Cloudflare 522). Try again later, or contact Cerberus support (support.cerberusftp.com).

Pick the window

Partner pollers connect around the clock (~360 connections an hour overnight), so no hour is empty. Polling clients retry, so a short outage costs them a few missed polls, not files.

Prefer Tuesday to Thursday. Weekends aren’t quieter (the meal vendors run every day), and a Friday problem sits over the weekend. To recheck the pattern before choosing, run Traffic by hour.

Announce the outage with Execute a Planned Outage, so Accounts can decide whether partners need notice.

2. Back up (right before the window)

Cerberus backup (RDP)

1. Open PowerShell as administrator and pick the backup <stamp> once. Write it down: the zip, the folder copy, the snapshots and the cleanup all use this same value.

$stamp = Get-Date -Format yyyyMMdd-HHmm
$stamp

2. In the Cerberus admin UI, use Tools → Backup Users and Settings. Save the zip to E:\ as cerberus_server_backup_<stamp>.zip.
3. In the same PowerShell window, copy the whole config folder too:

robocopy "C:\ProgramData\Cerberus LLC\Cerberus FTP Server" "E:\cerberus-backup-$stamp" /E /XD log tempFiles /R:1 /W:1
# robocopy exit codes 8 and above mean some files failed to copy.
if ($LASTEXITCODE -ge 8) { throw "Config backup failed (robocopy exit $LASTEXITCODE). Fix it before taking snapshots." }

Keep both on E: and don’t copy them to S:. They hold every partner account’s password hash and SSH key. The E: snapshot below is the off-VM copy.

SSH host key (your laptop)

Record the host key fingerprint that partners have pinned, to compare after the upgrade:

ssh-keyscan -T 10 -p 22 13.77.105.190 2>/dev/null | ssh-keygen -lf -

On 2026-10-28 it was 3072 SHA256:aBiM6rQAaJkw4c4FtiLg75hoGEQe8bWIlVEPcgs/Bmo (RSA), the only host key offered.

Disk snapshots (your laptop)

Take them after the Cerberus backup, so the E: snapshot contains it. Snapshots don’t interrupt SFTP.

Use the backup’s <stamp> (yyyyMMdd-HHmm) in the snapshot names, and never reuse a name. az snapshot create with an existing name overwrites that snapshot, so retrying after a failed upgrade would replace the good rollback point with the broken state.

az snapshot create -g THB-PROD-SFTP-RG -n thbpsftpwin-os-pre-cerberus-<target-version>-<stamp> --source thbpsftpwin-osdisk-20250924-203359 --incremental true

az snapshot create -g THB-PROD-SFTP-RG -n thbpsftpwin-data-pre-cerberus-<target-version>-<stamp> --source thbpsftpwin-datadisk-001-20250924-203359 --incremental true

Then stop anyone from downloading them. They’re only needed for a restore inside Azure:

az snapshot update -g THB-PROD-SFTP-RG -n thbpsftpwin-os-pre-cerberus-<target-version>-<stamp> --network-access-policy DenyAll --public-network-access Disabled

az snapshot update -g THB-PROD-SFTP-RG -n thbpsftpwin-data-pre-cerberus-<target-version>-<stamp> --network-access-policy DenyAll --public-network-access Disabled

S: (the hasftp Azure file share) is not on either disk. The upgrade doesn’t touch it.

3. Upgrade

1. In the admin UI, open Connections and let any active transfers finish.
2. Open Help → Check for Update and confirm the offered version is the target. Click Update. This downloads the installer to the installers\ folder and starts it.
3. If a dialog says the End User License Agreement has been updated and asks you to run the full installer, click OK. The service is still running the old version at this point. Then:

   1. Check the downloaded installer’s signature in PowerShell. Status must be Valid and the signer Redwood Software, Inc.:

      Get-AuthenticodeSignature "C:\ProgramData\Cerberus LLC\Cerberus FTP Server\installers\CerberusUpdate_amd64_<version>.exe" | Format-List Status, SignerCertificate

   2. Run that .exe as administrator.

4. In the installer:
   • Service credentials: keep Use Current: .\Cerberus and click Next. Don’t change anything under Advanced: that account owns the file permissions.
   • Install path: it’s locked to the existing path. Tick I agree to the license terms and click Next.
   • It installs prerequisites (Visual C++ redistributable), then stops the service. The outage starts here.
5. On successfully installed, click Finish. Reboot only if the installer asks.

If the updater isn’t offered, or fails, download the 64-bit installer from cerberusftp.com/download instead. Before running it, check it the same way as above: the signature Status must be Valid, the signer Redwood Software, Inc., and the version the target:

$i = Get-Item "<path to downloaded installer>.exe"
$i.VersionInfo.FileVersion
Get-AuthenticodeSignature $i.FullName | Format-List Status, SignerCertificate

Then exit the admin UI with File → Exit, run Stop-Service "Cerberus FTP Server", and run the installer as administrator.

4. Verify

From your laptop

Save this as cerberus-check.ps1 and run it with Run Command. Set $since to the time the upgrade finished:

$since = '<upgrade finish time, yyyy-MM-dd HH:mm:ss>'
$t = [datetime]::MinValue
if (-not [datetime]::TryParseExact($since, 'yyyy-MM-dd HH:mm:ss', [Globalization.CultureInfo]::InvariantCulture, 'None', [ref]$t) -or $t -gt (Get-Date)) { throw 'Set $since to the upgrade finish time first (yyyy-MM-dd HH:mm:ss, not in the future).' }
$b = "$env:ProgramData/Cerberus LLC/Cerberus FTP Server"
$svc = Get-CimInstance Win32_Service -Filter "Name='Cerberus FTP Server'"
"service: $($svc.State) / $($svc.StartMode) / $($svc.StartName)"
"version: " + (Get-Item (($svc.PathName -replace '"','') -split ' -')[0].Trim()).VersionInfo.FileVersion
$pids = (Get-Process | Where-Object ProcessName -like 'Cerberus*').Id
"listening: " + ((Get-NetTCPConnection -State Listen | Where-Object { $pids -contains $_.OwningProcess } | Select-Object -ExpandProperty LocalPort -Unique | Sort-Object) -join ',')
$l = Get-Content (Get-ChildItem "$b/log" -Filter 'server*.log' | Sort-Object LastWriteTime | Select-Object -Last 1).FullName | Where-Object { $_.Length -gt 21 -and $_.Substring(1,19) -gt $since }
"logins since: " + (($l | Where-Object { $_ -match "Signature verified with public key presented by user '([^']+)'" } | ForEach-Object { $matches[1] } | Group-Object | ForEach-Object { "$($_.Name):$($_.Count)" }) -join ' ')
$known = @(
  'Unable to add to Windows Firewall Exception list: Access is denied'
  "Unable to open file handle to 'C:\\FTP\\[^']*\\incoming\\[^']*' : The system cannot find the file specified"
  "\]:imperial - Unable to (access local directory '[^']*\\_gsdata_'|open directory handle to '/incoming/_gsdata_')"
  "Public key 1 does not match key presented by user 'wellcare_icon'"
) -join '|'
"unexpected errors: " + (($l | Where-Object { $_ -match ' ERROR ' -and $_ -notmatch $known } | Select-Object -First 5) -join ' || ')
foreach ($t in 'MoveFTPFiles','ArchiveMealReferrals') { $i = Get-ScheduledTaskInfo -TaskName $t; "{0}: last={1} result=0x{2:X}" -f $t, $i.LastRunTime.ToString('s'), $i.LastTaskResult }

az vm run-command invoke -g THB-PROD-SFTP-RG -n THB-P-SFTP-WIN --command-id RunPowerShellScript --scripts "@cerberus-check.ps1" --query "value[0].message" -o tsv

Then rerun the host key check:

ssh-keyscan -T 10 -p 22 13.77.105.190 2>/dev/null | ssh-keygen -lf -

Expected:

• The host key fingerprint is unchanged. Partners and our Cloud Run jobs pin it, so a new key breaks all of them, and we’d only find out from the 07:00 job. If it changed, restore certificates\ from E:\cerberus-backup-<stamp>\ and restart the service, or roll back.
• service is Running / Auto / .\Cerberus, and version is the target.
• listening shows the same ports as before: 21,22,990,8443,10001,58000.
• logins since lists partners, including wellcare_gafoods, which connects about once a minute with the oldest SSH client (see Old SSH clients). If it’s missing, roll back.
• unexpected errors is empty. The script filters out these known, harmless errors:
  • Unable to add to Windows Firewall Exception list: Access is denied, logged at startup. The service account can’t add firewall rules, and the existing allow rule for CerberusGUI.exe already covers the same path.
  • An occasional Unable to open file handle to 'C:\FTP\...\incoming\...' : The system cannot find the file specified. \MoveFTPFiles moved an upload to S: before the partner’s client opened it again.
  • imperial: Unable to access local directory '...\_gsdata_', hourly. Its GoodSync client looks for a metadata folder that doesn’t exist.
  • wellcare_icon: Public key 1 does not match key presented, nightly at 01:00. Its client offers a non-matching key first, then logs in with the right one.
• MoveFTPFiles shows result 0x0 after its next 10-minute run.

In the admin UI

• Vulnerability Assessment: Current Version = Latest Version, and Vulnerabilities: None. This panel only covers known CVEs in the installed version. Algorithm warnings such as “SSH hmac-sha1 Mac is active” appear separately under System Messages and don’t affect it. On 2026-10-27 it showed None with hmac-sha1 still enabled. Leave those warnings alone (see Old SSH clients).
• Server Manager → Protocols → SSH SFTP: Mask Server Identification (SSH and HTTP/S) is still ticked. It hides the Cerberus version from anyone probing port 22.
• Spot-check User Manager: the user count, a couple of users’ folders and SSH keys.

The next morning

• Check that the 07:00 and 08:00 Cloud Run jobs (gcp_scan_cloud_run_user, gcp_centene_cloud_run_user) logged in and moved files.
• Run the SHA-1 client script and check that the other legacy-client accounts logged in. They connect far less often than wellcare_gafoods, so give them a day or two.

5. Roll back

Use the smallest step that fixes the problem:

1. Settings lost or wrong. Use Tools → Restore with the zip, or stop the service, copy E:\cerberus-backup-<stamp>\ back over C:\ProgramData\Cerberus LLC\Cerberus FTP Server\, and start the service again.
2. New version won’t run. Reinstall the previous version from installers\CerberusUpdate_amd64_<previous>.exe, then restore the settings as above.
3. The VM is broken. Create a disk from the OS snapshot and swap it in as the OS disk:

az disk create -g THB-PROD-SFTP-RG -n thbpsftpwin-osdisk-restored-<stamp> --source thbpsftpwin-os-pre-cerberus-<target-version>-<stamp>

az vm stop -g THB-PROD-SFTP-RG -n THB-P-SFTP-WIN

az vm update -g THB-PROD-SFTP-RG -n THB-P-SFTP-WIN --os-disk thbpsftpwin-osdisk-restored-<stamp>

az vm start -g THB-PROD-SFTP-RG -n THB-P-SFTP-WIN

Run Verify again. Once it passes, the old OS disk is detached but still billed. Delete it:

az disk delete -g THB-PROD-SFTP-RG -n <old OS disk name from the table>

Then, in a PR, update the OS disk name in the table and in the snapshot commands, so the next upgrade snapshots the live disk.

6. Clean up

After about a week of normal partner traffic, delete the rollback copies. They hold partner credentials, and the snapshots cost money.

1. The snapshots, from your laptop:

az snapshot delete -g THB-PROD-SFTP-RG -n thbpsftpwin-os-pre-cerberus-<target-version>-<stamp>

az snapshot delete -g THB-PROD-SFTP-RG -n thbpsftpwin-data-pre-cerberus-<target-version>-<stamp>

2. The Cerberus backups on E:, on the VM. Keep only the newest pair if you want a config fallback for the next upgrade:

Remove-Item "E:\cerberus_server_backup_<stamp>.zip", "E:\cerberus-backup-<stamp>" -Recurse -Force

Reference

Old SSH clients

Some partners still negotiate old algorithms. On 2026-10-27 these accounts used hmac-sha1: wellcare_gafoods (client SSH-2.0-SecureBlackbox, with aes128-cbc + ssh-rsa, on every session), simpra, Aetna and abilis_healthaxis. Every other account uses SHA-2, CTR, GCM or ChaCha20. Don’t disable hmac-sha1, aes128-cbc or the ssh-rsa host key algorithm, even if the dashboard recommends it, and don’t take an upgrade that removes any of them, until those partners have updated their clients.

To see which accounts use SHA-1 now:

$sha1 = @{}; $user = @{}
Get-ChildItem "$env:ProgramData/Cerberus LLC/Cerberus FTP Server/log" -Filter 'server*.log' | ForEach-Object { Get-Content $_.FullName } | ForEach-Object {
  if ($_ -match '\[\s*(\d+)\] - \[[^\]]*\]:([^ ]*) - (.*)$') {
    # Copy out of $matches first: the -match below overwrites it.
    $id = $matches[1]; $u = $matches[2]; $msg = $matches[3]
    if ($msg -like 'Agreed upon*') { $sha1[$id] = ($msg -match "hmac-sha1, none'") }
    if ($u -and -not $user.ContainsKey($id)) { $user[$id] = $u }
  }
}
$sha1.Keys | Where-Object { $sha1[$_] } | ForEach-Object { $user[$_] } | Group-Object | ForEach-Object { "$($_.Name): $($_.Count) sha1 sessions" }

Session IDs restart after a service restart, so run this on logs from a single run of the service. Right after an upgrade, wait a few hours.

Traffic by hour

Connections and file opens per hour (CT) across the retained logs, excluding lynx_test, which polls every few seconds:

$rx = '^\[\d{4}-\d\d-\d\d (\d\d):\d\d:\d\d\]:\s*\w+ \[\s*\d+\] - \[[^\]]+\]:([^ ]*) - (Incoming connection|File open)'
$rows = Get-ChildItem "$env:ProgramData/Cerberus LLC/Cerberus FTP Server/log" -Filter 'server*.log' | ForEach-Object { Get-Content $_.FullName } | ForEach-Object { if ($_ -match $rx -and $matches[2] -ne 'lynx_test') { [pscustomobject]@{ h = [int]$matches[1]; k = $matches[3] } } }
0..23 | ForEach-Object { $h = $_; $x = $rows | Where-Object h -eq $h; '{0:D2}  conns={1,5}  opens={2,5}' -f $h, ($x | Where-Object k -eq 'Incoming connection').Count, ($x | Where-Object k -eq 'File open').Count }

Gotchas

• The in-app updater stops at a licence agreement change. The dialog says to run the full installer, which it has already downloaded to installers\. Nothing has changed at that point.
• The log format changed in 2026.x. Session IDs are padded with spaces ([ 0]). Match them with \[\s*\d+\].
• admin - Connection timed out in the log is an idle admin-UI session, not a partner.
• Automatic IP Blocking is off. If you turn it on, exempt the partner IPs first: pollers connect about once a minute and can get blocked. The NSG already limits port 22 to partner IPs.
• Expired server.pem. The TLS certificate is a self-signed CN=HAWEB01 cert that expired on 2025-11-14. It only affects FTPS and HTTPS, which the NSG blocks from the internet. The SFTP host keys are separate and valid until 2043.
• The dashboard shows Host HAWEB01. That’s the Windows computer name kept from the backup the VM was restored from. It’s expected.