sftpcopy

A command-line tool for copying files from an SFTP source

For full documentation and usage examples, see the sftpcopy project README.

Important: Windows users should execute these commands using WSL (Windows Subsystem for Linux).

Build

1. Follow Setup instructions in README

2. Ensure GitHub CLI is setup

   ./bin/gh auth login

3. Build sftpcopy

   ./zig/zig build sftpcopy

This will:

1. Clone the private thehelperbees/sftpcopy repository
2. Build the CLI binary using Go
3. Output the sftpcopy binary into ./bin
4. Clean up the cloned source

Note: Requires gh CLI authentication to access the private repository.

Setup External SFTP Connection (Cloud Run Job)

1. Generate a sftpcopy config.yml file

   ./bin/sftpcopy template > config.yml

2. Modify template’s source key with the correct

   • Host,
   • Port,
   • User,
   • Password or SSH Private Key (Base64 encoded)

3. Test the configuration

   ./bin/sftpcopy --config config.yml

4. Continue to modify the config.yml file with:

   Note: This is a standalone CLI tool that does not communicate with external data sources or keep track of files it has already processed. Choose your filters and automation schedule carefully to avoid duplicate processing!

   • Filters
   • Deadmans Snitch URL
   • Papertrail
   • Encryption or Decryption

5. Create a secret in GCP Secret Manager

   • Follow this example PR
   • Test w/ : ./zig/zig build plan -- common-infra
   • Commit changes and submit PR

6. Create a new secret version with the config.yml data

   gcloud secrets versions add <SECRET_NAME> --data-file=config.yml --project=prj-bu1-c-common-infra-c4aa

   Replace <SECRET_NAME> with your secret name.

7. Create Cloud Run job

   • Follow this example PR
   • Test w/ : ./zig/zig build plan -- common-infra
   • Commit changes and submit PR

Make tool accessible from anywhere in your terminal

# build
./zig/zig build sftpcopy

# move
sudo mv ./bin/sftpcopy /usr/local/bin/sftpcopy

# now accessible globally
which sftpcopy