Restore onto a new cluster
Copy the key first. The archive in your bucket is encrypted, and the key that opens it lives in the cluster’s vault and nowhere else. Delete the cluster without copying that key off it and nobody can ever open the archive, you included. Step 2 is the copy. Don’t delete anything until you’ve done it.
This page takes an app and its database from one cluster to a new one, with only a bucket surviving in between. You don’t need SSH on either cluster.
That bucket is the only off-server copy this page needs. A second copy (shc backup mirror) stays optional.
What you need
Section titled “What you need”-
shc0.1.56 or newer, and no older than the cluster. Get it or update it with Reach a portal cluster from your laptop, then check whichshcyou’re running:Terminal window $ command -v shc$ shc versionThe path should be the folder that page puts it in, and the version the one
https://downloads.selfhostedcloud.org/shc/dev/RELEASEnames (shc/RELEASEon the production portal), or newer. An oldershcfrom an earlier install still signs in, then sends requests the cluster refuses, such asX400976TNTNONfromshc backup sync. -
The
shccommand line, signed in to the cluster:Terminal window $ shc auth login --url https://<cluster domain>You sign in through the portal. You have to be the cluster’s registrant, an owner or an admin, which gives you
platform:manageon it. An organisation member or viewer is refused, and every step below needs that role.
1. Set the backup destination
Section titled “1. Set the backup destination”Follow Set a backup destination through its step 3.
Skip it if shc config get --scope global backup.repo already names your
bucket. Keep hz.json: the new cluster needs it in step 6.
2. Copy the keys off the cluster
Section titled “2. Copy the keys off the cluster”Two keys, kept together, somewhere that isn’t the cluster.
The first opens the repository. The cluster made it for you:
$ shc vault get backup/password > backup.key$ chmod 600 backup.keyThe second opens the app’s own secrets inside the backup. You make it:
$ openssl rand -base64 32 > secrets.pw$ chmod 600 secrets.pwWithout it the app’s secrets are sealed under the cluster’s vault password, which the portal doesn’t show you.
3. Take the backup
Section titled “3. Take the backup”--all-connected takes the app and its database together, so they restore to
the same moment. The app is paused while it’s read, so do this when nothing is
writing. <env> is default unless you installed the app under another name.
$ shc backup create <app> -t <tenant> -e <env> --all-connected \ --secrets-password-file secrets.pwFor an app with no database of its own, leave --all-connected off.
If your shc doesn’t know --secrets-password-file, update it and the cluster
first. An older cluster loses the password on --all-connected and still reports
success, and that backup won’t open anywhere else.
A schedule can’t name a password, so a scheduled backup is sealed under the vault password and won’t open on a new cluster. Take this one by hand.
4. Check it’s really there
Section titled “4. Check it’s really there”$ shc backup ls -s <app> -t <tenant> -e <env> -o json$ shc backup verify <id> -t <tenant> --deepCheck the size against what you put in the app. A backup can complete and hold less than you expected. Then look at the bucket in the Hetzner console: it should hold objects, and its size should have grown by about that much.
Keep the deployment names and the backup ids. You’ll want them in step 7.
5. Delete the cluster
Section titled “5. Delete the cluster”Delete it in the portal. The bucket stays, because nothing in the delete touches it. Check that it’s still there in the Hetzner console before you go on.
6. Build the new cluster and give it the keys
Section titled “6. Build the new cluster and give it the keys”Build it, sign in with shc auth login as above, and make the tenant again if
your app wasn’t in master and shc tenant ls doesn’t list it:
$ shc tenant add <tenant> --no-adminConnect and grant the bucket again, with hz.json: step 2 of Set a backup
destination. Then hand over
the repository key and point backups at the bucket, in this order:
$ shc vault set --scope global old/backup-key --value-file backup.key$ shc config set --scope global backup.password 'ref+vault://shc/system/old/backup-key'$ shc config set --scope global backup.repo @hzbak/<bucket>All three are global, because the orphan scan in step 7 reads a repository with the key set at the same scope as the repository.
This cluster’s own backups now go to the same bucket under the old key. That’s fine. Its retention only removes what it wrote itself.
The cluster’s Console does the same from Manage > Backups: press Backup
destination, pick the connection, type the bucket, paste backup.key into
Repository key and secrets.pw into Secrets password, and press
Save. It stores both keys in the vault and sets the three keys globally.
7. Find the backup
Section titled “7. Find the backup”The new cluster has no record of your backup, so it shows up as an orphan:
$ shc backup orphan scan -t master$ shc backup orphan ls --all-tenantsEach row has an ID, the deployment it came from under STACK and its age.
foreign-estate under REASON means another cluster wrote it. TENANT says
master, because the repository belongs to the cluster. SOURCE TENANT is where
the app was. locked: secrets-key under STATE is expected here: step 8 brings
that key.
8. Restore
Section titled “8. Restore”Name every id that belongs together. Look at the plan first:
$ shc recover -b <database id> -b <app id> -t <tenant> -e <env> \ --secrets-password-file secrets.pw --dry-runThen run it:
$ shc recover -b <database id> -b <app id> -t <tenant> -e <env> \ --secrets-password-file secrets.pw --yesIf you pick two backups that weren’t taken together, it’s refused:
recovery inputs have different consistency ids.
The app comes back under the hostname it had on the old cluster. See Give the app its address.
In the Console, the backup is on the Orphans tab of Manage > Backups. Open it and use Recover, which fills in its id.
For one deployment on its own, this also works:
$ shc backup restore --id <id> --as <name> --into <tenant> -t master -e <env> \ --secrets-password-file secrets.pw9. Read your file back
Section titled “9. Read your file back”$ shc ls -t <tenant>Both deployments should say running. Open the app, sign in with the password
it had before, and read back a file you put there. Compare it to the original
by its hash, not by looking at it.
Give the app its address
Section titled “Give the app its address”The app answers on the hostname it had before. Point that name’s DNS at the new
cluster. To bring it up under another name instead, add
--override hosts=<new host> to the shc recover above, or
--override <deployment>:hosts=<new host> to rename one deployment only.
When it refuses
Section titled “When it refuses”| You see | It means | Do this |
|---|---|---|
X400486BKPNRP | No backup.repo is set | Step 1 |
X400976TNTNON though you passed -t | Your shc is older than the cluster and dropped the tenant | Update it, see What you need |
X200962CLIOLD | The same, said before anything is refused | Update it, see What you need |
X403910CONGRT | The connection isn’t granted to this tenant | The grants on Set a backup destination |
X403925CONBKT | The grant doesn’t name this bucket | The grants on Set a backup destination |
X503956BKPIRF from orphan scan | A repository didn’t open, usually a wrong backup.password | Check backup.key is the file from step 2, then step 6 |
X409948BKPOLK, needs backup.password | The repository’s key is wrong or missing | Step 6 |
X409948BKPOLK, needs backup.secrets_password | No secrets password is set or given | Pass --secrets-password-file secrets.pw, the file from step 2 |
X409958BKPSNO | No password you gave opens the app’s secrets | Check secrets.pw is the file from step 2. A backup taken without --secrets-password-file opens only on the cluster that made it |
If you’d rather not pass the file each time, store it on the new cluster:
$ shc vault set --scope global old/secrets-key --value-file secrets.pw$ shc config set --scope global backup.secrets_password 'ref+vault://shc/system/old/secrets-key'Related
Section titled “Related”- Set a backup destination: connect the bucket and set
backup.repo - Enable backups: what a backup is and which keys it uses
- S3 connections: connections, grants and
external-s3 - Losing a location: the same recovery when the cluster survives