Economical append-only offsite backups with restic and Wasabi on Debian 10

This guide shows how to install and configure the free backup software restic on Debian Buster to store off-site backups at the cloud storage company Wasabi. The main advantage of this guide is that it provides obligatory append-only backups. This means that once a backup has been made, it is secure in the sense that it cannot be overwritten or erased from the system the backup was made from. This means that the backup archives are protected from both natural disasters at the server of origin (for example fire) as well as man-made accidents (such a mistake at the root shell prompt). The backups are also protected in the sense that they are stored encrypted, which means that only the person or system that has the password for the backup can read it.

Please note that this guide is provided as-is without any warranty whatsoever. The backup software restic has not reached version 1.0 yet, and you should therefore never use it as your only backup solution. To be protected for eventual mishaps, please make sure to take a backup of the system you intend to use for the purpose of this guide before you begin. Also make sure to test that the backup you have taken really works through a test restore.

The guide is based and tested on Debian Buster (Debian 10). Older Debian versions are out of scope of this guide, but if you want to experiment on your own it should be possible to add the Debian Buster apt package repository to install recent versions of restic, tinyproxy and rclone also on Debian Jessie or Stretch.

This guide is structured in five parts:

In the first part, we will prepare an account at the Wasabi cloud storage where we later will store the backups. Wasabi provides a free 30 day trial for testing purposes.

Secondly, we will install and configure rclone, which on one hand provides access to the storage at Wasabi, and on the other hand offers an access interface for restic towards the storage.

Thirdly, tinyproxy - a simple proxy server - will be prepared, as it is needed as an intermediate for making the storage to work in an append-only manner together with restic.

Fourthly, we will set up the backup program restic.

As the very last step we will test both making a backup and a partial restore.

1 Prepare an account at Wasabi cloud storage

1.1 Start a Wasabi account

Wasabi has a 30-day free trial which can be found here:

https://wasabi.com/sign-up/

After filling out the form at the web page above you will receive your access data for the web interface of Wasabi. Use your access data to log in at this page:

https://console.wasabisys.com/

1.2 Create one bucket for data and another for locking

We will now create two so-called "Buckets". A bucket is a storage container in which you store data.

Create the first bucket through clicking on the red icon named Create Bucket which you find in the top right of the web page. The first bucket will be where all the backup data will be stored, which we will refer to later as the DATABUCKET. You know need to choose a bucket name. For example, you can use the following format as a template for the bucket name:

yourdomain.com-restic-data

Please replace yourdomain.com in the template name above with the domain name of the computer you want to backup. After setting the  DATABUCKET name, please write it down. You will need it later to replace the placeholder text DATABUCKET in the templates that will follow later in this guide.

Then choose the region you want the bucket to be in. For optimum performance, it is good to choose the region/continent of the server you want to backup. So in case your server is in Europe you could choose:

eu-central-1

Then press Next.

On the next page two choices are displayed: for Bucket Versioning and Bucket Logging respectively. You can let these choices stay at their default values, i.e. Suspend Versioning and Suspend Logging. Then click Next, and click Create Bucket.

Now, repeat the same procedure and create another bucket. This second bucket will only be used to store the locking information of restic. We will later refer to it as the LOCKBUCKET. Click on the Create Bucket button again, and now choose a name for this second bucket, for example using this format:

yourdomain.com-restic-locks

Please replace yourdomain.com in the template above with the domain name of the computer to be backed up. After setting the LOCKBUCKET name, please write it down. You will need it later to replace LOCKBUCKET in the templates that will follow later in this guide.

(Restic uses locks to make sure that only one client at a time does backups or restores. The reason we need a separate bucket for the locking information is that restic needs to be able to both add and delete locks, and therefore we need a bucket that permits deletions.)

1.3 Create a user

Click on the IAM icon in the topmost icon row. You have now entered Identity & Access management. Click on Users in the left column. Then click on the red icon named + Create user and choose a logical user name, such as the domain name of the server you want to backup. For example, the user name could be system1.example.com. After setting the user name, please write it down so that you can replace USERNAME in the templates that will follow later in this guide with your value.

Under the Access section in the dialogue, check the checkbox named Programmatic (create API key). Then click Next. Create a group for the user. If all your servers were situated under one main domain, this main domain could be a suitable group name. Now click Next. On the following page (policy creation) just click Next. Finally, click on the Create user button.

To the right of the text Secret Key towards the bottom of the dialogue window you find a small link named Show, that you should click on. Save both the ACCESS KEY and the SECRET KEY in a text file. You will need this information later when we configure the access to Wasabi from the client computer you will backup.

1.4 Create a policy

In order for the user we have created to be allowed access to the two buckets, we need to create an access policy.

Click on Policies in the left column of the Wasabi web interface. Then click on the red button Create Policy. You can choose the domain name of the computer you want to backup for the policy name, for example system1.example.com . Write the policy name in the top left field.

Below is a template you will use as a base for the actual policy. Copy the template to a text editor, and then replace the two occurrences of DATABUCKET in the text file with the name you gave to the first bucket, and the two occurrences of LOCKBUCKET with the name you chose for the second bucket.

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": "s3:ListAllMyBuckets",
 "Resource": "arn:aws:s3:::*"
 },
 {
 "Effect": "Allow",
 "Action": "s3:*",
 "Resource": [
 "arn:aws:s3:::DATABUCKET",
 "arn:aws:s3:::DATABUCKET/*",
 "arn:aws:s3:::LOCKBUCKET",
 "arn:aws:s3:::LOCKBUCKET/*"
 ]
 }
 ]
}

Copy the policy you have created from your text editor and paste it in the big, main field of the Wasabi web page dialogue window. Then click Save.

1.5 Apply the policy

The policy now needs to be applied to the user. Click on Users to the left and then click on the user you have created. Now click on the Permissions tab (to the right in the grey horizontal menu). Then start typing the first letters of the policy name you chose when you created the policy in the text field under Start typing to find policies for the user. Click on the name of your policy. Now you have applied the policy to the user.

1.6 Set the data bucket to be immutable (append-only)

For security reasons we want it to be impossible to overwrite already made backups from the client (the computer the backups are made from). In order to accomplish this we use the Wasabi immutable feature. In the web interface of Wasabi click on Storage (in the top bar). Then click on the DATABUCKET (probably the one that ends with restic-data). Click on the white cogwheel for Settings (towards the upper right). Then click on Compliance in the white upper bar. Click on the slider to the right of Compliance Mode to activate this mode. If you want to you can activate Delete After Retention but you have to make sure to put a large value in Retention Time, this is the minimum time every file stays after the client tries to delete it. For example, put 100 years here. (It seems like you cannot leave this field empty for an infinite retention time.) This means that data is safe for 100 years during which period it cannot be deleted nor modified. Press Save to save your settings.

1.7 Create the lock folder

In the web interface of Wasabi click on Storage (in the top bar). Then click on the LOCKBUCKET (the one that ends with restic- locks).

Now press the green button named Create folder. Name the folder locks so that it can be used for the restic lock files later on.

We are now finished with the configuration in the Wasabi web interface. In the next section, we will start to work at the terminal of the computer we want to backup.

2 Install and configure rclone

2.1 First, make a backup via your existing system

Now it is time to access the computer the backups shall be taken from.

Please remember to take a backup of this computer with an existing system! Also please test that the backup works through a test restore before you continue. In case something would not work as expected, it is critical to be able to revert to a functioning stage.

2.2 Install rclone

All lines shown below should be typed into the terminal followed by Enter.

Connect to the terminal of the computer you want to make backups from, for example via SSH. Then issue this command to become root:

su -

followed by Enter and the root password. Root access is needed, as we will install new programs from the Debian repository. 

Now install rclone:

apt install rclone

2.3 Configure rclone to connect to Wasabi

Then to configure rclone, type this command:

rclone config

Then type:

n

Now it is time to choose a name for the config. Please note that. (dot) is not allowed in the name, but on the other hand - (dash) is allowed. In order to name your rclone configuration use this template, but first replace yourdomain-com with your own domain name, first replacing. (dots) in the domain name with - (hyphens):

wasabi-yourdomain-com-restic

Save the name you give the rclone config, as you later will replace the placeholder RCLONECONFIG with the name.

Then print 4 for Amazon S3 Compliant Storage Providers and Enter

Then print 7 for Wasabi Object Storage and Enter

Then print 1 for Enter AWS credentials in the next step and Enter

Copy the ACCESS KEY that you saved earlier in a file and paste it, then press Enter

Then copy and paste the SECRET KEY you also saved in the same file earlier, followed by Enter

Print the region you choose for your buckets. You can find the region if you click on Storage in the top row of the Wasabi web interface. For example type this region for Europe and then Enter:

eu-central-1

For Endpoint for S3 API, type this if the bucket is in the EU region followed by Enter:

s3.eu-central-1.wasabisys.com

If your bucket is in the US, please find the name of your endpoint in this document:

https://wasabi-support.zendesk.com/hc/en-us/articles/360015106031-What-are-the-service-URLs-for-Wasabi-s-different-regions-

Leave Location constraint empty and press Enter

Press 1 for Owner gets FULL_CONTROL and then Enter

Press y for Edit advanced config and then Enter

Press Enter for default value for chunk size

Press  Enter for default value for disable checksum

Press  Enter for default value for session token

Type  16 for Upload concurrency and then Enter

Press Enter for default value for force_path_style 

Press Enter for default value for v2_auth followed by Enter

Press y for Yes this is OK and Enter

Press q to quit config.

Now the configuration for rclone is finished.

2.4 Test that rclone works with Wasabi

Now create a test file through this command :

echo test >> /tmp/test.txt

Transfer the test file through the command below. However, you first need to change RCLONECONFIG to your value, and also DATABUCKET to your value for this parameter. Everything is on one line:

rclone -v sync /tmp/test.txt RCLONECONFIG:DATABUCKET/

If everything works you should now see the test.txt file in the Wasabi web interface for the DATABUCKET. Please note that sometimes it can take up to approx. 1 minute until the file is shown. Also test a transfer with the other bucket, the one ending with restic-locks:

rclone -v sync /tmp/test.txt  RCLONECONFIG:LOCKBUCKET/

Now check the Wasabi web interface again, for the text file that should be in the LOCKBUCKET.

2.5 Setup rclone to serve restic via systemd

Rclone has a built-in feature of using a cloud storage space (Wasabi in our case) and serve it to restic in a compatible way. This way of running rclone is done via the command rclone serve restic which works as a server process. We will need two of these server processes: one main for almost everything, and a second one which only deals with the restic lock files.

First, create a restic user:

adduser restic

Copy the rclone configuration from the root user to the restic user:

su restic
mkdir -p /home/restic/.config/rclone
exit
cp -a /root/.config/rclone/rclone.conf /home/restic/.config/rclone/
chown restic.restic /home/restic/.config/rclone/rclone.conf

Create a systemd control file, for example through the nano editor:

nano /etc/systemd/system/restic-data.service

The file should contain the data below. Please note that as before you need to change the RCLONECONFIG value as well as the DATABUCKET value to your settings.

[Unit]       
Description=Rclone serve restic data on wasabi
After=network.target

[Service]
Type=simple
User=restic
Group=restic
ExecStart=/usr/bin/rclone serve restic RCLONECONFIG:DATABUCKET --addr=127.0.0.1:8001 --append-only --retries 10 --transfers 20 --s3-upload-concurrency 8 --s3-chunk-size 16M
Restart=always
RestartSec=5
StartLimitInterval=0

[Install]
WantedBy=multi-user.target

Save the file. The purpose of it is for rclone to provide a restic interface towards the Wasabi cloud storage bucket. The systemd service we have just created will start automatically on the next boot, but first we need to activate it through these commands:

systemctl daemon-reload
systemctl enable restic-data
systemctl start restic-data

Then we check that the service have started:

systemctl status restic-data

You should now see Active: active (running) if everything works correctly. Press q to quit the status display.

Now we have installed the rclone serve restic service on port 8001 which connects to a bucket that is append-only and will store the backups.

Next, we need another service on port 8002 for the restic locking system, connected to the second bucket which is read+write. Therefore we will repeat the steps we just have done once more with minor modifications.

Create a systemd control file :

nano /etc/systemd/system/restic-locks.service

The file should contain the data below. The line that begins with ExecStart= is a single, long line split into three lines below. Please change the RCLONECONFIG value as well as the LOCKBUCKET value to your settings.

[Unit]       
Description=Rclone serve restic locks on wasabi
After=network.target

[Service]
Type=simple
User=restic
Group=restic
ExecStart=/usr/bin/rclone serve restic RCLONECONFIG:LOCKBUCKET --addr=127.0.0.1:8002
Restart=always
RestartSec=5
StartLimitInterval=0

[Install]
WantedBy=multi-user.target

Save the file and activate it through:

systemctl daemon-reload
systemctl enable restic-locks
systemctl start restic-locks

Check that the service have started:

systemctl status restic-locks

You should now see Active: active (running).

Press q to quit.

Now we have created both rclone serve restic systemd services, first on port 8001 for the main service in append-only mode and then on port 8002 for the locking service in read+write mode.

3 Prepare tinyproxy

Tinyproxy will fuse together the two services into a unified front presented to restic. It is installed in a root terminal of the computer you want to back up:

apt install tinyproxy

Then create a backup of the tinyproxy configuration file:

cp -a /etc/tinyproxy/tinyproxy.conf /etc/tinyproxy/tinyproxy.conf.bak

If the file cannot be found here, it can often be found at / etc/tinyproxy.conf instead.

N ow, edit the configuration file:

nano /etc/tinyproxy/tinyproxy.conf

You should make a number of changes to the files which will now be detailed. First, comment away the preset port number and set a new port number:

#Port 8888
Port 8000

Secondly, it is important to only allow connections from localhost, thus add a Listen line:

#Listen 192.168.0.1
Listen 127.0.0.1

Thirdly, change the timeout value to one hour:

#Timeout 600
Timeout 3600

Fourthly, connect tinyproxy to the two rclone serve restic systemd services. They have to be written in the order shown below: the general rule first and the specific rule last, as the last rule in tinyproxy always win in case of ties.

ReversePath "/" "http://127.0.0.1:8001/"
ReversePath "/locks" "http://127.0.0.1:8002/locks"

Finally, let tinyproxy operate exclusively in reverse-only mode:

#ReverseOnly Yes
ReverseOnly Yes

Now, all necessary edits have been made. Please save the configuration file, in nano you do this by pressing Ctrl+O, and then exit nano with Ctrl+X to go back to the terminal.

In order to activate tinyproxy we will now restart it:

systemctl stop tinyproxy
systemctl start tinyproxy

Finally please check that tinyproxy is running correctly:

systemctl status tinyproxy

Press q to exit. Now, tinyproxy is configured for use by restic. In the next section, we will setup restic in itself.

4 Set up the restic backup program

Finally, we will perform the last step of the configuration, which is to set up the restic backup program. First, install restic:

apt install restic

Then we need to initialize the environmental variables for restic. First we specify the restic repository location by entering this in a terminal:

export RESTIC_REPOSITORY='rest:http://127.0.0.1:8000/'

Please now decide on a strong passphrase for your backups. It is important that you store the passphrase in a secure place. The passphrase is required to be able to access the backups as these are fully encrypted. Replace PASSPHRASE below with the password you have chosen and then enter the command in the terminal:

export RESTIC_PASSWORD='PASSPHRASE'

Now let us create the restic repository:

restic init

Let's test that restic works as it should. We have already stored the password in the RESTIC_PASSWORD environmental variable, so we can access the repository to list snapshots:

restic snapshots

If everything goes as it should, we should now get an empty list (as we have not made any backup yet) and the text 0 snapshots.

We have now prepared everything for the storage of the backups and can move on to the final steps of performing backups and restores.

5 Test both backup and a partial restore

In the last step of this tutorial we will do a full backup of the system, and then test a partial restore. For this purpose two shell scripts will be created (one for backup and another for restore).

5.1 Prepare a backup script

Firstly, you need to have a backup script that can be run via cron at a suitable interval, for example, one time per day. The backup script can, for example, be created as a file only root has access to and then edited with nano:

touch /usr/local/bin/backup-restic.sh 
chown root.root /usr/local/bin/backup-restic.sh
chmod 700 /usr/local/bin/backup-restic.sh
nano /usr/local/bin/backup-restic.sh

Below is an example of the content of the backup script, in which you need to change PASSPHRASE to the passphrase you chose earlier.

#!/bin/sh
export RESTIC_REPOSITORY='rest:http://127.0.0.1:8000/'
export RESTIC_PASSWORD='PASSPHRASE'      
restic backup / --exclude=/dev --exclude=/proc --exclude=/sys --exclude=/run --exclude=/tmp --exclude=/mnt --exclude=/root/.cache

Please observe the exclusions made above – you may want to change the directories that are excluded from backup.

5.2 Run the backup script

Save the file and run it through the terminal:

/usr/local/bin/backup-restic.sh

After some seconds you should see how restic traverses your files and also an estimated percentage of how much of the backup has been completed. ETA is a rough estimate of how much time is left until finalization in minutes and seconds. The first backup takes a quite long, but then the following backups are rapid as only files that have changed or have been created since the last backup need to be transferred.

In order for the backup to run regularly, add a line to the crontab. First edit /etc/crontab with:

nano /etc/crontab

Then add the scheduling for the backup. For example, if the backup should run at 02:00 in the early morning each day add this line to /etc/crontab :

00 02           * * *   root    /usr/local/bin/backup-restic.sh

(Please note that cron operates on the format mm:hh – that is minutes before hours.)

5.3 Test that restore works

A backup system is never complete until one has tested that restores work well. We will use the fuse mount of restic to browse through the latest backup. Type this in the terminal, changing PASSPHRASE to the restic passphrase you chose earlier:

mkdir /mnt/restic
export RESTIC_REPOSITORY='rest:http://127.0.0.1:8000/'
export RESTIC_PASSWORD='PASSPHRASE'
restic mount /mnt/restic &
cd /mnt/restic
cd snapshots
cd latest
ls

Now you will be able to see the last snapshot (the last backup made). You can compare with diff that a file has been backed up correctly, for example for /etc/fstab :

diff /mnt/restic/snapshots/latest/etc/fstab /etc/fstab

If you get no output from diff, the two files are identical and the backup has worked for that file. If you, on the other hand, have changed the file since the last backup you will see which lines in the files that differ.


Finally, we will unmount the restic fuse mount:

cd ~
umount /mnt/restic/

5.4 Final remarks

Congratulations, if everything went well you have now accomplished the setup of a secondary backup system that has several robust features:

The backup system is off-site, which means that it protects data from accidents that could happen to the computer of which the backups are made.

It is also append-only, which means no virus or accident on the computer the backups are made from can ever overwrite a backup already made earlier. The only way to delete backups is through login via the Wasabi web interface, and therefore you should protect your login data to the Wasabi web interface. Preferably you can enable 2FA (two factor authentication) for higher security.

If you enable the cron job the backup system is also automatic. However, always remember to test restores regularly so that you can verify that the backup works.

Compared to many other online cloud storage services using Wasabi has the advantage of speedy transfers to a competitive price.

This system works well for secondary backups, as extra insurance in case, the first backup system would fail for any reason. Finally, one important advantage is that it scales to whatever size you need - you simply pay per GB storage (albeit you always have to pay for minimum 1 TB) and practically do not have any threshold for maximum storage space.

Share this page:

4 Comment(s)