LabSrc

In my last post, I touched on ways you could automate remote database backups using SSH and passphrase-less keys. I didn't really go in-depth as I felt this topic deserved its own article and wouldn't you know it, here it is!

Now, as you can tell by the title, we're going to be focusing on using SFTP as our transfer protocol of choice instead of SSH or SCP as I've found its a lot easier to secure access with an SFTP only account. Because we'll be using passphrase-less keys to allow for automated backups to our prison, locking down our backup account becomes priority one. If someone gets ahold of those keys, we want to severely limit what that person can do on the backup server. One of the ways we'll be doing that is by chrooting our SFTP account's home directory.

If this is the first time you've heard of the term chroot, it basically means the creation of a directory jail which an account is confined to. Once connected via SFTP, our backup account will not be able to traverse the filesystem like a normal user. Its home directory will instead appear as the "/" or root directory. Along with chrooting, I'll also show you how to use inotify & chattr to lockdown the permissions of our jail and keep our backups pristine and untouched.

Please keep in mind that if you're a hardcore rsync devotee, this guide probably isn't for you. Rsync requires shell access which an SFTP only account doesn't provide. Keep in mind that preventing shell access provides a huge benefit in terms of security so even if you're hooked on rsync, I still encourage you to give this article a read. There are a ton of other backup utilities & methods that do work with SFTP only accounts. I'll even provide some examples at the end of this article as an added incentive for you not to tap out here.

As usual, I'll be using the latest LTS release of Ubuntu which at the time of writing this article is 18.04, but setting this up on any Linux distro should be fairly similar.

Creating Our Prisoner

First up, we'll create a new account on our backup server.

sudo useradd -s /usr/sbin/nologin backupuser

This command creates a new account with the username backupuser. The account is created without a password which means it will not be allowed to login to the server with standard password authentication. Don't worry as we'll be using our SSH keys to allow our account remote access.

As an added safeguard, we've also created our backup account with it's shell set to /usr/sbin/nologin instead of the ever-popular bash. In actuality, nologin is not even a shell. It's a command which automatically boots the user from the system if a login is attempted.

*NOTE: Other Linux distros may house nologin in a different directory such as /sbin/nologin. If you're not using Ubuntu, make sure to locate the correct path of nologin before creating your backup account by running whereis nologin.

Now that we have our prisoner, it's time to build a nice, cozy jail to live in.

Building the Jail

We'll now create the backup account's home directory with a few minor alterations. If you chose a different username in the previous step, make sure to change backupuser to the username that was used.

sudo install -d -g backupuser -m 750 -v /home/backupuser

Some of you may be wondering why we didn't just have the useradd command automatically create the home directory for us. The reason is we need to make sure the backup account's home directory is owned by root and the backup account is given read and execute permissions only. If it isn't, our chrooted SFTP jail will refuse connections and we won't be able to login with the SSH keys we'll be creating. We also don't want all the default shell files created within the home directory.

When all is said and done, you can run ls -la /home to check that the backup account's home directory has the following ownership and permissions.

*TIP: The install command may be new to some of you. In fact, I only stumbled upon it while creating this guide. Despite the name, its main purpose is for copying files, but as you can see, it's a versatile utility that can also be used to create directories and set ownership and permissions all in one go.

The last thing we need to do is create a directory within our jail that our backup account can upload to. This is necessary because the backup account does not have write access to it's home directory.

sudo install -d -o backupuser -g root -m 370 -v /home/backupuser/upload

You may notice the unique permissions given to our backup account for the upload directory. The backupuser account has write & execute permissions to the upload directory, but doesn't have read permissions. This permits the creation and deletion of files, but doesn't allow the listing or viewing of any existing files.

These permissions work fine with command-line SFTP clients like sftp & LFTP, more on this later, but doesn't seem to play nice with GUI-based applications like Filezilla & WinSCP. This shouldn't be a problem if you plan on automating your backups, but will be an issue if you can't seem to break the chains of the GUI...RIP Dany. If that's the case, you'll be forced to make the permissions more lax and change them to 770 instead, but I would advise against it.

So we've got our prisoner, we've built the jail, now we need to open the gates to the prison.

Opening the Gates

We'll be using OpenSSH to create a user specific config, which will restrict the backup account to SFTP only connections. No SSH or SCP connections will be allowed.

sudo nano /etc/ssh/sshd_config

Append the following lines to the end of your /etc/ssh/sshd_config file. These configs will apply strictly to the backup account and won't affect any other user connecting to the server via OpenSSH. Like before, if you chose a different name for your backup account, make sure to change backupuser to the chosen username.

After making the changes to the OpenSSH server config, we'll need to bounce our ssh service for it to take affect.

sudo systemctl restart ssh

To umask or Not to umask

A lot of people include a umask permission flag when configuring their SFTP chrooted jails to provide added protection for their uploaded files. One problem I've found with this method, is that umask only removes permissions and doesn't add permissions to files and directories that are uploaded with more restrictive rights.

Another issue is that no matter what permissions you set with umask, the SFTP account will always have the ability to delete any and all uploaded files. This is because in Linux, as long as a user has write permissions to a parent directory, they always have the ability to delete or modify any files and directories within that directory regardless of their permissions or ownership. It took me awhile to figure this out, but it's true. If you're sceptical, I urge you to test this out yourself...I'll wait.

My last gripe with umask is it doesn't have the ability to change the ownership of uploaded files and directories.

Don't worry about the woes of using umask as we'll be utilizing inotify & chattr to overcome these shortcomings.

The last thing that needs to be done to grant remote access to our SFTP jail is create a set of passphrase-less SSH keys. I know security conscious individuals may be reluctant to do so, but the whole point of this article is to setup a jail that allows automated backups. We've already done a lot to mitigate what this backup account can do in the event your keys are compromised. And as just mentioned, we'll be taking further steps to prevent the tampering of your backups, but first, let's create some keys!

The Keys to the Prison

To make things easier, we'll first create the backup account's SSH directory. By now you should be familiar with the install command.

sudo install -d -o backupuser -g backupuser -m 500 /home/backupuser/.ssh

Next we'll create the keypair within the .ssh directory and change its permissions. Hit Enter twice when prompted.

sudo ssh-keygen -t rsa -b 4096 -f /home/backupuser/.ssh/backupkey
sudo chown backupuser:backupuser /home/backupuser/.ssh/* && sudo chmod 400 /home/backupuser/.ssh/backupkey.pub

We then need to create the authorized_keys file to allow authentication with our SSH key pair.

sudo install -o backupuser -g backupuser -m 400 /dev/null /home/backupuser/.ssh/authorized_keys
sudo cat /home/backupuser/.ssh/backupkey.pub > /home/backupuser/.ssh/authorized_keys

You'll have to securely copy the private key, backupkey to the server you'll be backing up from. This can either be done via SSH or physically with a USB flash drive. If you'll be storing backups from multiple servers, the key can be copied to each of them. Just remember to keep this key safe.

sudo scp /home/backupuser/.ssh/backupkey user@remoteserver:/home/user/.ssh

*NOTE: In the command above, the key is scp'd to the ssh directory of a user on the remote server. For security purposes, the key should then be moved to the ssh directory of a dedicated backup account or to the root account's ssh directory if one is not available.

Once you have it copied to its final destination or destinations, delete the private key from the backup server.

sudo rm /home/backupuser/.ssh/backupkey

Now we're finally ready to test if we can connect to our jail from our remote server.

Testing the Locks

We'll use the command-line SFTP client to test out our newly constructed jail from one of the servers we copied our private key to.

sftp -i ~/.ssh/backupkey backupuser@backupserverIP

Make sure you specify the correct location of the backupkey on the remote server and change backupserverIP to the IP address or hostname of your backup server.

If you did everything correctly and the Linux gods have smiled upon you, you should be connected to the remote server via SFTP and chrooted to the backup account's home directory. You can confirm this by running pwd while connected. It should show the directory as Remote working directory: /. Running ls -la should reveal the .ssh & upload directories.

Don't be worried if the owner and group shows up as numbers or even a different name. In Linux, users and groups are identified by ID number. If an ID number on the backup server matches a user or group on your remote server, that name and not backupuser will show up instead. If the ID doesn't match anything on your remote server, the ID number itself will be listed as shown above.

Prison Lockdown

Earlier in the article, I mentioned that we'd take steps to further enhance the protection of the backups within our chrooted SFTP jail. Our main goal here is to prevent the backup account from being able to read, delete, modify or download any backups once they've been uploaded. This means that even if your SSH keys are compromised, the only thing an attacker can do is upload new files. We'll accomplish this by using two packages, inotify-tools & chattr.

Without going too indepth, inotify-tools provides a set of utilities that will allow us to monitor any changes to the jail's upload directory. We can then use chattr to change the attributes of any newly uploaded files and directories. The reason chattr is necessary is because Linux permissions are very broad. Attributes on the other hand, are a lot more granular and can lockdown files and directories in a much more specific manner.

As mentioned in the previous section about umask, once a user has write access to a directory, they also have the ability to delete or modify any files within that directory regardless of the ownership or permissions of said files. Setting attributes on files after they are uploaded will allow us to get around this problem and ensure our backups aren't tampered with.

Creating the Security Monitor

If you're using Ubuntu, chattr is already installed as part of the e2fsprogs package, so all we need to do is install inotify-tools.

sudo apt install inotify-tools

Now that both packages are installed, we'll need to create a simple shell script that will monitor our upload directory and make any new files immutable or undeletable & unmodifiable (even by root) and any new directories appendable only. Don't worry, these settings are not set in stone and can be changed by root.

sudo install -d -o root -g root -m 700 /opt/inotify
sudo nano /opt/inotify/backup-dir-monitor.sh

Copy and paste the following lines into the script

#!/bin/sh
username="backupuser"
monitorDir="/home/backupuser/upload"
inotifywait -m -r -e create --format '%w%f' "$monitorDir" | while read newObject
do
	# If object is a directory
	if [ -d "$newObject" ]; then
		chown "$username":root "$newObject"
		chmod 370 "$newObject"
		chattr +a "$newObject"
	# If object is a file
	else
		chown root:root "$newObject"
		chmod 600 "$newObject"
		chattr +i "$newObject"
	fi
done

Next, we'll change the permissions on the script.

sudo chmod 700 /opt/inotify/backup-dir-monitor.sh

Now we'll launch the script as a background process.

sudo /opt/inotify/backup-dir-monitor.sh &

With our script running, it's time to test if inotify & chattr are doing their jobs.

Testing the Safeguards

Start off by creating a test file by running, touch testfile on your remote server.

Next, connect to the SFTP jail's upload directory from the remote server.

sftp -i ~/.ssh/backupkey backupuser@backupserverIP:/upload

Run the following commands in the sftp prompt.

put testfile
ls
rm testfile
mkdir testdir
cd testdir
rmdir testdir
put testfile
ls
rm testfile
exit

Don't freak out if a bunch of those commands failed. All of the ls, rm & rmdir commands should fail if our inotify/chattr script is working properly.

On a terminal connected to the backup server, let's take a look at the ownership and permissions within the SFTP jail's upload directory.

sudo ls -la /home/backupuser/upload

The testdir directory remains owned by the backup account because we still need the ability to add new files to it, but the testfile file is owned by and permissioned only for root.

You won't be able to use ls to see the attributes set with chattr. For that, you'll need to run lsattr instead.

sudo lsattr /home/backupuser/upload

As you can see, the testdir has an "a" in its attributes meaning the directory is append only. The backup account, or any account for that matter, will not be able to delete or modify any files within the directory because of this attribute.

The testfile has an "i" in its attributes meaning its immutable or as stated earlier, completely undeletable & unmodifiable even by root.

Flipping the Switch

If you have a script that deletes old backups or just want to manually do some housekeeping, you first need to remove the attributes from the files and directories in the upload directory.

sudo chattr -Rai /home/backupuser/upload

This command will remove the immutable and append only attributes from all files and directories within upload directory because of the recursive flag, "-R".

Reapplying attributes for backups with sub-directories may be more complicated depending on how complex your directory structure is. For a simple setup with a parent directory housing just files, you can easily reapply proper attributes with the following commands.

sudo chattr +a /home/backupuser/upload/testdir
sudo chattr +i /home/backupuser/upload/testdir/*

To reapply attributes for more complex directory structures, it's best to write a shell script that will apply the append only attribute to all directories and the immutable attribute to all files.

Below is a simple script that will recursively go through the jail's upload directory and do just that.

#!/bin/bash
uploadDir="/home/backupuser/upload"
find "$uploadDir" | while read newObject
do
	if [ -d "$newObject" ]; then
		chattr +a "$newObject"
	else
		chattr +i "$newObject"
	fi
done

Either of these methods can be worked into a script that cleans up old backups.

Automating the Lockdown

The last thing we're going to do is make sure our script runs at startup. Ubuntu now uses Systemd to manage startup processes, so we'll be creating a unit file for our inotify script which will be started automatically on bootup. It also makes it easier to start and stop our script.

sudo nano /etc/systemd/system/inotifysftp.service

Copy and paste the following into the unit file.

[Unit]
Description=inotify SFTP watch

[Service]
ExecStart=/opt/inotify/inotifyChattr.sh

[Install]
WantedBy=multi-user.target

Now we'll start the service and check on it's status.

sudo systemctl start inotifysftp
sudo systemctl status inotifysftp

If everything is active and green, we can go ahead and enable the service so it starts up at boot time.

sudo systemctl enable inotifysftp

Reboot your backup server and check the status one more time to make sure the inotifysftp started up automatically.

Prison Transports

In this final section, I'll provide examples for a couple of utilities you can use to transport your precious backups to our prison.

The first one, LFTP, is a straightforward alternative for the SFTP command-line client. It works seamlessly with our SFTP chrooted jail and the additional security measures we've implemented with inotify & chattr. It has the added benefit of being able to accept piped data so you don't need to save files locally before uploading.

The second, restic, in my opinion, is a much improved solution for those of you reliant on rsync for backups. Restic stores backups in a repository which provide for more efficient incrementals. More importantly, restic provides strong built-in encryption. While it does take some changes and maneuvering to get restic to work with our restrictive SFTP jail, it can be done, where with rsync, it's just not possible.

Did you say L...FTP?

If you want an easy way to take advantage of the prison we've built along with all of the safeguards, I suggest using LFTP as your SFTP client. LFTP, works with the passphrase-less SSH keys we've created and has the ability to transfer data piped directly to it. Backups can easily be scripted with LFTP or just run directly as a cron job.

Here's an example creating a Bzip2 archive of a directory and piping directly to LFTP.

sudo tar -C /etc -cjvp /etc/importantdir | sudo lftp -u backupuser, sftp://backupserverIP -e "set sftp:connect-program 'ssh -a -x -i /root/.ssh/backupkey'; cd upload; put /dev/stdin -o importantdir_`date +\%Y-\%m-\%d`.tar.bz2"

*TIP: The -C in the tar command forces tar to change to the parent directory before compressing the target directory. This allows you to include only the target directory in the archive and not the entire parent directory tree.

Below are a couple of examples using mysqldump piped directly to LFTP. The second example is for those of you coming from my previous article, Automating MariaDB Backups.

mysqldump testdb | sudo lftp -u backupuser, sftp://backupserverIP -e "set sftp:connect-program 'ssh -a -x -i /root/.ssh/backupkey'; cd upload; put /dev/stdin -o testdb_`date +\%Y-\%m-\%d`.sql;"

*NOTE: Because we're piping mysqldump directly to LFTP, you're never given the opportunity to input your SQL password. You'll instead need to create an Option .cnf file to store your mysqldump credentials or risk exposing your password while the process is running. Again, see my last post for more info.

sudo mysqldump --routines --triggers --events --quick --single-transaction --all-databases | sudo bzip2 | sudo openssl smime -encrypt -binary -text -aes256 -outform DER /etc/mysql/mdbbackup-pub.key | sudo lftp -u backupuser, sftp://backupserverIP -e "set sftp:connect-program 'ssh -a -x -i /root/.ssh/backupkey'; cd upload; put /dev/stdin -o alldb_`date +\%Y-\%m-\%d`.sql.bz2.enc;"

Restic > Rsync

In the intro, I mentioned that rsync won't work with an SFTP only account because it requires shell access. If you need a backup method that works with an SFTP jail and does incrementals & backup rotation, while also throwing verification & encryption into the mix, restic is exactly what you're looking for. I only recently discovered restic myself and while the setup was a little challenging, the features are vast and ever growing.

The only drawback with restic in terms of our SFTP jail, is that you won't be able to utilize the inotify script (backup-dir-monitor-.sh) as is. Like rsync, restic needs to be able to read and modify the files and directories created within the SFTP jail. You could script the changes to the permissions & attributes before and after restic is run, but depending on the size of your backups, it may significantly extend your runtime. After more testing, I hope to do a future article on restic which references back to this guide and includes a restic-friendly inotify script.

If you'd like to give it a try now, I highly recommended grabbing the latest binary for your architecture from restic's GitHub page. Since binaries are by nature self-contained, there's no dependencies that need to be installed. I recommend creating a directory for it in /opt and then making an alias or adding the directory to the PATH in your .bashrc or .bash_profile. It would also be a timesaver to create a symbolic link to the latest binary and naming it restic to allow for easier future upgrades.

I can't take you through the entire setup process in this article, but once you have a restic repository created within the SFTP Jail, you should add an SSH config file to the SSH directory of your remote server's backup or root account. This is to allow restic to automatically use the SSH private key during the SFTP login process.

sudo install -o root -g root -m 600 /dev/null /root/.ssh/config
sudo nano /root/.ssh/config

Add the following lines to your config file

Below are a couple of example restic backup commands.

sudo restic -r --password-file /root/.restic sftp:backupuser@backupserver:upload/restic-repo --verbose backup /etc/importantdir

*TIP: Restic backup repositories require a password when accessing. For automation purposes, you can save the password in a text file and reference it like so--password-file /root/.restic. Just remember to chmod it to 600.

sudo mysqldump --routines --triggers --events --quick --single-transaction --all-databases | sudo bzip2 | sudo openssl smime -encrypt -binary -text -aes256 -outform DER /etc/mysql/mdbbackup-pub.key | sudo restic --password-file /root/.restic -r sftp:backupuser@backupserver:upload/restic-repo --verbose backup --stdin --stdin-filename alldb.sql.bz2.enc

*NOTE: Restic version tracks each backup, so it's important that backup files always have the same name. This is why it's best not to include the date in your backup filenames.

In Closing...

I'm happy I was able to write this as a follow-up to my Automating MariaDB Backups guide as I feel the two go hand-in-hand. I know the article looks long & complicated, but in reality, it doesn't take very long to set up an SFTP Chrooted Jail and the security it provides is well worth the effort. If you have any questions, feel free to comment down below and remember, Always keep your guard up!