Very long post ahead (sorry for the wall of text), part of a series of posts on some sysadmin topics, see post 1 and post 2. I want to show you how I set up my tiny dedicated server to have encrypted partitions, and to reinstall it from scratch. All of this without ever accessing the actual server console.

Introduction

As much as my provider may have gold standards on how to do things (they don't, there are some very bad practises in the default installation, like putting their SSH key into root's authorized_keys file), I wouldn't trust an installation done by a third party. Also, I wanted to have all my data securely encrypted.

I know this is not perfect, and there are possible attacks. But I think it is a good barrier to have to deter entities without big budgets from getting my data.

I have done this twice on my servers, and today I was reviewing each step as my friend was doing the same thing (with some slight differences) on his brand new server, so I think this is all mostly correct. Please, tell me if you find a bug in this guide.

This was done on my 12 £/month Kimsufi dedicated server, sold by OVH (see my previous post on why I chose it), and some things are specific to them. But you can do the same thing with any dedicated server that has a rescue netboot image.

The process is to boot into the rescue image (this is of course a weak link, as the image could have a keylogger, but we have to stop the paranoia at some point), manually partition the disk, set-up encryption, and LVM; and then install a Debian system with debootstrap.

To be able to unlock the encrypted disks, you will have to ssh into the server after a reboot and enter the passphrase (this is done inside the initrd phase). Once unlocked, the normal boot process continues.

If anything fails, you end up with an unreachable system: it might or might have not booted, the disk might or might not be unlocked, etc. You can always go back into the rescue netboot image, but that does not allow you to see the boot process. Some providers will give you real remote console access, OVH charges you silly money for that.

They used to offer a "virtual KVM", which was a bit of a kludge, but it worked: another netboot image that started a QEMU connected to a VNC server, so by connecting to the VNC server, you would be able to interact with the emulated boot process, but with a fake BIOS and a virtual network. For some unspecified reason they've stopped offering this, but there is a workaround available. The bottom line is, if you have some kind of rescue netboot image, you can just download and run QEMU on it and do the same trick.

Update: Online.net (which has a similar offering to OVH/Kimsufi) now provides serial console access, which is way more convenient than the virtual KVM, but note that it will be attached to the second serial port (ttyS1). This drove me crazy for hours!

The gritty details

Start by netbooting into your rescue image. For OVH, you'd go to the control panel, in the Services/Netboot section and select "rescue pro". Then reboot your server. OVH will mail you a temporary password when it finishes rebooting.

Connect to it, without saving the temporary SSH key:

$ ssh -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no root@${IP}

Note that you will need these packages installed in your rescue image:

cryptsetup lvm2 debian-archive-keyring debootstrap

If they are not present, now it is a good time to install them.

Disk partitioning and encryption

For the rest of the text, I am assuming you have one hard drive called /dev/sda. We start by partitioning it:

# fdisk /dev/sda

Start a new partition table with o, and then create two primary partitions: a small one for /boot at the beginning (100 to 300 MB would do), and a second one with the remaining space. Set both as type 83 (Linux), and don't forget to activate the first one, as this servers refuse to boot from the hard drive without that.

Create the file system for /boot, and the encrypted device:

# mkfs.ext4 -L boot /dev/sda1
# cryptsetup -s 512 -c aes-xts-plain64 luksFormat /dev/sda2

The encryption parameters are the same as the ones used by the Debian Installer by default, so don't change them unless you really know what you are doing. You will need to type a passphrase for the encrypted device, be sure not to forget it! This passphrase can later be changed (or secondary passphrases added) with the cryptsetup tool.

Look up the crypt device's UUID, and save it for later:

# cryptsetup luksDump /dev/sda2 | grep UUID:
UUID:           xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Open the encrypted device (type the passphrase again), and set up the LVM volume group:

# cryptsetup luksOpen /dev/sda2 sda2_crypt
# pvcreate /dev/mapper/sda2_crypt
# vgcreate vg0 /dev/mapper/sda2_crypt

Create the logical volumes, this is of course a matter of personal taste and there are many possible variations. This is my current layout, note that I put most of the "big data" in /srv.

# lvcreate -L 500m -n root vg0
# lvcreate -L 1.5g -n usr vg0
# lvcreate -L 3g -n var vg0
# lvcreate -L 1g -n home vg0
# lvcreate -L 10g -n srv vg0
# lvcreate -L 500m -n swap vg0
# lvcreate -L 100m -n tmp vg0

Some possible variations:

  • You can decide to use a ramdisk for /tmp, so instead of creating a logical volume, you would add RAMTMP=yes to /etc/default/tmpfs.
  • You can merge / and /usr in one same partition, as neither of them change much.
  • You can avoid having swap if you prefer.
  • You can put /home in /srv, and bind mount it later.

Now, create the file systems, swap space, and mount them in /target. Note that I like to use human-readable labels.

# for i in home root srv tmp usr var; do 
  mkfs.ext4 -L $i /dev/mapper/vg0-$i; done
# mkswap -L swap /dev/mapper/vg0-swap

# mkdir /target
# mount /dev/mapper/vg0-root /target
# mkdir /target/{boot,home,srv,tmp,usr,var}
# mount /dev/sda1 /target/boot
# for i in home srv tmp usr var; do
  mount /dev/mapper/vg0-$i /target/$i; done
# swapon /dev/mapper/vg0-swap

Don't forget to set the right permissions for /tmp.

# chmod 1777 /target/tmp

If you want to do the /home on /srv, you'll need to do this (and then copy to /etc/fstab):

# mkdir /target/srv/home
# mount -o bind /target/srv/home /target/home

Debian installation

The disk is ready now. We will use debootstrap to install the base system. The OVH image carries it, otherwise consult the relevant section in the Install manual for details. It is important that at this point you check that you have a good GPG keyring for debootstrap to verify the installation source, by comparing it to a good one (for example, the one in your machine):

# gpg /usr/share/keyrings/debian-archive-keyring.gpg
pub  4096R/B98321F9 2010-08-07 Squeeze Stable Release Key <debian-release@lists.debian.org>
pub  4096R/473041FA 2010-08-27 Debian Archive Automatic Signing Key (6.0/squeeze) <ftpmaster@debian.org>
pub  4096R/65FFB764 2012-05-08 Wheezy Stable Release Key <debian-release@lists.debian.org>
pub  4096R/46925553 2012-04-27 Debian Archive Automatic Signing Key (7.0/wheezy) <ftpmaster@debian.org>

Now, for the actual installation. You can use any Debian mirror, OVH has their own in the local network. In OVH's case it is critical to specify the architecture, as the rescue image is i386. I didn't notice that and had to painfully switch architectures in place (which was absolutely not possible a couple of years ago).

# debootstrap --arch amd64 stretch /target http://debian.mirrors.ovh.net/debian

Or the more general case, using the httpredir service, which chooses your nearest public mirror:

# debootstrap --arch amd64 stretch /target http://httpredir.debian.org/debian

Post-install adjustments

After a few minutes downloading and installing stuff, you almost have a Debian system ready to go. Since this is not D-I, we still need to tighten a few screws manually. Let's mount some needed file systems, and enter the brand new system with chroot:

# mount -o bind /dev /target/dev
# mount -t proc proc /target/proc
# mount -t sysfs sys /target/sys
# XTERM=xterm-color LANG=C.UTF-8 chroot /target /bin/bash

The most critical parts now are to correctly save the parameters for the encrypted device, and the partitions and logical volumes. You'll need the UUID saved before:

# echo 'sda2_crypt UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx none luks' \
  > /etc/crypttab

Create the file systems table in /etc/fstab. Here I use labels to identify the devices:

# file system   mount point type    options             dump    pass
LABEL=root      /           ext4    errors=remount-ro   0       1
LABEL=tmp       /tmp        ext4    rw,nosuid,nodev     0       2
LABEL=var       /var        ext4    rw                  0       2
LABEL=usr       /usr        ext4    rw,nodev            0       2
LABEL=home      /home       ext4    rw,nosuid,nodev     0       2
# Alternative home in /srv:
#/srv/home      /home       auto    bind                0       0
LABEL=srv       /srv        ext4    rw,nosuid,nodev     0       2
LABEL=boot      /boot       ext4    rw,nosuid,nodev     0       2
LABEL=swap      none        swap    sw                  0       0

You can also just use the device mapper's names (/dev/mapper/<volume_group>-<logical_volume>), be sure not to use the /dev/<volume_group>/<logical_volume> naming, as there are some initrd-tools that choked on them.

# file system           mount point type    options             dump    pass
/dev/mapper/vg0-root    /           ext4    errors=remount-ro   0       1
/dev/mapper/vg0-tmp     /tmp        ext4    rw,nosuid,nodev     0       2
/dev/mapper/vg0-var     /var        ext4    rw                  0       2
/dev/mapper/vg0-usr     /usr        ext4    rw,nodev            0       2
/dev/mapper/vg0-home    /home       ext4    rw,nosuid,nodev     0       2
# Alternative home in /srv:
#/srv/home              /home       auto    bind                0   0
/dev/mapper/vg0-srv     /srv        ext4    rw,nosuid,nodev     0   2
/dev/sda1               /boot       ext4    rw,nosuid,nodev     0   2
/dev/mapper/vg0-swap    none        swap    sw                  0   0

Some tools depend on /etc/mtab, which now is just a symbolic link:

# ln -sf /proc/mounts /etc/mtab

Network configuration

Now configure the network. You most surely can use DHCP, but you might prefer static configuration, that's personal choice. For DHCP, it is very straightforward:

# cat >> /etc/network/interfaces
auto eth0
iface eth0 inet dhcp

For static configuration, first find the current valid addresses and routes as obtained by DHCP:

# ip address
# ip route

And then store them:

# cat >> /etc/network/interfaces
auto eth0
iface eth0 inet static
    address AAA.BBB.CCC.DDD/24
    gateway AAA.BBB.CCC.254
    pre-up /sbin/ip addr flush dev eth0 || true

Note that pre-up command I added; that is to remove the configuration that is to be done by the kernel during boot (more on that later), otherwise ifupdown will complain about existing addresses.

If your provider does IPv6, add it too. For OVH, the IPv6 set-up is a bit weird, so you need to add the routes in post-up. Your default gateway is going to be your /64 prefix, with the last byte replaced by ff, and then followed by :ff:ff:ff:ff. As you can see, that gateway is not in your network segment, so you need to add an explicit route to it. They have some information, but it is completely unreadable.

If your IPv6 address is 2001:41D0:1234:5678::1/64, you will add:

iface eth0 inet6 static
    address 2001:41D0:1234:5678::1/64
    post-up /sbin/ip -6 route add 2001:41D0:1234:56ff:ff:ff:ff:ff dev eth0
    post-up /sbin/ip -6 route add default via 2001:41D0:1234:56ff:ff:ff:ff:ff

You probably don't want the auto-configured IPv6 addresses, so disable them via sysctl:

# cat >> /etc/sysctl.conf
# Disable IPv6 autoconf 
net.ipv6.conf.all.autoconf = 0
net.ipv6.conf.default.autoconf = 0
net.ipv6.conf.eth0.autoconf = 0
net.ipv6.conf.all.accept_ra = 0
net.ipv6.conf.default.accept_ra = 0
net.ipv6.conf.eth0.accept_ra = 0

To have a working DNS resolver, we can use the local server (OVH in this case):

# cat > /etc/resolv.conf 
search $DOMAIN
nameserver 213.186.33.99

The most important part of a new install: choose a host name (and make the system use it).

# echo $HOSTNAME > /etc/hostname
# hostname $HOSTNAME
# echo "127.0.1.1 $HOSTNAME.$DOMAIN $HOSTNAME" >> /etc/hosts

Mirrors and timezones

If we want to speficy the BIOS clock to use UTC:

# echo -e '0.0 0 0.0\n0\nUTC' > /etc/adjtime

Set up your time zone:

# dpkg-reconfigure tzdata

Configure APT with your preferred mirrors. I also prevent APT from installing recommends by default.

# echo deb http://httpredir.debian.org/debian stretch main contrib non-free \
  >> /etc/apt/sources.list
# echo deb http://httpredir.debian.org/debian stretch-updates main contrib non-free \
  >> /etc/apt/sources.list
# echo deb http://security.debian.org/ stretch/updates main contrib non-free \
  >> /etc/apt/sources.list
# echo 'APT::Install-Recommends "False";' > /etc/apt/apt.conf.d/02recommends
# apt-get update

Preparing for remote unlock

Before installing any package, let's make sure that the initial ram disk (initrd) that is going to be created will allow us to connect. There will be no chance of using the root password during boot. Your public key is usually found in $HOME/.ssh/id_rsa.pub.

# mkdir -p /etc/initramfs-tools/root/.ssh/
# echo $(YOUR_PUB_RSA_KEY) > /etc/initramfs-tools/root/.ssh/authorized_keys

Update: Note that for dropbear versions 2016.73-1 and up, the location for this file has changed. You will get errors about the authorized_keys file when you run update-initramfs -u if this is the case. The new command should be as follows:

# echo $(YOUR_PUB_RSA_KEY) > /etc/dropbear-initramfs/authorized_keys

If you change this, or the host key stored at /etc/dropbear/dropbear_*_host_key, the /etc/crypttab, or any other critical piece of information for the booting process, you need to run update-initramfs -u.

Now we can install the missing pieces:

# apt-get install makedev cryptsetup lvm2 ssh dropbear busybox ssh \
  initramfs-tools locales linux-image-amd64 grub-pc kbd console-setup

During the installation you will have to choose where to install grub, I recommend directly on /dev/sda. Also, the magic initrd will be created. We want to double check that it has all the important pieces for a successful boot:

# zcat /boot/initrd.img-* | cpio -t conf/conf.d/cryptroot \
  etc/lvm/lvm.conf etc/dropbear/\* root\*/.ssh/authorized_keys \
  sbin/dropbear | sort
conf/conf.d/cryptroot
etc/dropbear/config
etc/dropbear/dropbear_dss_host_key
etc/dropbear/dropbear_ecdsa_host_key
etc/dropbear/dropbear_rsa_host_key
etc/lvm/lvm.conf
root-l2oUNW/.ssh/authorized_keys
sbin/dropbear

All these files need to be there. Most critically, we need to check that the cryptroot file has the right information to access the root file system:

# zcat /boot/initrd.img-* | cpio -i --to-stdout conf/conf.d/cryptroot
target=sda2_crypt,source=UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx,key=none,rootdev,lvm=vg0-root

If all that was correct, now we need to tell the kernel to configure the network as soon as possible so we can connect to the initrd and unlock the disks. This is done by passing a command-line option though grub. This should match what was done in /etc/network/interfaces: either DHCP or static configuration. For DHCP, this line should be changed in /etc/default/grub:

GRUB_CMDLINE_LINUX="ip=:::::eth0:dhcp"

For static configuration:

GRUB_CMDLINE_LINUX="ip=MY_IP_ADDR::MY_DEFAULT_GW:MY_NETMASK::eth0:none"

It is also a good idea to disable the quiet boot and graphical boot splash, in case we need to use QEMU to fix some booting issue:

GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_TERMINAL=console

If you have a serial console, you can instruct grub and the kernel to use it, use these lines in the grub configuration instead:

GRUB_CMDLINE_LINUX="ip=MY_IP_ADDR::MY_DEFAULT_GW:MY_NETMASK::eth0:none console=ttyS1,9600"
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --unit=1 --speed=9600 --word=8 --parity=no --stop=1"

Note that this assumes the serial console is attached to the second port, and disables completely the normal VGA console. It might also be a good idea to enable getty to login over the serial console, but that is outside this guide's scope.

Now make the changes effective:

# update-grub2

Final touches

Having fsck fix problems automatically can be a life-saver too:

# echo FSCKFIX=yes >> /etc/default/rcS

Get some very useful packages:

# apt-get install vim less ntpdate sudo

Create an user for yourself, possibly make it an administrator:

# adduser tina
# adduser tina sudo
# adduser tina adm

This is mostly done, exit the chroot, and unmount everything.

# exit  # the chroot.
# umount /target/{dev,proc,sys,boot,home,srv,tmp,usr,var}
# umount /target
# swapoff -a
# lvchange -an /dev/mapper/vg0-*
# cryptsetup luksClose sda2_crypt

Disable the netboot option from your administration panel, reboot, and hope it all goes well.

First boot

If you followed every step carefully, a few minutes later you should be able to ping your server. Use this snippet to enter the password remotely:

$ stty -echo; ssh -o UserKnownHostsFile=$HOME/.ssh/known_hosts.initramfs \
  -o BatchMode=yes root@"$HOST" 'cat > /lib/cryptsetup/passfifo'; \
  stty echo

It is very important that you close the pipe (with control-D twice) without typing enter. For my servers, I have a script that reads the passphrase from a GPG-encrypted file and pipes it directly into the remote server. That way, I only type the GPG passphrase locally:

$ cat unlock.sh 
#!/bin/sh
BASE="$(dirname "$0")"
HOST="$1"
gpg --decrypt "$BASE"/key-"$HOST".gpg | \
    ssh -o UserKnownHostsFile="$BASE"/known_hosts.initramfs -o BatchMode=yes \
        root@"$HOST" 'cat > /lib/cryptsetup/passfifo'

It might be a good idea to create a long, impossible to guess passphrase that you can use in the GPG-encrypted file, and that you can also print and store in somewhere safe. See the luksAddKey function in the cryptsetup(8) man page.

Once again, if everything went right, a few seconds later the openSSH server will replace the tiny dropbear and you will be able to access your server normally (and with the real SSH host key).

Recovery

As it happens, many times something will go wrong, and the server won't come back. In those situations, a serial or KVM console is very useful to see what is going on. But sometimes that is not enough; for example, because you made a mistake with the initrd or grub.

Both OVH and online.net provide "rescue" boot options, which basically netboot your server with some special image for performing troubleshooting. Here are some basic steps to quickly get back to your system from the recovery/rescue image.

If needed, install the required tools:

# apt-get install lvm2 cryptsetup

Unlock the encrypted partition:

# cryptsetup luksOpen /dev/sda2 sda2_crypt

Detect and activate LVM volumes:

# vgchange -aay

Mount the root and special filesystem:

# mkdir /target
# mount /dev/mapper/vg0-root /target
# mount -o bind /dev /target/dev
# mount -t proc proc /target/proc
# mount -t sysfs sys /target/sys

Mount the remaining filesystems using /etc/fstab:

# chroot /target mount -a

If that does not work, you can try the manual approach:

# mount /dev/sda1 /target/boot
# for i in home srv tmp usr var; do
    mount /dev/mapper/vg0-$i /target/$i; done
# swapon /dev/mapper/vg0-swap

Now you are ready enter the chroot, and work on your system:

# XTERM=xterm-color LANG=C.UTF-8 chroot /target /bin/bash

After you are finished, don't forget to cleanly unmount everything, as explained earlier.

Hope you find this article helpful! I would love to hear your feedback.

Updated 2017/02/06: I fixed a few more errors, as I was using this guide to install another server. Changed APT configuration to use the httpredir service, and default to the upcoming stretch release. Fix location for the authorized_keys file. Added recovery section.