Encryption of ZFS volumes using a remote / external key-system written in PHP

I thought a long time about what security benefits I have if I store the encryption key of a volume on the same system (locally). Let me share some of these thoughts with you. Then I’ll show you my approach using a self-written key-system in PHP (using RedBeanPHP and Sqlite) and finally I’ll show you how to use this with ZFS.

A few words of precaution. I do all this stuff because I’m interested into it. You might loose all your data if you encrypt your files and loose the keys used for decryption. So whatever you do, work with backups. My approach might have flaws which you and me aren’t aware of.

.oO( The Idea! )

(1) If I store the keys for a volume locally then anyone who starts my server and resets the root password may access my data. Furthermore because I am using a mirror-configuration for my discs it would be enough to just get 1 of my discs – nobody stops someone from booting this disc, resetting the root password and accessing my data. Yes, for this purpose it is not even necessary to boot from this disc.

(2) Adding a USB stick to my servers which will contain this key(s) does not make me feel much better. Such a USB stick is copied in seconds… Plus if the USB stick dies I have to bring or deliver a new USB stick (or ask them (the datacenter) to sell me a new USB stick and attach it to the server(s)) and I still need access to the system somehow to put the keys back onto it. This does not sound very comfortable. Does it?

(3) Having/Using chips which are only working and returning keys if they’re used in the same server (hardware) would remove the possibility to just take 1 disk. It would need to be the same exact server. And I assume those chips would be much more robust than a USB stick. However, I do not have something like this in my servers, yet.

In any case, you also have to think about making backups of those keys. If the USB stick dies: you’re lost. If the security chip dies: you’re lost. The latter might be even more funny. Because you may just place the usb stick into a new server. But you might have a hard time trying to get the security chip to work in another server. However, I’m not 100% sure about that. I haven’t had the opportunity to work with those, yet. 

 keylocation=prompt|file://</absolute/file/path>|https://<address> |http://<address>
   Controls where the user's encryption key will be loaded from by default for commands
   such as zfs load-key and zfs mount -l. This property is only set for encrypted datasets
   which are encryption roots.  If unspecified, the default is prompt.
Manual page zfsprops(7)

The manpage states that we can load a key from a remote (https/http) location. This allows me to decrypt my volumes using a key which I store in another location. If someone gets a disk from my system he/she needs to boot it, lookup the keylocation and then try to retrieve a key from it. Similarly if someone resets my root password and accesses my server. Now for the first case, I can limit the IP addresses. For the latter case I can deactivate access to the keys.

*cough* You can detect if someone plugs in a keyboard / monitor or uses IPMI to your server and because you need network for retrieving the key(s), you could send a request to deactivate the keys as soon as network connectivity is available *cough*.

I’m not sure if this effort is required just for protecting some family photos. I’d put more effort into a good backup strategy to be honest.

So the idea is, that I store the keys in a (sqlite) database in a virtual machine in the cloud and I get a backup of those keys regurlarly. This virtual machine is obviously not in the same area my servers are.

.oO( The implementation )

First of all I need some sort of ACL to limit access to my key-system. Because I only want that systems are able to retrieve keys from my key-system if I know those systems. You should be aware of a few things. It might be possible to fake IP addresses in your network. If this is the case an attacker might get the keys. Just because you / I put this key system to a not known subdomain it does not mean nobody will find it. Maybe you should not name the file index.php or keys.php. And just because I am sending the machine-id and pool-guid it does not mean nobody can fake this.

A Request could look like this: script.php/machine-id/pool-id/volume-name.

<?php
/**
 * Simple Key System
 *   taking care of ZFS keys used in encryption
 *
 * @see https://jeanbruenn.info
 * @see https://github.com/chani/SimpleZFSKeySystem
 * @author Jean Bruenn <himself@jeanbruenn.info> 
 */

$acls = [
        '1.3.5.7' => [
                'naib7KuoG8IeHahv2jieg1ieth8oQu4a' => '13416744104133899470',
        ],
        '2.4.6.8' => [
                'yai7IeJ2auGei0aex3ueG1Baetha5eLi' => '12453439812418493689'
        ]
];

Now check where the request comes from and send a 403 forbidden if the IP is not in the ACLs:

$protocol = $_SERVER['HTTP_PROTOCOL'];
$ip = $_SERVER['REMOTE_ADDR'];
if(!isset($acls[$ip])){
    header($protocol.' 403 Forbidden');
    die();
} else {

}

Additionally verify that the machine ID and pool ID match…

    $request = substr($_SERVER['REQUEST_URI'], 1);
    list($machineID, $poolID, $name) = preg_split('_/_', $request);
    if(isset($acls[$ip][$machineID]) && in_array($poolID, $acls[$ip][$machineID])){

    } else {
        header($protocol.' 403 Forbidden');
        die();
    }

I’m using RedBean because it simplifies lots of things. However, it introduces one more attack vector – maybe You want to use plain sqlite instead. However: I downloaded the sqlite-version and added the following to the top of my PHP script:

include('rb-sqlite.php');
R::setup('sqlite:_some-secret-place/keys.db');

Let’s store the machine and pool:

        $machine = R::findOne('machine', ' guid = ? ', [$machineID]);
        if(is_null($machine)){
            $machine = R::dispense('machine');
            $machine->guid = $machineID;
            $machine->address = $ip;
            R::store($machine);
        }

        $pool = R::findOne('pool', ' guid = ? AND machine = ? ', [$poolID, $machine->guid]);
        if(is_null($pool)){
            $pool = R::dispense('pool');
            $pool->guid = $poolID;
            $pool->machine = $machine->guid;
            R::store($pool);
        }

Now for the key: If the key does NOT exist, I create one. If it does exist, I return it if the key is active. The active-flag is something I can set manually. 

        $key = R::findOne('key', ' name = ? AND machine = ? AND pool = ? ', [ $name, $machine->guid, $pool->guid ]);
        if(is_null($key)){
            $keyvalue = bin2hex(openssl_random_pseudo_bytes(16));
            $key = R::dispense('key');
            $key->keyvalue = $keyvalue;
            $key->name = $name;
            $key->active = true;
            $key->machine = $machine->guid;
            $key->pool = $pool->guid;
            R::store($key);
        }

        if($key->active == true){
            die($key->keyvalue);
        } else {
            header($protocol.' 403 Forbidden');
            die();
        }

You can see that the resulting key has a length of 32 bytes. Let’s test so far. First I test with a fake machine-id:

~# curl -I https://secret.example.com/b4e611c248cba0aa9659f2b3de53719d/8021845125172543948/500 
HTTP/1.1 403 Forbidden
Server: nginx/1.24.0
Date: Sat, 11 Nov 2023 13:57:12 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive

Because the machine ID is not known it fails. Let’s see if I get a new key if I make the same request multiple times:

~# curl https://secret.example.com/naib7KuoG8IeHahv2jieg1ieth8oQu4a/13416744104133899470/100  -i
HTTP/1.1 200 OK
Server: nginx/1.24.0
Date: Sat, 11 Nov 2023 14:03:51 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive

1991d9693835c835806fe52a839dd97d

~# curl https://secret.example.com/naib7KuoG8IeHahv2jieg1ieth8oQu4a/13416744104133899470/100  -i
HTTP/1.1 200 OK
Server: nginx/1.24.0
Date: Sat, 11 Nov 2023 14:03:53 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive

1991d9693835c835806fe52a839dd97d

It works. The sqlite database also shows this:

sqlite> .tables
key machine pool

sqlite> SELECT * FROM machine;
1|naib7KuoG8IeHahv2jieg1ieth8oQu4a|1.3.5.7

sqlite> SELECT * FROM pool;
1|13416744104133899470|naib7KuoG8IeHahv2jieg1ieth8oQu4a

sqlite> SELECT * FROM key;
1|1991d9693835c835806fe52a839dd97d|100|1|naib7KuoG8IeHahv2jieg1ieth8oQu4a|13416744104133899470

Understand this code as Proof-of-Concept. Because you want to store the keys encrypted in the sqlite database and you want to escape / validate / sanitize input in this PHP script.

Take a look at: https://github.com/chani/SimpleZFSKeySystem. To get this code. Maybe I’ll find time and joy to implement the encryption part. But really, this is very very simple code. You can do yourself in your favorite programming / scripting language.

.oO( Using it )

Get the GUID from your Pool let’s say bpool:

~# zpool get guid bpool
NAME PROPERTY VALUE SOURCE
bpool guid 12878311658881279327 -

get the machine id from your system:

~# cat /etc/machine-id
8a3611c2a8cb40449659f283d353719d

Get the name of the volume you want to use this for – in my case it’s 100. So the request will look like:

8a3611c2a8cb40449659f283d353719d/12878311658881279327/100

Now create the encrypted dataset:

~# zfs create -o mountpoint=none -o encryption=aes-256-gcm -o keyformat=raw -o keylocation=https://secret.example.com/8a3611c2a8cb40449659f283d353719d/12878311658881279327/100 tank/kvm/100

Now check if the key is there:

~# sqlite3 .data/keys.db
SQLite version 3.37.2 2022-01-06 13:25:41
Enter ".help" for usage hints.
sqlite> SELECT * FROM key;
1|e8c69e831c229e234013900101d42619|100|1|8a3611c2a8cb40449659f283d353719d|12878311658881279327

check if ZFS shows the key as available:

tank/kvm/100 encryption aes-256-gcm -
tank/kvm/100 keylocation https://secret.example.com/8a3611c2a8cb40449659f283d353719d/12878311658881279327/100 local
tank/kvm/100 keyformat raw -
tank/kvm/100 pbkdf2iters 0 default
tank/kvm/100 encryptionroot tank/kvm/100 -
tank/kvm/100 keystatus available -

Unload the key (careful, I unload all)

~# zfs unload-key -a
1 / 1 key(s) successfully unloaded

Check that the key is unavailable now:

~# zfs get keystatus tank/kvm/100
NAME PROPERTY VALUE SOURCE
tank/kvm/100 keystatus unavailable -

check if we can load it again

~# zfs load-key -a
1 / 1 key(s) successfully loaded

~# zfs get keystatus tank/kvm/100
NAME PROPERTY VALUE SOURCE
tank/kvm/100 keystatus available -

Reboot your system – I cannot stress this enough. Check, Check, Check that you can access the key again. That you can get the volume decrypted… If anything goes wrong now you may loose all your data. You got no backup? Then don’t move data to that encrypted volume. :^) 

~# uptime
18:51:16 up 7 min, 1 user, load average: 0.00, 0.02, 0.00

~# zfs get keystatus tank/kvm/100
NAME PROPERTY VALUE SOURCE
tank/kvm/100 keystatus unavailable -

~# zfs load-key -a
1 / 1 key(s) successfully loaded

~# zfs get keystatus tank/kvm/100
NAME PROPERTY VALUE SOURCE
tank/kvm/100 keystatus available -

Oo. Wow. It works.

I am not using this to encrypt my whole root pool (Yes, I use Root on ZFS). However, all my data / files are in virtual machines. So encrypting my volumes might be enough – and when I initially installed my servers I’ve choosen without encryption.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.