ZFS Encryption: Mitigating Physical Attacks with Remote Key Management

This article documents the design and implementation of an external key management solution for ZFS encryption. This approach utilizes a custom PHP service to serve encryption keys on demand, specifically designed to mitigate physical and system-level compromises where local keys would fail. This deep dive explores the security architecture, the self-written PHP proof-of-concept (PoC), and the critical security caveats of building a custom Key Management System (KMS).

This is a deeply technical article about ZFS encryption and key management. All steps should be performed with rigorous backups and full understanding of the risks.

I have long contemplated the security trade-offs of storing encryption keys locally. The primary goal of ZFS encryption is to protect data at rest against physical attacks (e.g., theft of a single drive) or system-level compromises (e.g., unauthorized root access).

1. The Threat Model: Why Local Keys Fail

If encryption keys are stored locally, the protection is often inadequate. The security industry consistently confirms that threats against physical assets and servers are real, not theoretical.

• Single Drive Compromise (Theft at Rest): If a disk is mirrored, an attacker often needs only one drive. If they can boot the drive on another system and reset the root password, the key (if stored locally) is accessible. Data center theft, though rare, remains a concern, particularly in unstaffed or third-party edge locations.
• System-Level Compromise: Any attacker gaining persistent root access can scan the local file system for the key file. According to Verizon’s Data Breach Investigations Report, insider threats (whether malicious or negligent) accounted for a significant percentage of incidents in 2023.
• Key Retrieval as Defense: The ZFS man page itself supports remote key retrieval (keylocation=https://<address>), signaling this as the intended high-security method to separate the encryption key from the encrypted data.

Relevance for Private Users

The perception that only large corporations are targets is dangerous. For private users utilizing a self-hosted server in a data center or colocated facility, the primary risk is rarely a targeted nation-state attack. Instead, the risk is often opportunistic theft (disk theft, system disposal without sanitization) or simple loss of control (e.g., a shared administration environment). Remote key management ensures that even if the physical drive leaves your possession or the host system is compromised, the data remains inaccessible, protecting family photos, tax records, and other sensitive personal data.

2. Architectural Solution: Remote Key Retrieval

My solution is to store keys in an off-site, external key management system (KMS) written in PHP, accessible only via HTTPS. This external VM is geographically isolated from the ZFS servers.

The system enforces a basic Access Control List (ACL) by coupling the key retrieval to the requesting system’s identity:

• Source IP Check (Limits access to known servers).
• System Identity Check (Validates machine-id and zpool guid).

3. The Key Management Proof-of-Concept (PHP)

This simple key system, built in PHP using RedBeanPHP and SQLite, demonstrates the necessary security logic. The source code for this PoC is available on GitHub, potentially in a newer version: https://github.com/chani/SimpleZFSKeySystem

Key Retrieval Logic

The request URL is structured as: script.php/machine-id/pool-id/volume-name.

$acls = [
    '1.3.5.7' => [
        'naib7KuoG8IeHahv2jieg1ieth8oQu4a' => '13416744104133899470',
    ],
    // ... other allowed IPs and Machine IDs ...
];

// 1. Check IP against ACL
$protocol = $_SERVER['HTTP_PROTOCOL'];
$ip = $_SERVER['REMOTE_ADDR'];
if (!isset($acls[$ip])) {
    header($protocol.' 403 Forbidden');
    die();
}

// 2. Validate machine ID and pool ID
list($machineID, $poolID, $name) = preg_split('_/_', substr($_SERVER['REQUEST_URI'], 1));
if (isset($acls[$ip][$machineID]) && in_array($poolID, $acls[$ip][$machineID])) {
    // ... proceed to database logic
} else {
    header($protocol.' 403 Forbidden');
    die();
}

// 3. Database: Retrieve or create key
// R::setup('sqlite:_some-secret-place/keys.db');
$key = R::findOne('key', ' name = ? AND machine = ? AND pool = ? ', [ $name, $machine->guid, $pool->guid ]);
if (is_null($key)) {
    // Key generation uses a cryptographically secure function
    $keyvalue = bin2hex(openssl_random_pseudo_bytes(16));
    // ... store new key ...
}

// 4. Check 'active' status and return the key
if ($key->active == true) {
    die($key->keyvalue); // Returns 32-byte hexadecimal key
} else {
    header($protocol.' 403 Forbidden');
    die();
}

Note on Implementation: This code serves as a Proof-of-Concept (PoC). A production system must include proper encryption of the keys in the SQLite database and robust input validation/sanitization.

4. ZFS Integration and Verification

The key is passed directly to the ZFS system via the keylocation property during dataset creation.

Creating the Encrypted Dataset

The ZFS system automatically fetches the key when needed.

# Get pool GUID and machine ID
zpool get guid bpool
cat /etc/machine-id

# Create encrypted dataset using the remote location
zfs create -o mountpoint=none \
           -o encryption=aes-256-gcm \
           -o keyformat=raw \
           -o keylocation=https://secret.example.com/MACHINE_ID/POOL_GUID/100 \
           tank/kvm/100

Verification and Key Lifecycle

Verification confirms that the key is successfully fetched and the volume can be mounted after a reboot or key unloading.

# Check key availability after creation
~# zfs get keystatus tank/kvm/100
NAME PROPERTY VALUE SOURCE
tank/kvm/100 keystatus available -

# Unload key (simulates reboot/shutdown)
~# zfs unload-key -a
1 / 1 key(s) successfully unloaded

# Load key again (System retrieves key from remote endpoint)
~# zfs load-key -a
1 / 1 key(s) successfully loaded

Critical Reminder: You must stress-test the key retrieval after a full system reboot (zfs load-key -a) and confirm that the key is successfully loaded before moving any critical data onto the encrypted volume.

5. Security Caveats and Recommendations

5.1 TLS Certificate Validation Weakness

The ZFS key retrieval process often relies on basic kernel/userland libraries with limited options for custom certificate validation or pinning. This is a critical risk: An attacker could potentially perform a Man-in-the-Middle (MITM) attack if they can trick the ZFS system into trusting a rogue TLS certificate.

• Mitigation: To harden the process, ensure your key server uses a widely trusted public Certificate Authority (CA). For maximum security, the Certificate Authority that issued the key server’s certificate must be explicitly included in the ZFS host’s trusted certificate store (/etc/ssl/certs).

5.2 IP Spoofing and Local Network Threats

The primary ACL relies on the Source IP (REMOTE_ADDR). If your ZFS server and the key server are located in the same data center or share the same Layer 2 network domain, IP Spoofing is a trivial attack. The attacker could fake the trusted IP to bypass the first ACL layer. The combined checks (IP + Machine ID) offer more protection but are still vulnerable if the machine ID is compromised.

5.3 Professional Key Management Systems

For professional or high-security environments, I recommend moving away from a custom PoC to an established, hardened Key Management System. Consider integrating solutions like HashiCorp Vault or other established Open Source KMS solutions, which provide advanced authentication (e.g., AppRole, token-based) and audited key rotation capabilities far beyond the scope of a simple PHP script.

---

6. Sources / See Also

• ZFS Documentation. Encrypting ZFS File Systems. https://docs.oracle.com/cd/E26502_01/html/E29007/gkkih.html
• zfsprops(7) Manpage (OpenZFS). Details on keylocation, keyformat, and encryption. https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html
• ZFS on Linux / Ubuntu Launchpad. Discussion on keylocation and system dependency. https://answers.launchpad.net/ubuntu/+question/698843
• PHP Documentation. openssl_random_pseudo_bytes(): Generate a pseudo-random string of bytes. https://www.php.net/manual/de/function.openssl-random-pseudo-bytes.php
• Verizon Data Breach Investigations Report (DBIR). Current statistics on insider threats and physical breaches. https://www.verizon.com/business/resources/infographics/2025-dbir-smb-snapshot.pdf

---