Category: System Security

OpenSSH Hardening Strategy: Auditing Policies and Mitigating Low-Strength Ciphers

OpenSSH ships with a default configuration that prioritizes high compatibility. However, this compatibility comes at a price: some of the included ciphers and algorithms may be outdated or contain known vulnerabilities. To strengthen the encryption and gain a transparent overview of known weaknesses, ssh-audit is the essential auditing tool.

My hardening strategy uses the Mozilla Security Guidelines on OpenSSH as a base, which I then refined using the specific findings from ssh-audit.

Part I: Initial Server Hardening and Auditing

Before tuning algorithms, I enforce core security policy: limiting access to specific users, disabling root login, and preventing password authentication.

AuthenticationMethods publickey
PermitRootLogin no
# AllowUsers replace-with-your-usernames,separate-by-comma

1. Installation and Usage of ssh-audit

I use pip3 to install and maintain ssh-audit to ensure I have the most current version, which is necessary for accurate vulnerability assessment.

Server Audit (Debian 12 Default) Output: Running ssh-audit localhost on a default installation reveals critical weaknesses. The output serves as our baseline:

# general
(gen) banner: SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u2
(gen) software: OpenSSH 9.2p1
(gen) compatibility: OpenSSH 8.5+, Dropbear SSH 2018.76+
(gen) compression: enabled (zlib@openssh.com)

# key exchange algorithms
(kex) sntrup761x25519-sha512@openssh.com    -- [info] available since OpenSSH 8.5
(kex) curve25519-sha256                     -- [info] available since OpenSSH 7.4, Dropbear SSH 2018.76
                                            `- [info] default key exchange since OpenSSH 6.4
(kex) curve25519-sha256@libssh.org          -- [info] available since OpenSSH 6.4, Dropbear SSH 2013.62
                                            `- [info] default key exchange since OpenSSH 6.4
(kex) ecdh-sha2-nistp256                    -- [fail] using elliptic curves that are suspected as being backdoored by the U.S. National Security Agency
                                            `- [info] available since OpenSSH 5.7, Dropbear SSH 2013.62
(kex) ecdh-sha2-nistp384                    -- [fail] using elliptic curves that are suspected as being backdoored by the U.S. National Security Agency
                                            `- [info] available since OpenSSH 5.7, Dropbear SSH 2013.62
(kex) ecdh-sha2-nistp521                    -- [fail] using elliptic curves that are suspected as being backdoored by the U.S. National Security Agency
                                            `- [info] available since OpenSSH 5.7, Dropbear SSH 2013.62
(kex) diffie-hellman-group-exchange-sha256 (3072-bit) -- [info] available since OpenSSH 4.4
                                                      `- [info] OpenSSH's GEX fallback mechanism was triggered during testing. Very old SSH clients will still be able to create connections using a 2048-bit modulus, though modern clients will use 3072. This can only be disabled by recompiling the code (see https://github.com/openssh/openssh-portable/blob/V_9_4/dh.c#L477).
(kex) diffie-hellman-group16-sha512         -- [info] available since OpenSSH 7.3, Dropbear SSH 2016.73
(kex) diffie-hellman-group18-sha512         -- [info] available since OpenSSH 7.3
(kex) diffie-hellman-group14-sha256         -- [warn] 2048-bit modulus only provides 112-bits of symmetric strength
                                            `- [info] available since OpenSSH 7.3, Dropbear SSH 2016.73
(kex) kex-strict-s-v00@openssh.com          -- [info] pseudo-algorithm that denotes the peer supports a stricter key exchange method as a counter-measure to the Terrapin attack (CVE-2023-48795)

# host-key algorithms
(key) rsa-sha2-512 (2048-bit)               -- [warn] 2048-bit modulus only provides 112-bits of symmetric strength
                                            `- [info] available since OpenSSH 7.2
(key) rsa-sha2-256 (2048-bit)               -- [warn] 2048-bit modulus only provides 112-bits of symmetric strength
                                            `- [info] available since OpenSSH 7.2
(key) ecdsa-sha2-nistp256                   -- [fail] using elliptic curves that are suspected as being backdoored by the U.S. National Security Agency
                                            `- [warn] using weak random number generator could reveal the key
                                            `- [info] available since OpenSSH 5.7, Dropbear SSH 2013.62
(key) ssh-ed25519                           -- [info] available since OpenSSH 6.5

# encryption algorithms (ciphers)
(enc) chacha20-poly1305@openssh.com         -- [info] available since OpenSSH 6.5
                                            `- [info] default cipher since OpenSSH 6.9
(enc) aes128-ctr                            -- [info] available since OpenSSH 3.7, Dropbear SSH 0.52
(enc) aes192-ctr                            -- [info] available since OpenSSH 3.7
(enc) aes256-ctr                            -- [info] available since OpenSSH 3.7, Dropbear SSH 0.52
(enc) aes128-gcm@openssh.com                -- [info] available since OpenSSH 6.2
(enc) aes256-gcm@openssh.com                -- [info] available since OpenSSH 6.2

# message authentication code algorithms
(mac) umac-64-etm@openssh.com               -- [warn] using small 64-bit tag size
                                            `- [info] available since OpenSSH 6.2
(mac) umac-128-etm@openssh.com              -- [info] available since OpenSSH 6.2
(mac) hmac-sha2-256-etm@openssh.com         -- [info] available since OpenSSH 6.2
(mac) hmac-sha2-512-etm@openssh.com         -- [info] available since OpenSSH 6.2
(mac) hmac-sha1-etm@openssh.com             -- [fail] using broken SHA-1 hash algorithm
                                            `- [info] available since OpenSSH 6.2
(mac) umac-64@openssh.com                   -- [warn] using encrypt-and-MAC mode
                                            `- [warn] using small 64-bit tag size
                                            `- [info] available since OpenSSH 4.7
(mac) umac-128@openssh.com                  -- [warn] using encrypt-and-MAC mode
                                            `- [info] available since OpenSSH 6.2
(mac) hmac-sha2-256                         -- [warn] using encrypt-and-MAC mode
                                            `- [info] available since OpenSSH 5.9, Dropbear SSH 2013.56
(mac) hmac-sha2-512                         -- [warn] using encrypt-and-MAC mode
                                            `- [info] available since OpenSSH 5.9, Dropbear SSH 2013.56
(mac) hmac-sha1                             -- [fail] using broken SHA-1 hash algorithm
                                            `- [warn] using encrypt-and-MAC mode
                                            `- [info] available since OpenSSH 2.1.0, Dropbear SSH 0.28

# fingerprints
(fin) ssh-ed25519: SHA256:---
(fin) ssh-rsa: SHA256:---

# algorithm recommendations (for OpenSSH 9.2)
(rec) -ecdh-sha2-nistp256                   -- kex algorithm to remove 
(rec) -ecdh-sha2-nistp384                   -- kex algorithm to remove 
(rec) -ecdh-sha2-nistp521                   -- kex algorithm to remove 
(rec) -ecdsa-sha2-nistp256                  -- key algorithm to remove 
(rec) -hmac-sha1                            -- mac algorithm to remove 
(rec) -hmac-sha1-etm@openssh.com            -- mac algorithm to remove 
(rec) !rsa-sha2-256                         -- key algorithm to change (increase modulus size to 3072 bits or larger) 
(rec) !rsa-sha2-512                         -- key algorithm to change (increase modulus size to 3072 bits or larger) 
(rec) -diffie-hellman-group14-sha256        -- kex algorithm to remove 
(rec) -hmac-sha2-256                        -- mac algorithm to remove 
(rec) -hmac-sha2-512                        -- mac algorithm to remove 
(rec) -umac-128@openssh.com                 -- mac algorithm to remove 
(rec) -umac-64-etm@openssh.com              -- mac algorithm to remove 
(rec) -umac-64@openssh.com                  -- mac algorithm to remove 

# additional info
(nfo) For hardening guides on common OSes, please see: <https://www.ssh-audit.com/hardening_guides.html>
(nfo) Be aware that, while this target properly supports the strict key exchange method (via the kex-strict-?-v00@openssh.com marker) needed to protect against the Terrapin vulnerability (CVE-2023-48795), all peers must also support this feature as well, otherwise the vulnerability will still be present.  The following algorithms would allow an unpatched peer to create vulnerable SSH channels with this target: chacha20-poly1305@openssh.com.  If any CBC ciphers are in this list, you may remove them while leaving the *-etm@openssh.com MACs in place; these MACs are fine while paired with non-CBC cipher types.

Part II: Mitigating Weak Cryptography

1. Hardening Diffie-Hellman (GEX) Moduli

I use awk to filter the /etc/ssh/moduli file to enforce a minimum size of 3072 bits, eliminating low-strength moduli.

awk '$5 >= 3071' /etc/ssh/moduli > /etc/ssh/moduli.tmp && mv /etc/ssh/moduli.tmp /etc/ssh/moduli

2. Replacing Host Keys

I replace the default host keys with stronger settings, which is essential for modern security. The Ed25519 key is the current gold standard for key exchange performance and security.

rm /etc/ssh/ssh_host_*
ssh-keygen -t rsa -b 4096 -f /etc/ssh/ssh_host_rsa_key -N ""
ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key -N ""

3. Finalizing the SSHD Configuration

I explicitly define the allowed algorithms in a dedicated drop-in file (/etc/ssh/sshd_config.d/ciphers.conf) to ensure that only the secure algorithms identified by ssh-audit remain.

# /etc/ssh/sshd_config.d/ciphers.conf
# Ciphers

HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256

KexAlgorithms curve25519-sha256@libssh.org,curve25519-sha256,sntrup761x25519-sha512@openssh.com,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512

Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr

MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com

Part III: Auditing Clients and Avoiding Pitfalls

1. Client-Side Hardening

I recommend applying a strong default set of algorithms to the client’s ~/.ssh/config file to reduce its attack surface, overriding weak defaults when connecting to external hosts.

# ~/.ssh/config

Host *
    HashKnownHosts yes
    HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-ed25519
    KexAlgorithms curve25519-sha256@libssh.org,curve25519-sha256,diffie-hellman-group16-sha512,...
    Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,...
    MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com

2. Defense Tool Context (RKhunter and Lynis)

When using system auditing tools, I recognize that their checks can be incomplete or based on outdated assumptions.

  • RKhunter Context: RKhunter incorrectly read the active drop-in configuration (sshd_config.d), confirming the necessity of manual verification.
  • Lynis Context: Changing port 22 or disabling modern compression offers minimal additional protection on systems already hardened with public-key authentication. Do not apply settings blindly.

3. Reducing Service Footprint (Defense-in-Depth)

Finally, I hide the OpenSSH version for minor defense-in-depth:

Plaintext

# /etc/ssh/sshd_config.d/other.conf
Banner none
DebianBanner no

This reduces the public information available to automated scanners.

Sources / See Also

  1. Mozilla Security Guidelines. OpenSSH Recommended Configuration. https://wiki.mozilla.org/Security/Guidelines/OpenSSH
  2. SSH-Audit. SSH Hardening Guides for Common OSes. https://www.ssh-audit.com/hardening_guides.html
  3. OpenSSH. Release Notes for OpenSSH 7.4 (Removal of pre-auth compression). https://www.openssh.com/txt/release-7.4
  4. OpenSSH. Release Notes for OpenSSH 4.2 (Delayed compression). https://www.openssh.com/txt/release-4.2
  5. GitHub (OpenSSH Portable). Source Code Reference for GEX fallback mechanism. https://github.com/openssh/openssh-portable/blob/V_9_4/dh.c#L477
  6. CISOfy Lynis. Control Reference for SSH Hardening (SSH-7408). https://cisofy.com/controls/SSH-7408/

Suricata Alert Analysis: Tuning Rules and Promoting Detection to Prevention

This is a follow-up to my last post in which I set up Suricata as an IPS. This article demonstrates how to effectively work with the Suricata engine—specifically, how I analyze its log output, silence unnecessary alerts, and promote specific detection rules to prevention rules.

1. Performance and Rule Management Setup

LibTCMalloc Integration

To enhance Suricata’s performance and stability, I integrate Google’s TCMalloc library to achieve memory usage improvements.

  1. Install the library: apt-get install libtcmalloc-minimal4
  2. Edit the Systemd service (systemctl edit suricata) to preload the library:
# /etc/systemd/system/suricata.service.d/override.conf
[Service]
Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4"

Rule Update Path Management

I correct the Debian setup where the default rule path conflicts with the update path. I align the configuration to use the dedicated data directory (/var/lib/suricata/rules) for updates, simplifying maintenance.

  1. Edit /etc/suricata/suricata.yaml to point the default rule path:default-rule-path: /var/lib/suricata/rules
  2. I ensure that update.yaml is configured correctly, and remove all initial rules from /etc/suricata/rules to avoid duplicate-rules warnings.

2. Alert Analysis and Rule Tuning (Observability in Practice)

By default, Suricata operates as an IDS (Intrusion Detection System). The critical first step is analyzing the generated alerts (fast.log) to separate actual threats from alert noise.

Initial Alert Frequency Analysis

The following command provides a crucial initial overview by counting unique alert messages and sorting them by frequency. This step is essential to understand the top sources of load and noise.

# awk '{$1=""; $2=""; $3=""}1' fast.log | sed 's_\[\*\*\].*__g' | sed 's_ group [0-9]*__g' | sort | uniq -c | sort -h

# Log Analysis (Excerpt showing frequency)
[..]
    100    GPL RPC portmap listing UDP 111 
    103    SURICATA STREAM 3way handshake excessive different SYN/ACKs 
    176    ET SCAN Suspicious inbound to PostgreSQL port 5432 
    216    ET SCAN Suspicious inbound to mySQL port 3306 
    223    SURICATA UDPv4 invalid checksum 
    236    SURICATA STREAM Last ACK with wrong seq 
    241    GPL ICMP_INFO PING speedera 
    325    ET SCAN Suspicious inbound to MSSQL port 1433 
    ...
  12872    GPL ICMP_INFO PING *NIX

The Decision to Silence Noise

Alerts like the simple GPL ICMP_INFO PING *NIX often provide no actionable security value and must be disabled to prevent log flooding. I disable logging of ping probes by identifying the specific Signature IDs (SIDs) and adding them to a custom disable.conf file.

Code-Snippet

# /etc/suricata/disable.conf (Excerpt for ICMP PINGs)
# Disabled ping logging
2100366
...
2100480

3. Promotion to IPS: Hardening the Drop Policy

For the system to transition from passive detection to active prevention (IPS), specific detection rules must be promoted to drop rules.

I promote the ET DROP Dshield Block Listed Source rule, as it targets known hostile IPs, by adding its SID to drop.conf.

# /etc/suricata/drop.conf
# Rules matching SIDs in this file will be converted to drop rules.
2402000 # SID for 'ET DROP Dshield Block Listed Source'

After running suricata-update, the engine confirms the change: -- Dropped 1 rules.

Verifying the Drop (Active Defense Check)

I verify the success of the active drop policy by specifically filtering for dropped packets in the logs.

# Command to output only dropped packets, showing the specific rule that triggered the block:
# awk '/Drop/{...}' fast.log | sort | uniq -c | sort -hr
# Example Output:
   6505 IP dropped due to fail2ban detection
    638 ET DROP Dshield Block Listed Source

4. Advanced Rule Tuning: Leveraging Variables and Custom Logic

My advice is to use the variables whenever possible. By ensuring that network variables ($HOME_NET, $SMTP_SERVERS, etc.) correctly reflect your environment, you maximize the accuracy of existing rules. This prevents false positives and improves performance.

Enhancing Accuracy with Custom Rules

It’s crucial not just to disable bad rules, but to write custom rules that leverage these network variables for precise defense.

Example: Traffic Segregation Rule

To save resources, I would write a custom rule that only inspects for a vulnerability (e.g., a specific HTTP exploit) when the traffic comes from the external network and is destined for the correct server type.

# Example: Only check for sensitive SQL traffic if it comes from the EXTERNAL net.
# This prevents wasting resources checking internal-to-internal traffic.
# alert tcp $EXTERNAL_NET any -> $SQL_SERVERS 3306 (msg:"ET Custom: External Access to SQL Port"; ...)

This ensures that network resources are conserved by avoiding redundant checks on internal traffic.

5. Modern Analysis: Migrating from Bash to Structured Data

While the Bash pipeline is functional, high-traffic environments quickly overwhelm it. For modern Observability and SecOps analysis, the logs must be processed as structured data.

Migrating to EVE JSON

Suricata can output events in the EVE JSON format, which is ideal for ingestion into systems like Elasticsearch (ELK) or Splunk. This eliminates the slow and unreliable Bash parsing of fast.log.

Configuration Change (in suricata.yaml):

To migrate from the legacy fast.log format, you simply need to enable the EVE logger in your configuration.

# Output module setup in suricata.yaml
outputs:
  - eve-log:
      enabled: yes
      file: eve.json
      # Other settings (e.g., adding flow/metadata fields)

Python for High-Performance Analysis

Instead of relying on slow awk and sed pipelines, I recommend using Python for high-performance log analysis. Python’s built-in json library is optimized to read and aggregate large eve.json files far more efficiently. This elevates the analysis layer of the architecture to a production standard.

Sources / See Also

  1. Suricata Documentation. High-performance AF_PACKET IPS mode configuration and usage. https://docs.suricata.io/en/latest/install/af-packet.html
  2. Suricata Documentation. Working with Suricata-Update (Ruleset Management). https://suricata-update.readthedocs.io/en/latest/update.html
  3. Suricata Documentation. EVE JSON Output for Structured Logging. https://docs.suricata.io/en/latest/output/eve/eve-json-format.html
  4. Google Development. Google Perftools (TCMalloc) Documentation. https://github.com/google/gperftools
  5. Emerging Threats (Proofpoint). Information on the Emerging Threats Open Ruleset. https://www.proofpoint.com/us/security-awareness/blog/emerging-threats
  6. Elastic Stack (ELK) Documentation for Log Analysis. https://www.elastic.co/what-is/elk-stack
  7. Linux Manpage: ethtool (Network Offload Configuration). https://man7.org/linux/man-pages/man8/ethtool.8.html

ZFS Encryption: Mitigating Physical Attacks with Remote Key Management

This article documents the design and implementation of an external key management solution for ZFS encryption. This approach utilizes a custom PHP service to serve encryption keys on demand, specifically designed to mitigate physical and system-level compromises where local keys would fail. This deep dive explores the security architecture, the self-written PHP proof-of-concept (PoC), and the critical security caveats of building a custom Key Management System (KMS).

Continue reading ZFS Encryption: Mitigating Physical Attacks with Remote Key Management

Redis Instance Isolation: Running Multi-Instance Redis with systemd Templates

Instead of running a single global Redis server, I prefer to use multiple isolated instances. This allows me to precisely limit resources like memory (maxmemory) and apply specific tuning per instance. This approach is fundamental to reliable operation in a shared environment. I used systemd templates to manage this, creating an instance for Amavisd-new as a practical example.

Continue reading Redis Instance Isolation: Running Multi-Instance Redis with systemd Templates