Nginx Dynamic Modules: Automating Recompilation with APT Hooks

If you’ve ever dealt with Nginx and its dynamic modules, you know the drill. An Nginx package update hits, and suddenly your custom modules – like ModSecurity or GeoIP2 – are no longer compatible. The whole process is a headache: you have to stop Nginx, recompile your modules against the new version, copy the files, and restart the service.

I was looking for a way to automate this. The goal was simple: ensure that dynamic modules are always compatible with a new Nginx version. And if the recompilation fails for any reason, the entire Nginx update must be aborted (a critical fail-safe).

The Challenge: Choosing the Right Hook (And Why it’s Hard)

My initial thought was to use a simple dpkg hook (pre-install.d/). It seemed logical: the hook runs right before the package is unpacked. However, for some reason, it did not work at all; I was unable to find a reason for this but the script was not run at all.

The solution was to switch to an APT Hook, specifically DPkg::Pre-Invoke. This hook triggers reliably before the dpkg transaction even starts. If the hook’s script exits with a non-zero code, the entire apt command fails. This is exactly the kind of fail-safe mechanism I needed to guarantee the Nginx update would be blocked if anything went wrong.

# /etc/apt/apt.conf.d/10nginx-pre-invoke
DPkg::Pre-Invoke { "/usr/local/bin/nginx-pre-invoke-hook.sh"; };

The Once And Only Once Principle (OAOO)

As I was building the script, I knew I wanted to avoid hardcoding and make it easy to manage new modules in the future. I’m a big fan of the “Once And Only Once” (OAOO) principle.

My solution: I used a Bash array called MODULE_REPOS. Now, to add a new module, I just add its Git URL to the array. The script handles the rest automatically—cloning, pulling, compiling, and copying. No more repetitive code blocks for each module.

MODULE_REPOS=(
    "https://github.com/owasp-modsecurity/ModSecurity-nginx.git"
    "https://github.com/leev/ngx_http_geoip2_module.git"
)

I also built some helper functions, like run_command_quietly and manage_repo, to streamline the code. These functions handle common tasks like error checking, debugging, and Git operations, which makes the main script clean and easy to read. (These helper functions are included in the main script below.)

Critical Lessons Learned: Stability and State

Building a production-ready script isn’t just about writing code; it’s about hardening it against failure and the unpredictable nature of apt environments. My journey revealed several vital lessons:

  • The State Problem: The DPkg::Pre-Invoke hook is executed so early that I hypothesize it struggles to reliably determine the true intent of explicit actions (like reinstall or downgrade), often leading to failures to trigger when needed. The final logic must rely on a combination of version status and a targeted check for explicit actions.
  • The Segfault Fix (Problem 2): After a successful module recompilation, the Nginx daemon must be restarted (not just reloaded) to load the new .so binary. Failing to do so—even if the module is correctly copied—will lead to immediate segmentation faults because the running Nginx process is still linked to the old binary’s API signature.
    • Solution: I added a second hook, DPkg::Post-Invoke, which runs after the package installation is complete, to enforce a service restart. I use set +e in this restart script to ensure the apt transaction itself isn’t broken if the service fails to start for any reason.
  • The Nginx -t Test: My first version included an nginx -t test. The pre-invoke environment only sees the old Nginx binary, not the new one that’s about to be installed. The version mismatch would cause the test to fail, incorrectly blocking a perfectly valid update. I had to remove it.
  • Redirecting Status Messages: When using command substitution, any output to STDOUT gets captured. My manage_repo function was echoing status messages, which was corrupting the module path variable. The fix? Redirecting all status messages to STDERR (>&2).
  • Safe Cleanup: You can’t be too careful with rm -rf. I hardened the cleanup command by prefixing it with /tmp/. I also made sure the script changes out of the temporary directory before attempting to delete it.

Putting It All Together: The Code and Configuration

The final solution requires two hooks to manage the process: one for compilation (fail-safe) and one for service restart (stability).

1. The Compilation Script (/usr/local/bin/nginx-pre-invoke-hook.sh)

This script uses robust logic to determine whether a compilation is needed: It checks if the current version differs from the candidate version OR if an explicit install/reinstall is requested via simulation.

#!/bin/bash
# Nginx Dynamic Module Pre-Invoke Hook
# Runs BEFORE any package operation begins.
# Aborts the entire 'apt' operation if module compilation fails.

set -e

# --- CONFIGURATION ---

DEBUG="false"
BASE_MODULE_DIR="/usr/src/dynamic-nginx-modules"
NGINX_MODULE_PATH="/etc/nginx/modules/"
NGINX_PACKAGE_NAME="nginx"

# Array of module repository URLs.
MODULE_REPOS=(
    "https://github.com/owasp-modsecurity/ModSecurity-nginx.git"
    "https://github.com/leev/ngx_http_geoip2_module.git"
)

# --- HELPER FUNCTIONS ---

# Runs a command and suppresses output if DEBUG=false.
run_command_quietly() {
    local cmd=("$@")
    if [ "$DEBUG" = "false" ]; then
        if ! "${cmd[@]}" &>/dev/null; then
            return 1
        fi
    else
        if ! "${cmd[@]}"; then
            return 1
        fi
    fi
}

# Clones or pulls a Git repository.
manage_repo() {
    local repo_url="$1"
    local local_repo_name="$2"
    local repo_path="$BASE_MODULE_DIR/$local_repo_name"

    if [ ! -d "$repo_path" ]; then
        echo "$local_repo_name repository not found. Cloning into $repo_path..." >&2
        if ! run_command_quietly git clone "$repo_url" "$repo_path"; then
            echo "ERROR: Git clone failed for $local_repo_name. Aborting update." >&2
            exit 1
        fi
    else
        echo "Updating $local_repo_name repository..." >&2
        if ! (cd "$repo_path" && run_command_quietly git pull); then
            echo "ERROR: Git pull failed for $local_repo_name. Aborting update." >&2
            exit 1
        fi
    fi
    echo "$repo_path" # ONLY the path is sent to STDOUT for capture
}

# --- MAIN EXECUTION START ---

echo "--- Starting Nginx Pre-Invoke Check (Package: $NGINX_PACKAGE_NAME) ---" >&2

# 1. CHECK FOR VERSION DIFFERENCE / EXPLICIT ACTION

CURRENT_VERSION=$(dpkg-query -W -f='${Version}' "$NGINX_PACKAGE_NAME" 2>/dev/null || echo "not-installed")
POLICY_OUTPUT=$(apt-cache policy "$NGINX_PACKAGE_NAME")
CANDIDATE_VERSION=$(echo "$POLICY_OUTPUT" | awk '/Candidate/{print $2; exit}')

# Check 1: Nginx not installed or no version available (Exit)
if [ "$CURRENT_VERSION" = "not-installed" ] || [ -z "$CANDIDATE_VERSION" ]; then
    echo "Nginx is not installed or no candidate version is available. Exiting hook." >&2
    exit 0
fi

ACTION_REQUIRED="false"

# Trigger 1: Version difference detected (Upgrade or Downgrade)
if [ "$CURRENT_VERSION" != "$CANDIDATE_VERSION" ]; then
    echo "Version difference detected ($CURRENT_VERSION -> $CANDIDATE_VERSION). Starting recompilation." >&2
    ACTION_REQUIRED="true"

# Trigger 2: Explicit action requested (Reinstall/Downgrade/Install)
# This is the crucial fallback for manual commands like 'apt install pkg=version' or '--reinstall'.
elif apt-get --simulate install "$NGINX_PACKAGE_NAME" 2>/dev/null | grep -E -q " (install|reinstall|downgrade) $NGINX_PACKAGE_NAME" ; then
    echo "Explicit install/reinstall/downgrade detected (Version $CURRENT_VERSION). Starting recompilation." >&2
    ACTION_REQUIRED="true"
fi

# Check 3: No change required (Exit)
if [ "$ACTION_REQUIRED" = "false" ]; then
    echo "Nginx version is stable and no change was explicitly requested. Exiting hook." >&2
    exit 0
fi


# --- EXECUTION ---

echo "Nginx package change detected. Starting dynamic module recompilation." >&2

# Determine Core Version from Candidate Version (for source download)
NGINX_CORE_VERSION=$(echo "$CANDIDATE_VERSION" | sed -E 's/([0-9]+\.[0-9]+\.[0-9]+).*/\1/')

if [ "$DEBUG" = "true" ]; then
    echo "Target Nginx Version (Core): $NGINX_CORE_VERSION" >&2
fi

# 2. PREPARING NGINX SOURCE
echo "Preparing Nginx source code (Downloading/Unpacking)..." >&2

TEMP_FOLDER_NAME="nginx_src_$NGINX_CORE_VERSION"
TEMP_PATH="$TEMP_FOLDER_NAME"

mkdir -p "/tmp/$TEMP_PATH"
cd "/tmp/$TEMP_PATH"

NGINX_TARBALL="nginx-$NGINX_CORE_VERSION.tar.gz"
NGINX_DOWNLOAD_URL="https://nginx.org/download/$NGINX_TARBALL"

if [ ! -f "$NGINX_TARBALL" ]; then
    if ! run_command_quietly wget "$NGINX_DOWNLOAD_URL"; then
        echo "ERROR: Could not download Nginx source code $NGINX_CORE_VERSION from nginx.org. Aborting update." >&2
        exit 1
    fi
fi

tar -xzf "$NGINX_TARBALL"
cd "nginx-$NGINX_CORE_VERSION"

# Clean up any leftover *.so files from previous runs.
rm -f objs/*.so

# 3. MANAGE REPOS AND BUILD ARGS
echo "Managing dynamic module repositories..." >&2

DYNAMIC_MODULES_ARGS=()
mkdir -p "$BASE_MODULE_DIR"

# Loop through the repositories, manage the source, and build the configure arguments
for i in "${!MODULE_REPOS[@]}"; do
    REPO_URL="${MODULE_REPOS[$i]}"
    LOCAL_NAME="module$((i + 1))"
    
    # Clone/Pull the repository and get the path
    MODULE_PATH=$(manage_repo "$REPO_URL" "$LOCAL_NAME")
    
    # Collect arguments for ./configure
    DYNAMIC_MODULES_ARGS+=( "--add-dynamic-module=$MODULE_PATH" )
done

# 4. MODULE COMPILATION
echo "Starting compilation of the dynamic modules..." >&2

if ! run_command_quietly ./configure "${DYNAMIC_MODULES_ARGS[@]}" --with-compat; then
    echo "ERROR: ./configure for modules failed. Aborting update. Check log if DEBUG=true." >&2
    exit 1
fi

if ! run_command_quietly make modules; then
    echo "ERROR: make modules failed. Aborting update. Check log if DEBUG=true." >&2
    exit 1
fi

# 5. UNCONDITIONAL COPY
echo "Copying compiled modules to $NGINX_MODULE_PATH..." >&2

mkdir -p "$NGINX_MODULE_PATH"

# Copy all compiled .so files at once.
if ! cp objs/*.so "$NGINX_MODULE_PATH/"; then
    echo "ERROR: Could not copy one or more modules to $NGINX_MODULE_PATH. Aborting update." >&2
    exit 1
fi

# 6. CLEANUP
echo "SUCCESS: Modules compiled and copied for the new Nginx version. Nginx restart is required." >&2

cd ..
rm -rf "/tmp/$TEMP_PATH"

exit 0

2. The Service Restart Hook (DPkg::Post-Invoke)

The second hook ensures stability after the installation.

  • APT Hook Configuration:

    # /etc/apt/apt.conf.d/20nginx-post-invoke
    DPkg::Post-Invoke { "/usr/local/bin/nginx-post-invoke-hook.sh"; };
  • Restart Script (/usr/local/bin/nginx-post-invoke-hook.sh):

    #!/bin/bash
    # Nginx Dynamic Module Post-Invoke Hook: Ensures restart after successful module compilation/copy

    set +e # IMPORTANT: Allows the script to exit non-zero without aborting APT.

    NGINX_PACKAGE_NAME="nginx"

    # Check if Nginx is installed
    if ! dpkg -s "$NGINX_PACKAGE_NAME" 2>/dev/null | grep -q "Status: install ok installed" ; then
    exit 0
    fi

    echo "NGINX Post-Invoke: Forcing systemctl restart nginx.service to load new modules." >&2

    # Attempt the restart. Logs a warning on failure. systemctl restart nginx.service || {
    echo "WARNING: Nginx restart failed. Please check 'systemctl status nginx.service' for details." >&2
    }

    exit 0

Test and Final Thoughts

If everything works, the script will output SUCCESS: Modules compiled and copied for the new Nginx version. Nginx restart is required. before the normal package installation continues.

While simple, this two-hook approach embodies the core principles of Infrastructure as Code (IaC). It automates a critical and often brittle manual process, which is a significant step toward making my setup more robust in the long run.

References

  • Nginx.org. Nginx Core Module. https://nginx.org/en/docs/http/ngx_http_core_module.html
  • Nginx.org. Building dynamic modules. https://nginx.org/en/docs/configure.html#features
  • Debian Manpages. apt.conf(5) man page. https://manpages.debian.org/testing/apt/apt.conf.5.en.html#DPkg::Pre-Invoke
  • OWASP. ModSecurity-nginx GitHub Repository. https://github.com/owasp-modsecurity/ModSecurity-nginx
  • leev. ngx_http_geoip2_module GitHub Repository. https://github.com/leev/ngx_http_geoip2_module

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.