Freorit

Why Server Hardening Matters

A freshly installed Linux server is functional but not secure. It listens on many ports, accepts password authentication, logs everything to disk, and has minimal protection against brute-force attacks.

Server hardening is the process of systematically reducing these attack surfaces. It's not a single action — it's a layered approach.

The Problem

Default Linux installations make sense for development, not production. Common security oversights:

• SSH allows password authentication (vulnerable to brute force)
• All ports are open, advertising available services
• No rate limiting on failed login attempts
• Logs grow unbounded, consuming disk space
• No backup strategy means data loss is inevitable, not possible
• System updates are manual, leaving known CVEs unpatched

The Solution

This guide walks through six layers of hardening, each building on the previous:

1. Foundation: Users, permissions, updates
2. SSH: Key-based authentication, protocol restrictions
3. Firewall: Stateful rules, egress filtering
4. Intrusion Prevention: Fail2ban rate limiting
5. Monitoring: Logs, metrics, alerting
6. Backup: Automated recovery, tested restores

Layer 1: System Foundation

Before SSH, firewall, or Fail2ban, the foundation must be solid: minimal user privileges, filesystem permissions, and current software.

Principle of Least Privilege

Create accounts only for what's necessary. Disable unused services.

sudo useradd -m -s /bin/bash -G sudo deploy
sudo passwd deploy
# Grant sudo without password for specific commands (optional)
# echo "deploy ALL=(ALL) NOPASSWD: /usr/bin/systemctl" | sudo tee -a /etc/sudoers.d/deploy
    

Never use root for daily tasks. Always use a regular user with sudo for administrative actions.

Filesystem Permissions

Restrict sensitive files to root-only read:

sudo chmod 600 /etc/ssh/sshd_config
sudo chmod 700 /root
sudo chmod 644 /etc/passwd
sudo chmod 640 /etc/shadow
    

Automatic Security Updates

Enable unattended upgrades to patch CVEs automatically:

sudo apt-get install -y unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades
    

Verify it's running:

sudo systemctl status unattended-upgrades
sudo grep -r "^Unattended-Upgrade::Automatic-Reboot:" /etc/apt/apt.conf.d/
    

Audit Kernel Parameters

Harden kernel behavior to mitigate common exploits:

# Disable IP forwarding (unless router)
net.ipv4.ip_forward = 0

# Enable SYN cookies (protect against SYN flood)
net.ipv4.tcp_syncookies = 1

# Ignore ICMP redirects (prevent MITM)
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.accept_redirects = 0

# Log suspicious packets
net.ipv4.conf.all.log_martians = 1

# Ignore ICMP ping requests (optional, breaks some monitoring)
# net.ipv4.icmp_echo_ignore_all = 1
    

Apply changes:

sudo sysctl -p
    

Layer 2: SSH Hardening

SSH is the primary remote access mechanism. Misconfiguration here is the quickest path to compromise.

SSH Key-Based Authentication

Generate a strong ED25519 key (preferred over RSA for new keys):

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_prod -C "deploy@server"
    

Copy the public key to the server:

ssh-copy-id -i ~/.ssh/id_ed25519_prod.pub deploy@server.example.com
    

Disable Password Authentication

Once keys are in place, disable password login:

Port 22
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

# Stronger algorithms
KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com
MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com

# Limit concurrent sessions
MaxSessions 2
MaxStartups 10:30:100

# Client keepalive
ClientAliveInterval 300
ClientAliveCountMax 2

# Disable X11, tunneling, agent forwarding unless needed
X11Forwarding no
AllowAgentForwarding no
PermitTunnel no

# Restrict to specific users
AllowUsers deploy
    

Validate and reload:

sudo sshd -t
sudo systemctl reload ssh
    

Port Changes (Optional Defense-in-Depth)

Moving SSH off port 22 reduces automated scanner noise (not a security measure, just operational hygiene):

Port 2222
    

Update your local SSH config:

Host server.example.com
Port 2222
User deploy
IdentityFile ~/.ssh/id_ed25519_prod
    

Layer 3: Firewall Configuration

A firewall blocks unsolicited inbound traffic and optionally restricts outbound.

UFW (Uncomplicated Firewall)

UFW is a frontend for iptables that's easier to manage:

sudo apt-get install -y ufw

# Set default policies: deny inbound, allow outbound
sudo ufw default deny incoming
sudo ufw default allow outgoing

# Allow SSH (essential to not lock yourself out)
sudo ufw allow 22/tcp
# Or if using a custom port:
# sudo ufw allow 2222/tcp

# Allow HTTP/HTTPS for web service (if running)
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

# Enable firewall
sudo ufw enable

# Verify
sudo ufw status
    

Rate Limiting

Limit repeated connections from the same IP to prevent brute force:

sudo ufw limit 22/tcp
    

This allows max 6 connections per 30 seconds from a single IP.

Logging

Enable firewall logging to catch dropped packets:

sudo ufw logging on
sudo ufw logging high
    

View logs:

sudo tail -f /var/log/ufw.log
    

Layer 4: Intrusion Prevention with Fail2ban

Even with key-based auth, attackers will probe for weak passwords. Fail2ban automatically blocks IPs after repeated failed login attempts.

Installation & Configuration

sudo apt-get install -y fail2ban

# Start and enable
sudo systemctl enable fail2ban
sudo systemctl start fail2ban
    

Create a local config override (don't edit /etc/fail2ban/jail.conf):

[DEFAULT]
bantime = 3600
findtime = 600
maxretry = 5

# IP whitelist (trusted network, office, VPN)
ignoreip = 127.0.0.1/8 192.168.1.0/24 10.0.0.0/8

[sshd]
enabled = true
port = ssh,2222
logpath = /var/log/auth.log
maxretry = 3
findtime = 600
bantime = 3600
    

Reload Fail2ban:

sudo systemctl reload fail2ban
    

Monitor Active Bans

sudo fail2ban-client status
sudo fail2ban-client status sshd
    

Unban an IP if needed:

sudo fail2ban-client set sshd unbanip 192.0.2.100
    

Layer 5: Monitoring & Logging

Logs are only useful if centralized, rotated, and monitored. A single server filling its disk with logs is a denial-of-service scenario.

Log Rotation

Configure logrotate to compress old logs:

/var/log/syslog
/var/log/auth.log
/var/log/ufw.log
{
daily
rotate 14
compress
delaycompress
notifempty
create 0640 syslog adm
sharedscripts
postrotate
/lib/systemd/systemd-logind-control reload-config > /dev/null 2>&1 || true
endscript
}
    

Journal Limits

Systemd journal can grow large. Limit its size:

SystemMaxUse=500M
SystemMaxFileSize=100M
MaxRetentionSec=30day
    

Reload:

sudo systemctl restart systemd-journald
    

Key Logs to Monitor

• /var/log/auth.log — SSH logins, sudo usage, Fail2ban blocks
• /var/log/syslog — System events, service restarts
• /var/log/ufw.log — Firewall dropped packets
• journalctl -u SERVICE — Specific service logs

Layer 6: Backup & Recovery

No security hardening helps if you can't recover from compromise or data loss. Backups are not optional.

Backup Strategy: 3-2-1 Rule

• 3 copies: Original + 2 backups
• 2 media types: Local disk + cloud
• 1 offsite: At least one backup off-site

Local Automated Backup

Use rsync for efficient daily backups to a secondary disk:

#!/bin/bash
BACKUP_DIR="/backup"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

# Backup critical directories
rsync -av --delete \
--exclude='/proc' \
--exclude='/sys' \
--exclude='/dev' \
--exclude='/tmp' \
--exclude='/var/tmp' \
--exclude='/var/log' \
/ "$BACKUP_DIR/backup_$TIMESTAMP/"

# Keep only last 7 daily backups
find $BACKUP_DIR -maxdepth 1 -type d -name "backup_*" -mtime +7 -exec rm -rf {} \;
    

Offsite Backup (Cloud)

For smaller data, use S3 or equivalent:

sudo apt-get install -y awscli

# Configure AWS credentials
aws configure

# Backup to S3 (example)
aws s3 sync /home/deploy/app s3://my-backups/app/ --delete
    

Test Your Restore

A backup you can't restore from is useless. Test regularly:

# Restore a single file
rsync -av /backup/backup_20250610_120000/etc/passwd /tmp/restored_passwd

# Verify
cat /tmp/restored_passwd
    

Best Practices

1. Defense in Depth

No single layer is sufficient. SSH keys alone don't stop a compromised user account. Firewalls alone don't stop application exploits. Layer protections.

2. Minimal Exposure

Only expose what's necessary. If a service isn't needed, uninstall it.

sudo systemctl disable apache2
sudo apt-get remove apache2
    

3. Change Detection

Detect unauthorized modifications with AIDE (Advanced Intrusion Detection Environment):

sudo apt-get install -y aide aide-common
sudo aideinit
sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

# Check for changes
sudo aide --check
    

4. User Accountability

Audit sudo usage and SSH logins. Know who logged in and when:

sudo grep "sudo:" /var/log/auth.log | tail -5
sudo lastlog
    

5. Keep Secrets Separate

Never commit passwords, API keys, or certificates to version control. Use environment variables or secret management tools.

export DB_PASSWORD="$(cat /root/.db_password)"
# Or use a secret manager
export DB_PASSWORD="$(vault read -field=password secret/db)"
    

Lessons Learned

Technical Insights

• SSH keys are not enough: Key-based auth is stronger than passwords, but doesn't prevent compromised accounts from SSH'ing normally. Layer with rate limiting and firewall rules.
• Logs are security: Logs aren't just for debugging; they're your audit trail. Lose them and you lose accountability.
• Backups are not backups until tested: A backup that fails to restore is worthless. Test quarterly.
• Updates break things occasionally: Automated updates are essential, but test them in staging first on critical systems.

Operational Insights

• Document your changes: Hardening gets complex. Document why each setting was chosen so future you understands.
• Automate everything: Manual hardening steps create inconsistencies. Use Ansible, Puppet, or configuration management.
• Firewall rules drift: Rules added ad-hoc become unmaintainable. Keep them in version control.
• Monitoring without alerting is theater: Pretty dashboards are useless if no one reads them. Set up alerts that wake you at 3am when necessary.

Open-Source Projects

This guide relies on industry-standard open-source tools:

• Linux — Free and open-source kernel and OS
• OpenSSH — Secure remote shell and file transfer
• Unattended Upgrades — Automatic security updates
• UFW (Uncomplicated Firewall) — Easy-to-use iptables frontend
• Fail2ban — Dynamic firewall-based intrusion prevention
• logrotate — Log file rotation and compression
• AIDE — Advanced Intrusion Detection Environment for file integrity
• rsync — Fast incremental backup utility