Managing log files is a critical task in maintaining server health and preventing disk space issues. Enabling log rotation helps manage the size of log files, but it needs to be configured properly to ensure it meets your requirements. Here’s how to set up and manage log rotation in Linux to prevent disk space issues effectively:

1. Understanding Log Rotation

Log rotation is the process of automatically compressing, archiving, and deleting old log files to prevent them from consuming too much disk space. The logrotate utility is commonly used for this purpose.

2. Setting Up Log Rotation

Let’s assume you want to rotate your log files every 7 days and keep a maximum of 4 weeks of logs.

Step-by-Step Guide

1. Install logrotate (if not already installed)

sudo apt-get install logrotate   # Debian/Ubuntu
sudo yum install logrotate       # CentOS/RHEL

Configure logrotate

Logrotate configurations are usually found in /etc/logrotate.conf for the global settings and in /etc/logrotate.d/ for service-specific settings.

Edit the global configuration file:

sudo nano /etc/logrotate.conf

Ensure the configuration file has sensible defaults. Below is an example of a global configuration:

# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
dateext

# compress log files
compress

# uncomment this if you want your log files to be mailed to you
# mail root@localhost

# specify a different location for rotated log files
# olddir /var/log/old

Create/Modify Service-Specific Logrotate Configurations

Create or modify configurations for specific services. For instance, let’s configure Apache log rotation.

Edit the Apache logrotate configuration file:

sudo nano /etc/logrotate.d/apache2

Example configuration:

/var/log/apache2/*.log {
    weekly
    rotate 4
    compress
    delaycompress
    missingok
    notifempty
    create 640 root adm
    sharedscripts
    postrotate
        if [ -f /var/run/apache2.pid ]; then
            /etc/init.d/apache2 reload > /dev/null
        fi
    endscript
}

1. Explanation of the parameters:
   • weekly: Rotate logs weekly.
   • rotate 4: Keep 4 weeks of log files.
   • compress: Compress old log files.
   • delaycompress: Delay compression of the most recent log file until the next rotation.
   • missingok: Do not report errors for missing log files.
   • notifempty: Do not rotate empty log files.
   • create 640 root adm: Create new log files with specified permissions.
   • sharedscripts: Run postrotate script only once.
   • postrotate: Command to run after log rotation (reload Apache to start logging to a new file).

3. Verify Logrotate Configuration

After setting up logrotate, it’s essential to verify the configuration to ensure it’s working correctly.

1. Check Logrotate Configuration Syntax

sudo logrotate -d /etc/logrotate.conf

The -d option runs logrotate in debug mode, which simulates the rotation without making any changes.

Force Log Rotation for Testing

sudo logrotate -f /etc/logrotate.conf

1. The -f option forces logrotate to rotate logs even if it doesn’t meet the criteria.

4. Monitoring Disk Usage

To ensure that disk space is being managed correctly, regularly monitor disk usage.

1. Check Disk Usage

df -h

Check Log File Sizes

du -sh /var/log/*

5. Automate Log Monitoring and Cleanup

Consider setting up automated monitoring and alerts for disk usage and log file sizes.

1. Set Up a Cron Job for Regular MonitoringCreate a script to monitor disk usage and send alerts if usage exceeds a threshold.Example script (/usr/local/bin/monitor_disk_usage.sh):

#!/bin/bash

THRESHOLD=80
USAGE=$(df -h / | grep -v Filesystem | awk '{print $5}' | sed 's/%//')

if [ $USAGE -gt $THRESHOLD ]; then
    echo "Disk usage has exceeded $THRESHOLD%. Current usage: $USAGE%" | mail -s "Disk Usage Alert" admin@example.com
fi

Make the script executable:

sudo chmod +x /usr/local/bin/monitor_disk_usage.sh

Add the script to cron to run daily:

sudo crontab -e

Add the following line:

0 1 * * * /usr/local/bin/monitor_disk_usage.sh

How to setup using Ansible

Using Ansible to set up log rotation across multiple servers can help you automate the configuration and ensure consistency. Below is a step-by-step guide to set up log rotation using Ansible.

1. Set Up Your Ansible Environment

Ensure Ansible is installed on your control machine. If not, install it using:

sudo apt-get install ansible  # Debian/Ubuntu
sudo yum install ansible      # CentOS/RHEL

2. Create an Inventory File

Create an inventory file (inventory.ini) to list your servers.

[webservers]
server1 ansible_host=192.168.1.10
server2 ansible_host=192.168.1.11
server3 ansible_host=192.168.1.12

3. Create an Ansible Playbook

Create a playbook file (logrotate.yml) to define the steps for configuring log rotation.

---
- name: Configure log rotation on multiple servers
  hosts: webservers
  become: yes

  tasks:
    - name: Install logrotate
      package:
        name: logrotate
        state: present

    - name: Create logrotate configuration for Apache
      copy:
        dest: /etc/logrotate.d/apache2
        content: |
          /var/log/apache2/*.log {
              weekly
              rotate 4
              compress
              delaycompress
              missingok
              notifempty
              create 640 root adm
              sharedscripts
              postrotate
                  if [ -f /var/run/apache2.pid ]; then
                      /etc/init.d/apache2 reload > /dev/null
                  fi
              endscript
          }

    - name: Create logrotate configuration for Nginx
      copy:
        dest: /etc/logrotate.d/nginx
        content: |
          /var/log/nginx/*.log {
              weekly
              rotate 4
              compress
              delaycompress
              missingok
              notifempty
              create 640 root adm
              sharedscripts
              postrotate
                  [ -f /run/nginx.pid ] && kill -USR1 `cat /run/nginx.pid`
              endscript
          }

    - name: Create global logrotate configuration
      copy:
        dest: /etc/logrotate.conf
        content: |
          # rotate log files weekly
          weekly

          # keep 4 weeks worth of backlogs
          rotate 4

          # create new (empty) log files after rotating old ones
          create

          # use date as a suffix of the rotated file
          dateext

          # compress log files
          compress

          # include specific logrotate files
          include /etc/logrotate.d

          # set the mail to the root
          mail root@localhost

          # rotate wtmp file monthly
          /var/log/wtmp {
              monthly
              create 0664 root utmp
              minsize 1M
              rotate 1
          }

          # rotate btmp file monthly
          /var/log/btmp {
              monthly
              create 0660 root utmp
              rotate 1
          }

    - name: Force log rotation for testing
      command: logrotate -f /etc/logrotate.conf

4. Run the Ansible Playbook

Execute the playbook to configure log rotation on all servers listed in your inventory.

ansible-playbook -i inventory.ini logrotate.yml

Explanation of the Playbook

• Install logrotate: Ensures logrotate is installed on all target servers.
• Create logrotate configuration for Apache: Copies the Apache logrotate configuration to the target servers.
• Create logrotate configuration for Nginx: Copies the Nginx logrotate configuration to the target servers.
• Create global logrotate configuration: Copies a global logrotate configuration to the target servers.
• Force log rotation for testing: Forces a log rotation to ensure the configurations are applied correctly.

Customizing the Playbook

You can customize the playbook according to your specific needs by modifying the logrotate configuration for other services or adjusting the rotation frequency and retention period.

Monitoring Log Rotation

After running the playbook, you can verify the log rotation setup:

Check the logrotate status and logs on each server:

sudo less /var/log/syslog | grep logrotate
sudo less /var/log/messages | grep logrotate

Manually trigger log rotation if necessary:

sudo logrotate -f /etc/logrotate.conf

Verify the rotated logs:

ls -l /var/log/apache2/
ls -l /var/log/nginx/