Managing High Disk Usage from Application Logs on Linux Servers

Overview

Linux servers may experience high disk usage on critical partitions, such as the root (/) partition, due to the uncontrolled accumulation of application logs. This can occur when logging mechanisms do not include automated rotation, archival, or deletion strategies. Persistent log growth can degrade system performance and stability, and in some cases, may lead to service outages if the partition becomes full. Proper diagnostics and maintenance practices are essential to prevent and resolve such issues.

Solution

The following steps outline how to identify, manage, and mitigate excessive disk usage caused by application log files:

1. Disk Usage Diagnostics

Use these commands to inspect disk space and identify problematic files or directories:

General Disk Usage

df -h #View overall disk space usage

df -i #Check inode usage for file count issues

Home Directory Analysis

du -sh /home/* #Analyze space per user

find /home -type f \( -name "*.iso" -o -name "*.zip" \) -exec ls -lh {} \; #Find large files

find /home -type f -size +500M -exec ls -lh {} \; #List files over 500MB

Application Directory (/opt)

du -sh /opt/* | sort -hr #Identify largest subdirectories

find /opt -type f -name "*.log" -mtime +7 -exec ls -lh {} \; #Find old log files

find /opt -type f -size +100M -exec ls -lh {} \; #Locate large files

Oracle or Custom Data Directories (/u01)

du -sh /u01/*

find /u01 -type f \( -name "*.trc" -o -name "*.log" \) -mtime +15 -exec ls -lh {} \;

find /u01 -type f -size +500M -exec ls -lh {} \;

Tomcat Logs and Caches

du -sh /opt/tomcat/*

tail -n 100 /opt/tomcat/logs/catalina.out

find /opt/tomcat/logs -type f -name "*.log" -mtime +7 -exec ls -lh {} \;

du -sh /opt/tomcat/work/*

du -sh /opt/tomcat/temp/*

 

2. Log Management Best Practices

Once large or unnecessary logs have been identified:

• Define a Retention Policy: Determine how long logs need to be retained for operational or audit purposes.

• Delete or Archive Older Logs:

rm /path/to/logs/*YYYY-MM*

Or compress and archive logs:

tar -czf archived_logs_Month.tar.gz /path/to/logs/*YYYY-MM*

• Implement Log Rotation: Use tools like logrotate to automate log rotation, compression, and cleanup.

• Optimize Logging Levels: Ensure application logs (e.g., Node.js, Tomcat) are not set to debug or verbose unless necessary.

 

3. Ongoing Monitoring

• Integrate disk usage alerts using tools such as Zabbix, Nagios, or Prometheus.

• Schedule regular audits or cron jobs to inspect and alert on abnormal log growth.

Reference Documentation

• Logrotate Manual

• Node.js Logging Best Practices

• Zabbix Disk Space Monitoring

<supportagent>

Additional References

• Related Ticket - #60030219
• Related Internal KBA - Disk Utilization Diagnostics for /home, /opt, /u01, and Tomcat

</supportagent>