Ubuntu: large syslog and kern.log files

Question

Logging into my Ubuntu machine, I get a warning that I am running out of disk space. Tracing back, I find that it is the syslogs, especially the kern.log(s) that are eating up my 1TB disk.

-rw-r----- 1 syslog adm 240G Feb 25 14:22 kern.log
-rw-r----- 1 syslog adm 516G Feb 21 07:59 kern.log.1
-rw-r----- 1 syslog adm 1.1K Feb 15 07:39 kern.log.2.gz
-rw-r----- 1 syslog adm  19K Feb  7 07:56 kern.log.3.gz
-rw-r----- 1 syslog adm  37K Feb  1 07:45 kern.log.4.gz
-rw-r----- 1 syslog adm  23G Feb 25 14:52 syslog
-rw-r----- 1 syslog adm  25G Feb 25 08:11 syslog.1
-rw-r----- 1 syslog adm 1.6G Feb 24 07:49 syslog.2.gz
-rw-r----- 1 syslog adm 1.7G Feb 23 08:18 syslog.3.gz
-rw-r----- 1 syslog adm 3.4G Feb 22 08:19 syslog.4.gz
-rw-r----- 1 syslog adm 3.6G Feb 21 07:59 syslog.5.gz
-rw-r----- 1 syslog adm 6.9G Feb 20 07:38 syslog.6.gz
-rw-r----- 1 syslog adm 7.3G Feb 19 07:36 syslog.7.gz

From the snippet above, you can easily find that kern.log and kern.log.1 is eating up 80% of my 1TB disk. I can get the space by deleting the files, but I think it won't solve the problem.

Does anyone have an idea on what the issue might be? I saw that you can get the logging level by:

cat /proc/sys/kernel/printk

and I get

4    4    1    7

Answer 1

This is an old question, but neither of the previous two answers are good solutions:

• The accepted answer doesn't explain why the disk problem goes away if you fix the underlying system issue (the answer is logrotate), plus your system may keep writing to the logs and fill up your disk before you can even figure out the underlying issue.
• The other answer removes and disables the logs entirely, which is not a good approach as it ignores the underlying issue. Also, you'll probably want those log files later when you're figuring out other system problems -- disabling syslog makes it more difficult to track down future issues!

Instead, here is a safer method that lets you keep the log files while reclaiming disk space while also stopping the log files from doing this again.

1. Safely clear the logs: after looking at (or backing up) the logs to identify your system's problem, clear them by typing > /var/log/syslog (including the >). You may need to be root user for this, in which case enter sudo su, your password, and then the above command).

• Then restart the syslog service (either systemctl restart syslog or service syslog restart).

2. Then, you can force the logs to rotate and delete automatically if they reach a certain size, using logrotate. In this case you can edit the config with sudo nano /etc/logrotate.d/rsyslog and add one line:

/var/log/syslog
{
    rotate 7
    daily
    maxsize 1G # add this line
    missingok
    notifempty
    delaycompress
    compress
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

• This will force your syslog to "rotate" (i.e., create a new log file and archive the previous log file) after either 1 day or when the file becomes 1GB, whichever comes first. Note that rotate 7 means your system will only keep 7 total syslog backups so it can only ever take up 7GB of space
• Note: you can change maxsize, rotate N, and other settings to customize your logs -- use the command man logrotate to see more.

3. While you're at it, you may want to add the same setting in the second part of the file, which governs the behavior of other log files (e.g. kern.log for kernel events, auth.log for authentication events, etc.). This setting will make it so that each of these other log files will only take 4GB in total.:

...
{
    rotate 4
    weekly
    maxsize 1G
...
}

This will allow your system to keep logging events without them filling your disk.

For more, see the manual and a similar question.

Answer 2

Have you checked the content of those files? There's obviously something going on with your server causing events to be generated. Resolve whatever issue is causing that, and your logs should return to their normal size.

To temporary solve the issue, type

echo "" > /var/log/kern.log
echo "" > /var/log/syslog
service syslog restart
journalctl --vacuum-size=50M

You need to be root user for this: enter sudo su, your password, and then the above commands

Answer 3

• Rotation of log files (EG system logs, kernel logs) is handled by logrotate
• Enter the following command to modify the logrotate configurations:

sudo nano /etc/logrotate.d/rsyslog

• Under the entries for the log files that are reaching problematic sizes (EG syslog, kern.log), if there is no configuration then add the configuration shown below, otherwise modify the existing configuration to look like the configuration shown below
• A configuration consists of one or more lines of directives enclosed in curly braces, type man logrotate and scroll down to the DIRECTIVES section for a description of these directives
• In particular, make sure to include the size 100M line, where 100M can be modified according to the maximum size you want your log files to take up, and make sure that there are no time-based rotation directives, EG daily, weekly, etc

{
        rotate 7
        size 100M
        missingok
        ifempty
        delaycompress
        compress
        postrotate
                /usr/lib/rsyslog/rsyslog-rotate
        endscript
}

• Rotation of log files can be scheduled by cron, and by default happens daily
• You can modify this behaviour to schedule rotation of log files to happen hourly instead of daily using the following command:

sudo mv /etc/cron.daily/logrotate /etc/cron.hourly/logrotate

• It is possible that the cron script for logrotate is disabled in favour of the systemd timer
• You can make sure that the cron script for logrotate is not disabled in favour of the systemd timer as follows:
  • Enter the command sudo nano /etc/cron.hourly/logrotate to view the contents of the cron script for logrotate (or sudo nano /etc/cron.daily/logrotate if you didn't previously move the script)
  • Check to see if the following four lines are present, and if they are, either comment them out by placing a # at the beginning of each line, or delete those lines entirely:

# skip in favour of systemd timer
if [ -d /run/systemd/system ]; then
    exit 0
fi

• You can also manually force rotation of log files using the following command:

sudo logrotate --force --verbose /etc/logrotate.conf

• To simply see what actions would be performed by the above command, without actually rotating or removing any log files, use the following command:

sudo logrotate --force --debug /etc/logrotate.conf

• If you find that the /var/log/journal folder is also getting very big, according to this answer, you can clear it with the following command:

sudo journalctl --vacuum-size=100M

• To make this happen automatically every time logrotate is called by cron, enter the command sudo nano /etc/cron.hourly/logrotate (or sudo nano /etc/cron.daily/logrotate if you didn't previously move the script) and insert the line journalctl --vacuum-size=100M (NB not including sudo)

Answer 4

As ascendants suggests: "Safely clear the logs: after looking at (or backing up) the logs to identify your system's problem"

The real problem was found in the logsys file: millions of lines with the following message "PCIe Bus Error severity Corrected".

The "PCIe Bus Error severity Corrected" error is basically a Linux report indicating that there is some problem, in my case hardware compatibility.

This problem caused several files or folders to grow out of proportion (30 GB or more): /var/log/kern.log, /var/log/syslog, /var/log/journal/, etc.

On this site they offer four solutions that offer a workaround, in my case only the last option worked for me.