When I got an email yesterday from the cron daemon on one of my servers telling me that logrotate segfaulted because it couldn't create a file, I thought perhaps some process had gotten out of control writing to a logfile and filled up the partition where /var/log lives. Imagine my surprise when I went to the /var/log directory, attempted to do an "ls -l", and then waited--and waited--and waited. Odd. Eventually the listing kicked off, and started listing thousands of files with names like "mgetty.log.ttyS0.1.1.1.2.4.1.2.1.1". I finally killed the listing after twenty minutes or so; it was currently listing files with names like "mgetty.log.ttyS0.2.2.1.1.1...". There must be hundreds of thousands of them. Oh goody.
And why is mgetty running anyway? This is a hosted server in a large data warehouse. Are they seriously connecting modems to these things?
Well, googling for "mgetty.log.tty files" quickly identified the culprit as a bad entry in /etc/logrotate.d/mgetty that gets shipped with standard RedHat distributions. The entry causes logrotate to rotate all files that match "mgetty.log.ttyS*". Oh that's just brilliant. This will cause logrotate to rotate not only the real mgetty.log.ttyS0 file, but also every rotated logfile, and then every rotated rotated logfile, and then every--well, you get the idea.
So . . . yikes! Some calculations based on how often this log file gets rotated and the size of the /var/log directory itself--just over 30 meg--leads me to believe that there are probably about half a million of the mgetty.log files in my /var/log directory!
I'm running "find /var/log -name 'mgetty.log.ttyS0.*' -exec rm -v {} \;" to get rid of the all the offending files. It's been running now for well over an hour.
The final challenge will be how to recover the space from the /var/log directory. Unless I'm mistaken, directory files in an ext3 file system will grow as needed but will not shrink on their own. So when these log files are all finally deleted--it will be interesting to see how long that actually takes--I'll be left with 30 meg of allocated but unused space on the drive. Any ideas? Shut down the log processes, create a new directory and move the subtree? Will that work without unforeseen side effects? That's the only thing I can think of. I've never faced this particular issue before, and I'd love to hear from anyone reading this who might have some ideas.
I have come to believe that a key quality of people who make good parents is the ability to remember what it is like to be a child.