Continuing on Friday’s theme of “Ben writes about some Linux basics”, I wanted to share a story of a bug I fixed recently. Our internal documentation server at work had been a little flaky. File copies from the build server would sometimes fail and the web server was being really slow. When I logged in, I noticed the root volume was full.
A full disk is a thing that happens sometimes, especially on small volumes, so I went off in search of the culprit. It turns out that the dead.letter file in root’s home directory was large (several gigabytes, if I recall). For a couple of years, the cron job that runs every 5 minutes to update the documentation page had been trying to send email, which failed since the MTA wasn’t configured.
Why was all of this output trying to be sent via email? Because at some point someone set up the cron job with redirection like so:
2>&1 > /dev/null
Let’s take a step back for a moment and explain what that means. There are two main output streams for command line programs: standard output (a.k.a. “STDOUT”) and standard error (a.k.a. “STDERR”). The former is generally regular output, whereas the latter is for the “important stuff,” like error messages. By default when you run a command in the terminal, they both go to your terminal so you can see them. But you might not always want to see them, so you might redirect to a file or to /dev/null.
Back to our woe-beset server. At first glance, you might say “okay, so both STDOUT (1) and STDERR (2) are being sent to /dev/null”. And you would be wrong. STDERR is being sent to wherever STDOUT is being sent, which at the time is still the terminal (or the cron output email), and then STDOUT is being redirected to /dev/null. So what was in place was effectively the same as:
Changing the order of the redirection to:
> /dev/null 2>&1
Kept the dead.letter file from slowly strangling the life out of the disk.