The Command-Line Tools You Think You Know
If you've spent any time in a command-line terminal, you've likely used pipes (|) and redirection (>). They are the fundamental building blocks of powerful, one-line commands. A pipe takes the output of one command and 'pipes' it directly into the input
of another, letting you chain tools together. Redirection lets you send a command's output to a file instead of the screen. A classic example combines both: finding all text files in a directory and saving the list. You might type something like ls -l | grep ".txt" > text_files.log. It works, and it feels efficient. This is where most people stop. They've learned the core function and it serves them well enough. But this simple understanding skips over a crucial architectural detail—a detail that explains why your scripts sometimes print errors to the screen even when you've told them to log everything to a file.
The Two Streams of Output
The hidden detail is this: programs on Unix-like systems (including Linux and macOS) don't just have one output stream; they have two. By convention, every command starts with three standard connections: standard input (stdin), standard output (stdout), and standard error (stderr). Stdin is for reading input, usually from the keyboard or a previous command in a pipe. Stdout is for the program's main, successful output. Stderr is a separate channel exclusively for error messages, warnings, and diagnostic information. By default, both stdout and stderr are displayed on your terminal, so they appear to be a single, unified stream of text. This is why you can see both the results of a command and its error messages on the same screen. The problem is that the standard pipe and redirection operators you know and love only work on stdout by default. When you use > or |, you are only redirecting the main output. The error stream, stderr, is left untouched and continues to print directly to your terminal.
Why Error Messages Escape Your Logs
This separation is the reason for a common frustration. Imagine running a script to find all log files on your system and log the result: find / -name "*.log" > all_logs.txt. You'll get a file, all_logs.txt, filled with paths. But your screen will likely be flooded with "Permission denied" messages. You told the command to send its output to a file, so why is this noise still cluttering your terminal? Because find is printing its successful results to stdout (which you correctly redirected), but it's printing all those error messages to stderr (which you didn't). Your log file is clean, but you've completely lost the context of what went wrong. For robust scripting and logging, this isn't good enough. You need to control both streams.
Taking Control of All Output
To truly master the command line, you need to manage stderr. You can do this using its file descriptor number, which is 2. Stdout is 1, and stdin is 0. With this knowledge, you can perform much more precise redirection. To redirect only the errors to a file, you use 2>: find / -name ".log" 2> errors.txt. This will show the successful results on the screen but save all the "Permission denied" errors in errors.txt. The most powerful technique is to redirect stderr to the same place as stdout. The classic syntax for this is 2>&1. This little piece of code tells the shell to send stderr (2) to the current destination of stdout (1). So, to capture everything—both results and errors—in a single file, you combine the redirects: find / -name ".log" > all_logs.txt 2>&1. The order is important here; you first point stdout to the file, then point stderr to where stdout is now going. This ensures every single line of output ends up in your log file, giving you a complete and accurate record of the command's execution.
