The Basic Magic Trick: Connecting Programs
On the surface, the pipe `|` is wonderfully straightforward. It takes the output of the program on its left and uses it as the input for the program on its right. It’s like a digital assembly line. Want to see all the running processes on your computer
but only the ones related to, say, Chrome? You can pipe the output of `ps aux` (list all processes) into `grep chrome` (filter for lines containing 'chrome'). `ps aux | grep chrome` This feels like magic. You’ve just created a new, temporary tool without writing a single line of code. Similarly, redirection with `>` seems obvious. `ls -l > file_list.txt` runs the command to list files in a detailed format and, instead of printing it to your screen, it dumps that output into a file named `file_list.txt`. Simple, elegant, and immediately useful. This is the 'looks simple' part, and it's what gets most people hooked. It’s a fast, efficient way to work with text and data, forming the foundation of automation for developers, system administrators, and data scientists.
The Hidden Plumbing: Three Standard Streams
Here's where the simplicity starts to fracture. Every command-line program doesn't just have one 'output'; it has three standard 'streams' it can communicate through. Think of them as three separate pipes coming out of a machine:
1. Standard Input (stdin): Where the program receives data. By default, this is your keyboard. When you use a pipe, you’re connecting the `stdout` of one program to the `stdin` of another.
2. Standard Output (stdout): Where the program sends its normal, successful results. By default, this is your screen.
3. Standard Error (stderr): Where the program sends error messages, warnings, and diagnostic information. This also goes to your screen by default, but it’s a separate stream from `stdout`.
The fact that errors and normal output are separate is the crucial detail. Have you ever redirected a command’s output to a file, only to still see an error message appear on your screen? That’s because you only redirected `stdout`. The error message, traveling through `stderr`, went right past your redirection and onto the screen as usual. This separation is a design feature, allowing you to save good data while still seeing problems as they happen.
Becoming a Plumber: Directing Every Stream
Once you understand the three streams, the world of redirection opens up. The simple `>` is actually shorthand for `1>`, which means 'redirect standard output.' If you want to redirect the error stream, you use `2>`.
Let’s say you run a script that might succeed or fail. You can capture its successful output in one file and its errors in another:
`my_script.sh > success.log 2> errors.log`
Now you’re talking. What if you want to save the successful output but just ignore any errors? In the Unix world, there's a special place that acts like a black hole: `/dev/null`. You can send unwanted streams there to make them disappear forever:
`my_script.sh 2> /dev/null`
This command runs the script and sends any errors into the void, keeping your screen clean. And what if you want to append to a file instead of overwriting it? Just use two greater-than signs: `>>`. This is essential for creating log files that accumulate information over time without erasing previous entries. These aren’t just tricks; they are fundamental tools for building robust, predictable automations.
The Power Moves: Splitting and Merging
This is where you graduate from apprentice to master. What if you want to see a command's output on the screen *and* save it to a file at the same time? A simple pipe won't do. For that, you need a new tool in the chain: `tee`. The `tee` command is named after a T-split in plumbing; it reads from standard input and writes to both standard output and one or more files.
`ls -l | tee file_list.txt`
Run this, and you'll see the file listing on your screen while a copy is silently saved to `file_list.txt`. It's perfect for monitoring a long-running process while also logging its progress. And what if you want to treat both the standard output and standard error as a single stream to pipe or redirect? There’s syntax for that, too. A common modern version is `&>`.
`my_script.sh &> all_output.log`
This command runs the script and redirects *everything*—both `stdout` and `stderr`—into a single log file. This ability to precisely control, separate, merge, and split the flow of information is what makes the command line an enduring and powerful environment. It’s not just a series of commands; it’s a philosophy of small, simple tools that can be combined in endlessly complex ways.
