The Clutter of Temporary Files
Let's be honest: creating temporary files feels like a small defeat. It works, but it’s messy. You pipe the output of your first command into `temp1.txt`. You do the same for the second into `temp2.txt`. Then you run your comparison tool, like `diff`,
on those two files. Afterward, you’re honor-bound to clean up after yourself with `rm temp1.txt temp2.txt`, but half the time you forget, leaving digital breadcrumbs all over your file system. It’s a clunky, multi-step process for what feels like a single thought. This workflow is a classic sign that you're missing a more elegant tool. The Unix philosophy is all about small, sharp tools that work together seamlessly. Creating intermediate files breaks that flow, forcing a detour through the filesystem.
The Big Reveal: Process Substitution
The feature that cleans all this up is called process substitution. It’s one of those things that, once you see it, you’ll wonder how you ever lived without it. The syntax looks a little strange at first: `<(command)`. What it does is pure magic: Bash runs the `command` inside the parentheses and presents its output as if it were a temporary file. It doesn't actually write a file to your directory; instead, it uses a special file descriptor that the next command can read from. This allows you to feed the output of a process directly to another command that normally expects a filename as an argument. It’s like a pipe, but for commands that don’t know how to read from standard input.
Putting It to Work: The Classic Example
The textbook example, and the one that makes the benefit immediately obvious, is comparing the contents of two different directories with `diff`. Normally, you might do this:
`ls -1 dir1 > temp1.txt`
`ls -1 dir2 > temp2.txt`
`diff temp1.txt temp2.txt`
`rm temp1.txt temp2.txt`
With process substitution, this entire four-step dance becomes a single, elegant line:
`diff <(ls -1 dir1) <(ls -1 dir2)`
That’s it. Bash runs `ls -1 dir1` and `ls -1 dir2` in parallel, and `diff` sees the output of each as a file it can read from. No mess, no cleanup. The command expresses the intent—comparing two lists—without the tedious mechanics of temporary file management. This works for any command that expects file paths as arguments. You could be comparing files on remote servers, the output of database queries, or logs from different services.
Beyond 'diff': Getting More Creative
Process substitution isn't just for `diff`. Its power comes from its versatility. Imagine you have a script that requires multiple sorted lists as input files. Instead of creating sorted temp files, you can just feed it sorted output directly:
`my_script <(sort list1.txt) <(sort list2.txt)`
Another powerful use case is getting around the frustrating variable scope issues with pipes. When you pipe a command into a `while read` loop, the loop runs in a subshell, meaning any variables you set inside it disappear once the loop is done. By using process substitution, you can keep the loop in the main shell:
`while read line; do`
`# process the line`
`done < <(grep 'ERROR' application.log)`
Here, the `while` loop’s standard input is redirected from the output of the `grep` command, but the loop itself remains in the current shell, preserving any variables you modify.
So Why Is It a Secret?
If process substitution is so useful, why isn't it common knowledge? Part of the reason is that it's not taught in most beginner Bash tutorials, which tend to focus on the more straightforward pipe (`|`) operator. The `<()` syntax can also look intimidating compared to a simple pipe. Furthermore, it's a feature specific to shells like Bash and Zsh, so it isn't strictly POSIX compliant, which can make some hardcore scripters shy away from it for maximum portability. But for the vast majority of developers and system administrators working in modern Linux or macOS environments, it's a safe, powerful, and ridiculously useful tool to have in your back pocket.
