Linux Piping and Redirection
- Linux Tips for Beginners
- Beginners guide to Reading and Finding Files in Linux
- Using Grep to Search for Text in Linux
- Understanding Linux File Permissions
- How to Archive, Compress and Extract files in Linux
- Linux Piping and Redirection
- Linux Hardlinks and Softlinks
- Basic Data Recovery in Linux
- Essential Guide to Working with Files in Linux
- Apache Administration on Linux
- MySql Administration on Linux
Every program which is run on the command line has three data streams and piping and redirection can be used to connect these streams between programs and files to direct data in interesting and useful ways.
The three data streams are:
- STDIN (0) - Standard input
- STDOUT (1) - Standard output
- STDERR (2) - Standard error
Let's have a look at each of these and how we can manipulate the streams to write files or send the output of one program to the input of another.
STDOUT is used to redirect output to a file. Redirecting output to a file will overwrite existing data if the file already exists. This can be handy to send the output of listings to a file for printing or emailing. Redirecting STDOUT to a file is as easy as using the greater than symbol.
ls -l > output.txt
There is also a double greater than (>>) which is also used to redirect output to file, however, this method will append the data to an existing file.
ls -l >| output.txt
STDERR is similar to STDOUT but it is used to handle error messages instead.
A simple way of demonstrating the difference between STDERR and STDOUT is to try and redirect the output of a command which generates in an error.
ls /etcw ls: cannot access /etcw: no such file or directory
We can then try and redirect this to a file:
ls /etcw > error.txt ls: cannot access /etcw: no such file or directory
Note, you can still see the error displayed on the screen. If you look at the contents of the error.txt, you will note that it is empty. This is because the STDOUT was redirected to the file, while the error message was sent to the console using a different pipe. By default, the greater than symbol output STDOUT only and since error messages are sent to STDERR, they were not redirected. We can redirect the error output to a log file by specifying the pipe when redirecting the output. This is done by entering the pipe number before the operator.
ls /etcw 2> error.txt
We've seen how to send output to a file, but we can also read the contents of a file and send it to a program's input. The output from one command can be piped directly into another command for further processing. This is done with the less than symbol symbol. We can use the diskfree command to show the available disk space on the system
We could pipe this to an output file using the output pipe
df -hlT > diskfree.txt
But in this example, I want to be able to send this data as an email to myself. As a system administrator I want weekly usage reports and to be able to take proactive action in the event of low disk space.
We can read in the contents of the text file and pipe it to the mail command which will read the STDIN and send the text file as an attachment.
mail -s "Disk Free" tim < diskfree.txt
This will then read in the file and send the contents as the body text of the mail message.
A here document is a special-purpose code block. It uses a form of I/O redirection to feed a command list to an interactive program or a command. They are very useful for creating small text files.
Using cat piped to a new file we can then use a here document to read input directly into the new file
cat > newfile <<END
You can now type text content into the file until you type in END. You don't have to use end, you could use something like EOF or anything else that wouldn't be in the input itself.
cat > newfile <<END This is a sample text file The cat sat on the mat END cat newfile This is a sample text file The cat sat on the mat
A lot of the little commands we have seen are designed to be plumbed together using pipes. That is the output from one command is directly fed into another command.
We've already seen one example when we piped the output of the find command to grep.
sudo apt --installed list | grep apache
We can also get a count of line numbers in a directory listing you can use
ls | wc -l 26
To examine the passwords file to get a count of logins. The passwords file lists the user logins along with the shell used.
head -n1 /etc/passwd root:x:0:0:root:/root:/bin/bash
Maybe we want to see the number of unique shell types that are in use.
We can see that the passwd is a : separated file
We can use cut command to split the line at : and that the shell command is the 7th field
cut -f7 -d: /etc/passwd /bin/bash /usr/sbin/nologin /usr/sbin/nologin /usr/sbin/nologin /bin/sync /usr/sbin/nologin /usr/sbin/nologin /usr/sbin/nologin /usr/sbin/nologin /usr/sbin/nologin /usr/sbin/nologin /usr/sbin/nologin /usr/sbin/nologin /usr/sbin/nologin /usr/sbin/nologin /usr/sbin/nologin /usr/sbin/nologin /usr/sbin/nologin /bin/false
Here is a list of all the default shells. We can sort this output
cut -f7 -d: /etc/passwd | sort
And then remove duplicate entries
cut -f7 -d: /etc/passwd | sort | unique
Now we can get a count of unique entries as well
cut -f7 -d: /etc/passwd | sort | unique | wc -l
Named pipes are used when allowing different processes to talk to each other. They don't hold any data, they merely marshal the data between two processes. Named pipes have a special file type of p which can be seen on the metadata block of ls -l
ls -l prw-r--r-- 1 timmy timmy 0 Aug 16 20:28 mynewpipe
Creating a pipe is done with the mkfifo command.
In this example, we'll see what happens when writing to this pipe in one ssh session and reading it in in another.
In terminal session 1, you can send output to the pipe the same as we have seen before
ls -l > mynewpipe
In session 2 you read the output in much the same way as before
wc -l < mynewpipe
Please note: The execution thread on the sending to the pipe will be locked until the data has been read out from the pipe. This means that if there is no service or command pulling data from the pipe, and commands or scripts sending data into the pipe will be blocked until the pipe is cleared.
As you may have noticed, when we pipe to a file or other command you cannot see the output on the screen. This is because the output was redirected away from stdout to the file or command.
We can, however, use the tee command to split the output of the screen and the output file or command.
ls > listing.txt [no outpt] ls | tee listing.txt drwxr-xr-x 2 timmy timmy 4096 Jun 24 18:01 Desktop drwxr-xr-x 2 timmy timmy 4096 Jul 8 21:08 Documents drwxr-xr-x 2 timmy timmy 12288 Aug 16 19:49 Downloads drwxr-xr-x 2 timmy timmy 4096 Sep 3 2016 Music
This will write the listing to the text file as well as show the listing on the screen.
Last updated on: Thursday 17th August 2017