103.4 Use streams, pipes and redirects

Key Knowledge Areas:

• Redirecting standard input, standard output and standard error.
• Pipe the output of one command to the input of another command.
• Use the output of one command as arguments to another command.
• Send output to both stdout and a file.

The following is a partial list of the used files, terms and utilities:

• tee
• xargs

Redirecting output and input

We can run the a command but sometimes have the output displayed on the screen is not useful or desired, other times it is just more helpful to send the output through to a file. By output we generally refer to standard output, but we have both standard output and error output. We can have the output go to the same file or separate files is required. Moreover we have input redirection where we can have a program read a file for its input.

• ls -l . /ect > listing.txt #send the output from ls -l to a file called listing.txt, Standard output to the screen is suppressed but errors will display on the screen
• ls -l . /ect > listing.txt 2>&1 # sent both standard output and error output to the same listing file
• ls -l . /ect > listing.txt 2>error.txt #Here we separate the output and errors are listed in the error.txt file
• ls -l . /ect > listing.txt 2>/dev/null #Here we ignore errors sending them to /dev/null

The ls command is looking at the the current directory , (.), and a misspelled directory (/ect). Assuming the directory /ect does not exist then ls will display both standard output, the listing of the current directory and standard error, saying the /ect is not a file or directory.

I all the instances we are overwriting the file if it exists (>), if we want to append use the >> symbols. To prevent overwriting of existing files you may set the shell option noclobber

set -o noclobber

to turn this off again

set +o noclobber

To display current settings

set -o

With the noclobber option set you can forcefully overwrite an existing file with >|

If we need to see the output on the screen and have it sent through to a file then we will need to pipe the output through tot he program tee. Piping uses the vertical bar character | and takes the output of one command to be the input of another, so we can take the output of ls and send it through to tee using the pipe :

ls . | tee file.txt

The output will display on screen and be sent to the file file.txt. the program tee allows this.

Using xargs

/usr/bin/xargs can take the output from another program and feed it through to another, by default if no output program is mentioned then xarhs will use echo. the reason the use xargs as an intermediary allows you to specify how the code should be dealt with, as a single string or by breaking it up into individual arguments that are sent through to the destination program. Piping alone would not be able to adjust the flow of data to the receiving program in this way

• echo {1,2,3} | xargs # will result in 1 2 3
• echo {1,2,3} | xargs -n1 # will result in
• 1
• 2
• 3
• each argument being treated separately by default program of xargs (echo)
• find . -maxdepth 1 -name “*.jpg” | xargs tar -czvf pictures.tgz #will archive all jpg files in the current directory