Linux Admin - Shell Scripting
Introduction to Bash Shell
Like flavors of GNU Linux, shells come in many varieties and vary in compatibility. The default shell in CentOS is known as the Bash or Bourne Again Shell. The Bash shell is a modern day, modified version of Bourne Shell developed by Stephen Bourne. Bash was the direct replacement to the original Thompson Shell on the Unix operating system developed at Bell Labs by Ken Thompson and Dennis Ritchie (Stephen Bourne was also employed by Bell Labs)
Everyone has a favorite shell and each has its strengths and difficulties. But for the most part, Bash is going to be the default shell across all Linux distributions and most commonly available. With experience, everyone will want to explore and use a shell that is best for them. However at the same time, everyone will also want to master Bash shell.
Other Linux shells include: Tcsh, Csh, Ksh, Zsh, and Fish.
Developing skills to use any Linux shell at an expert level is extremely important to a CentOS administrator. As we mentioned previously, unlike Windows, Linux at its heart is a command line operating system. A shell is simply a user interface that allows an administrator (or user) to issue commands to the operating system. If a Linux system administrator were an airlines pilot, using the shell would be similar to taking the plane off auto-pilot and grabbing the manual controls for more maneuverable flight.
A Linux shell, like Bash, is known in Computer Science terms as a Command Line Interpreter. Microsoft Windows also has two command line interpreters called DOS (not to be confused with the original DOS operating system) and PowerShell.
Most modern shells like Bash provide constructs allowing more complex shell scripts to automate both common and complex tasks.
Constructs include −
- Script flow control (ifthen and else)
- Logical comparison operations (greater than, less than, equality)
- Loops
- Variables
- Parameters defining operation (similar to switches with commands)
Using Shell Script Versus Scripting Language
Often when thinking about performing a task administrators ask themselves: Should I use a shell script or a scripting language such as Perl, Ruby or Python?
There is no set rule here. There are only typical differences between shells versus scripting languages.
Shell
Shell allows the use of Linux commands such as sed, grep, tee, cat and all other command-line based utilities on the Linux operating system. In fact, pretty much any command line Linux utility can be scripted in your shell.
A great example of using a shell would be a quick script to check a list of hosts for DNS resolution.
Our simple Bash Script to check DNS names −
#!/bin/bash
for name in $(cat $1);
do
host $name.$2 | grep "has address"
done
exit
small wordlist to test DNS resolution on −
dns
www
test
dev
mail
rdp
remote
Output against google.com domain −
[rdc@centos ~]$ ./dns-check.sh dns-names.txt google.com
-doing dns
dns.google.com has address 172.217.6.46
-doing www
www.google.com has address 172.217.6.36
-doing test
-doing dev
-doing mail
googlemail.l.google.com has address 172.217.6.37
-doing rdp
-doing remote
[rdc@centos ~]$
Leveraging simple Linux commands in our shell, we were able to make a simple 5-line script to audit DNS names from a word list. This would have taken some considerable time in Perl, Python, or Ruby even when using a nicely implemented DNS Library.
Scripting Language
A scripting language will give more control outside the shell. The above Bash script used a wrapper around the Linux host command. What if we wanted to do more and make our own application like host to interact outside the shell? This is where we would use a scripting language.
Also, with a highly maintained scripting language we know our actions will work across different systems for the most part. Python 3.5, for example, will work on any other system running Python 3.5 with the same libraries installed. Not so, if we want to run our BASH script on both Linux and HP-UX.
Sometimes the lines between a scripting language and a powerful shell can be blurred. It is possible to automate CentOS Linux administration tasks with Python, Perl or Ruby. Doing so is really quite commonplace. Also, affluent shell-script developers have made a simple, but otherwise functional, web-server daemon in Bash.
With experience in scripting languages and automating tasks in shells, a CentOS administrator will be able to quickly determine where to start when needing to solve a problem. It is quite common to start a project with a shell script. Then progress to a scripting (or compiled) language as a project gets more complex.
Also, it is ok to use both a scripting language and shell script for different parts of a project. An example could be a Perl script to scrape a website. Then, use a shell script to parse and format with sed, awk, and egrep. Finally, use a PHP script for inserting formatted data into MySQL database using a web GUI.
With some theory behind shells, let's get started with the basic building blocks to automate tasks from a Bash shell in CentOS.
Input Output and Redirection
Processing stdout to another command −
[rdc@centos ~]$ cat ~/output.txt | wc -l
6039
[rdc@centos ~]$
Above, we have passed cat'sstoud to wc for processing with the pipecharacter. wc then processed the output from cat, printing the line count of output.txt to the terminal. Think of the pipe character as a "pipe" passing output from one command, to be processed by the next command.
Following are the key concepts to remember when dealing with command redirection −
| Number | File descriptor | Character |
|---|
| 0 | standard input | < |
| 1 | standard output | > |
| 2 | standard error | |
| append stdout | >> |
| assign redirection | & |
| pipe stdout into stdin | | |
We introduced this in chapter one without really talking much about redirection or assigning redirection. When opening a terminal in Linux, your shell is seen as the default target for −
- standard input < 0
- standard output > 1
- standard error 2
Let's see how this works −
[rdc@centos ~]$ lsof -ap $BASHPID -d 0,1,2
COMMAND PID USER **FD** TYPE DEVICE SIZE/OFF NODE NAME
bash 13684 rdc **0u** CHR 136,0 0t0 3 /dev/pts/0
bash 13684 rdc **1u** CHR 136,0 0t0 3 /dev/pts/0
bash 13684 rdc **2u** CHR 136,0 0t0 3 /dev/pts/0
[rdc@centos ~]$
/dev/pts/0 is our pseudo terminal. CentOS Linux looks at this and thinks of our open terminal application like a real terminal with the keyboard and display plugged in through a serial interface. However, like a hypervisor abstracts hardware to an operating system /dev/pts abstracts our terminal to applications.
From the above lsof command, we can see under the FD column that all three file-descriptors are set to our virtual terminal (0,1,2). We can now send commands, see command output, as well as any errors associated with the command.
Following are examples for STDIN and STDOUT −
STDOUT
[root@centosLocal centos]# echo "I am coming from Standard output or STDOUT." >
output.txt && cat output.txt
I am coming from Standard output or STDOUT.
[root@centosLocal centos]#
It is also possible to send both stdout and stderr to separate files −
bash-3.2# find / -name passwd 1> good.txt 2> err.txt
bash-3.2# cat good.txt
/etc/pam.d/passwd
/etc/passwd
bash-3.2# cat err.txt
find: /dev/fd/3: Not a directory
find: /dev/fd/4: Not a directory
bash-3.2#
When searching the entire file system, two errors were encountered. Each were sent to a separate file for later perusal, while the results returned were placed into a separate text file.
Sending stderr to a text file can be useful when doing things that output a lot of data to the terminal like compiling applications. This will allow for perusal of errors that could get lost from terminal scrollback history.
One note when passing STDOUT to a text file are the differences between >>and >. The double ">>" will append to a file, while the singular form will clobber the file and write new contents (so all previous data will be lost).
STDIN
[root@centosLocal centos]# cat < stdin.txt
Hello,
I am being read form Standard input, STDIN.
[root@centosLocal centos]#
In the above command, the text file stdin.txt was redirected to the catcommand which echoed its content to STDOUT.
The pipe character "|"
The pipe character will take the output from the first command, passing it as an input into the next command, allowing the secondary command to perform operations on the output.
Now, let's "pipe" the stdout of cat to another command −
[root@centosLocal centos]# cat output.txt | wc -l
2
[root@centosLocal centos]#
Above, wc performs calculations on output from cat which was passed from the pipe. The pipe command is particularly useful when filtering the output from grep or egrep −
[root@centosLocal centos]# egrep "^[0-9]{4}$" /usr/dicts/nums | wc -l
9000
[root@centosLocal centos]#
In the above command, we passed every 4 digit number to wc from a text file containing all numbers from 65535 passed through an egrep filter.
Redirecting Output with &
Output can be redirected using the & character. If we want to direct the output both STDOUT and STDERR, into the same file, it can be accomplished as follows −
[root@centosLocal centos]# find / -name passwd > out.txt 2>&1
[root@centosLocal centos]# cat out.txt
find: /dev/fd/3: Not a directory
find: /dev/fd/4: Not a directory
/etc/passwd
[root@centosLocal centos]#
Redirecting using the & character works like this: first, the output is redirected into out.txt. Second, STDERR or the file descriptor 2 is reassigned to the same location as STDOUT, in this case out.txt.
Redirection is extremely useful and comes in handy while solving problems that surgace when manipulating large text-files, compiling source code, redirecting the output in shell scripts, and issuing complex Linux commands.
While powerful, redirection can get complicated for newer CentOS Administrators. Practice, research, and occasional question to a Linux forum (such as Stack Overflow Linux) will help solve advanced solutions.
Bash Shell Constructs
Now that we have a good idea of how the Bash shell works, let's learn some basic constructs, commonly used, to write scripts. In this section we will explore −
BASH Troubleshooting Hints
BASH can be a little tricky compared to a dedicated scripting language. Some of the biggest hang-ups in BASH scripts are from incorrectly escaping or not escaping script operations being passed to the shell. If you have looked over a script a few times and it is not working as expected, don't fret. This is common even with those who use BASH to create complex scripts daily.
A quick search of Google or signing up at an expert Linux forum to ask a question will lead to a quick resolution. There is a very likely chance someone has come across the exact issue and it has already been solved.
BASH scripting is a great method of quickly creating powerful scripts for everything from automating administration tasks to creating useful tools. Becoming an expert level BASH script developer takes time and practice. Hence, use BASH scripts whenever possible, it is a great tool to have in your CentOS Administration toolbox.
