User Tools

Site Tools


Sidebar

os_cp:scripting

Creating a shell script

(Non)interactive mode

The login shell in a terminal emulator runs in interactive mode.
When the shell executable is invoked with a file as its argument, it runs in non-interactive mode, and reads the commands from the file rather than the standard input.
Some diagnostics are not printed in the non-interactive mode.

Exercise 1 Edit a file filename so that in contains the text:

date
echo hello world
date

Then, run /bin/bash filename.

Shebang

Shebang is the name for the first line of a file whenever the line contains #! followed by a full path to an executable (and optionally arguments).

A file that has execute permission and starts with a shebang is called a script. When a script is executed, the executable indicated in the shebang is used to interpreted the file.

For instance, when an executable file called myFile that starts with the line #!/bin/prog_name myArg is executed by the standard POSIX execlp("./myFile", "./myFile", NULL); function, then the kernel, after verifying the execute permissions and noticing the shebang, executes in return execl("/bin/prog_name", "/bin/prog_name", "myArg", "./myFile", NULL);.

An executable file that starts with #!/bin/sh is a shell script.
(just like an executable file that starts with #!/usr/bin/python3 is a python script).

Exercise 2 Modify the file filename created for the last exercise so that it starts with #!/bin/sh shebang. Then, grant execute permission to the file (e.g., with chmod +x filename). Finally, run the file with ./filename.

The . command

The shell has a . command (the command is literally a single dot).
The . filename executes the commands from the file filename in the current shell (as if the contents of the file were typed to the standard input). The file does not need to be executable.

Notice that ./file (and sh file) interprets the file in a new shell and . ./file (and . file if PATH contains a .) executes the commands from the file in the current shell.

Processing an input line by the shell

The shell, in that order:

1. reads a line of input echo ${Y} "$Z 2+3=$((2+3))" > file;  l 'baz bar'
2. splits the line into tokens echo, ${Y}, "$Z 2+3=$((2+3))", >, file, ;, l , 'baz bar'
3. parses tokens into commands echo, ${Y}, "$Z 2+3=$((2+3))", >, file   l, 'baz bar'
4. resolves aliases echo, ${Y}, "$Z 2+3=$((2+3))", >, file ls, -alF, --color=auto, 'baz bar'
5. expands certain tokens echo, a, b,    c d 2+3=5,                >, file ls, -alF, --color=auto, baz bar
5. sets up redirectionsecho, a, b,    c d 2+3=5
6. executes the command

(cf. POSIX documentation)

Splitting line into commands

Commands can be separated by:

  • ;cmd ; … fully executes cmd and only then goes to the next command,
  • &cmd & … starts cmd in background and immediately goes to the next command.

Each process returns an exit value. Value of 0 indicates success, non-zero value indicates abnormal termination1).
Commands can be joined with logical operators that depending on the return values:

  • &&a && b runs a and if a succeeds, b is run;  a && b returns success iff both a and b succeed,
  • ||a || b runs a and if a fails, b is run;           a || b returns failure iff both a and b fail.

Commands can be grouped by:

  • {} – groups commands to be executed in this shell,
    Warning: commands in { … } must be terminated by a ; or a &
    Warning: a space must follow the { and precede the }.
  • () – runs a new shell (subshell) and executes the commands in the subshell

To understand the difference, consider the following commands executed from the /home/student directory:

  • { cd /tmp; pwd; }; pwd – enters /tmp and outputs /tmp twice,
  • (cd /tmp; pwd); pwd – runs a subshell, enters /tmp and outputs /tmp in the subshell, the subshell terminates, the shell outputs /home/student.

Exercise 3 Run the commands pwd and ls from the same line.

Exercise 4 Write a line that will run the command cd Desktop and if it succeeds, will run a ls command.

Job control

When a & terminates a command, then the process is run in background.

At a time, at most one process can run in foreground, but any number of processes may run in background or be suspended.
One may suspend a foreground process by pressing Ctrl+z.

The list of processes controlled by the shell, that includes the background processes and suspended processes, can be printed by the jobs command.

To move a background or suspended process to the foreground, one has to issue the fg command. To continue a suspended process in the background, one can issue the bg command.
With no arguments, the fg and bg target the most recently suspended process, and if there are no suspended process, then the most recently backgrounded process.

The fg, bg and kill shell commands accept a job specifiers to select a particular process by its number, name, ….

Exercise 5 Run a command ping put.poznan.pl. Suspend it with Ctrl+z. Inspect the jobs table with jobs. Put the command back in foreground and terminate it with Ctrl+c.

Exercise 6 Run a command ping put.poznan.pl &. Inspect the jobs table with jobs. Put the command back in foreground and suspend it. Then, issue a command that will continue it in background.

Shell expansions

The shell expands the following tokens:

  1. the tilde in ~, ~/… (and …:~ / …:~/… in variable assignment) is expanded to home directory of the current user,
    the tilde followed by a username (in contexts as above), e.g., ~user, is expanded to home directory of the user,
  2. $NAME and ${…} are substituted by a variable/parameter value
  3. $((…)) is substituted by the result of arithmetic calculations2),
  4. $(…) and `…` are substituted by the standard output of given commands
  5. patterns (text including *, ? and […]) are expanded to filenames wherever possible

Inside double quotes only expansions 2-4 are performed.
Inside single quotes no expansions are performed.

Importantly, the results of expansions 2-4 are split again into tokens whenever the expansions are not double-quoted.

Parameters / variables

Environment variables – overview

Environment variables are key-value pairs associated with a process. Their names must not contain the equals sign (=).
The usual naming convention is that the names start with a capital letter and contain only capital letters, numbers and underscore.
In the shell, the characters a-zA-Z0-9_ are valid in variable names, and the variables must not start with a number.

Each process has a separate set of environment variables. Upon fork, the environment variables are copied. Upon exec…, the variables are either retained or replaced with a given set (cf. execv vs execve).
The C language functions getenv and setenv provide access to the environment variables.

Parameters and variables in shell

For the shell, the name parameter denotes either script/function arguments or some special variables, while the name variable denotes an ordinary shell variable (that is closely related to an environment variable).
The shell variables marked as exported are passed to child processes as their environment variables. One can export a variable called NAME by the means of export NAME command.

To set a variable, one has to write: NAME=value
Warning: NAME =value attempts to run the program NAME with argument =value.
Warning: NAME = value attempts to run the program NAME with arguments = and value.
Warning: NAME= value attempts to run the program value with environment variable NAME set to empty value.

One may unset the variable with unset NAME.

The list of all variables can be obtained by the set command (with no arguments), and the list of all exported variables can be obtained by the env command (with no arguments).

The shell substitutes $X and ${X} with the value of the variable X.
Moreover the following expansions are performed:

  • ${#X} → the length of the value (in characters)
  • ${X:-expr} → value of X if X is set and nonempty, expr otherwise
  • ${X:+expr} → expr if X is set and nonempty, empty value otherwise
  • ${X:number} → value of X with first number of characters skipped
  • ${X::number} → the first number of characters of X's value
  • ${X:skip:length} → the first length characters starting from skip+1 of the X's value
  • ${X%pattern}X's value with the shortest match of the pattern removed from the beginning
    ${X%%pattern}X's value with the longest match of the pattern removed from the beginning
  • ${X#pattern}X's value with the shortest match of the pattern removed from the end
    ${X##pattern}X's value with the longest match of the pattern removed from the end

Exercise 7 Print the value of a variable VAR using the echo command. Then, set the variable VAR to text.
Finally, print the value of the variable VAR again.

Exercise 8 Assign the value of VAR to a variable OTHER and run echo that outputs the value of the OTHER variable.

Exercise 9 Assign the value 5 to a variable named SIZE.
Then, construct a command that uses the variable SIZE to print The file size is 5MB

Exercise 10 Set the variable PROG to ls and the variable ARG to /tmp. Then run in the shell $PROG $ARG.

Positional and special parameters

Arguments of a script or a function are referred to by ${1}, ${2}, ${3}, ….
The first nine arguments can also be referred to by $1, $2, $3, …, $9. Moreover, the following substitutions are performed:

$# the number of arguments
$*
$@
all arguments split into single words
"$*" all arguments as a single token
"$@" all arguments, each being a separate token

Examples

Other special parameters include:

$0 within script, $0 holds the name of the script
$$ process identifier of the shell
$? exit status of the last command
$! process identifier of the last backgrounded process

Examples

Exercise 11 Write a shell script that prints its first and third argument.

Exercise 12 Write a shell script that prints its name and the number of its arguments.

Selected standard variables

PATHColon-separated list of directories where binaries are looked up
Notice: it's the exec…p system call that looks up an executable files with a matching name in these paths
HOMEPath to the home directory of the current user
PS1
PS2
The main prompt
The prompt for next line of multi-line commands
PWDCurrent working directory
EDITORDefault text editor
LOGNAME / USER
UID
Name of the current user
(Numeric) user identifier of the current user
LANGLanguage, region and character encoding; for instance de_CH.UTF-8 means german, Switzerland and UTF-8 encoding
RANDOMA random number (generated upon each access)

Exercise 13 Set the variable LANG to the value ja_JP.UTF-8 and run the program date.
Then, change the value of LANG to de_DE.UTF-8 and run the command rm -rf /root/.ssh/nope.

Exercise 14 Display the value of the PS1 variable. Then, change it.

Exercise 15 Try to run the lspci command. Then modify the PATH variable so that it additionally contains the paths /sbin and /usr/sbin. Retry the lspci command.

Arithmetic expansion

The shell can do simple integer maths – it substitutes $(()) with the result of an expression contained in the parentheses.
The shell understands assignments and operators similar to those in C (cf. table of operators).
Inside $((…)) one can leave out the $ preceding variable names, e.g., X=$((X+2)).

Exercise 16 Set the value of X and Y to some two digit numbers, and then:
      • increment X by 1,
      • increment Y by 2,
      • set Z to the result of X multiplied by Y,
      • calculate the reminded of dividing Z by 128.

Command substitution

The shell substitutes for $(command) (and `command`) the standard output of the command.
Notice that first the commands contained in $(…) execute, then the shell replaces $(…) with the standard output (and if the $(…) was unquoted, the result are split into tokens), and only afterwards the line containing $(…) is run.

Exercise 17 Write a command that will print the text In the current directory there are N files, with N being replaced by the real count of the files in the current directory.

Exercise 18 The command date +%H_%M_%S outputs the current time. Write a command that redirects the output of pstree -au to a file processes_TIME.log with the TIME substituted by the current time.

Exercise 19 Write one line that will:
      • assign the current time in nanoseconds (the output of date +%s%N) to a variable START,
      • run the command sleep 1s,
      • assign the current time in nanoseconds to a variable END,
      • output the time elapsed between START and END

Aliases

Aliases are intended to provide user-defined alternative names for a command.
Once a line is split into commands by the shell (but before other expansions are done), the shell checks whether a word that is present where a command name is expected matches any alias, and if so, the word is replaced by the value of the alias.

For instance: some Linux distribution in default configuration files set up the alias la to ls -la and the alias ls to ls --color=auto.
Typing la first turns that into ls  -la, and then ls is turned into ls  --color=auto, yielding total of ls  --color=auto  -la

To create a new alias, one should issue:
alias word=value
For instance: alias la="ls -la"
An alias can be removed by unalias word.

The alias command with no arguments lists defined aliases.

Aliases, just like variables, are set only for the current shell process.

Exercise 20 Create an alias called year that will translate to cal -my. Use the alias. Use the alias to display the calendar for 2025.

The read command

The read command reads a line from the standard input.
read requires as arguments a list of variable names, e.g., read LINE or read A B C
Once a line is read, it is split into words, and subsequent words are assigned to subsequent variables in the list.
If there are less words in the input line than the variables in the list, then the remaining variables are set to an empty value.
If there are more words in the input line than the variables in the list, then the last variable in the list gets the extra text.

Example:

Exercise 21 Print the text What is your name: , then read the name, and print the text Hello name!.

Exercise 22 Read two numbers at a time and then output their sum.

The test command

The test command allows evaluating logical tests that involve strings, numbers and files.
Apart from the name test, the command can also be invoked by the name [.
There is a slight difference between test and [: the latter needs an extra ] argument as the last argument.
E.g., test "foo" != "foo" is identical to [ "foo" != "foo" ].

Expression True when
[ "$X" ]
[ -n "$X" ]
"$X" nas nonzero length
[ -z "$X" ] "$X" has zero length
[ "$X" = "$Y" ] The strings are the same
[ "$X" != "$Y" ] The strings differ

Warning: the expression [ $X = $Y ] is going to work if $X and $Y expand to single words, but if either of the variables is empty or contains a space, then a syntax error will be raised.

Numeric tests

Expression True when
[ "$X" -eq "$Y" ]The number "$X" is equal to "$Y"
[ "$X" -ne "$Y" ]The number "$X" is not equal to "$Y"
[ "$X" -lt "$Y" ]The number "$X" is lesser than "$Y"
[ "$X" -le "$Y" ]The number "$X" is lesser than or equal to "$Y"
[ "$X" -gt "$Y" ]The number "$X" is greter than "$Y"
[ "$X" -ge "$Y" ]The number "$X" is greter than or equal to "$Y"

File tests (a choice of)

Expression True when the file "$X" exists and…
[ -e "$X" ] (just exists)
[ -s "$X" ] is nonempty
[ -f "$X" ]
[ -d "$X" ]
is an ordinary file
is a directory
[ -r "$X" ]
[ -w "$X" ]
[ -x "$X" ]
the current user can read the file
the current user can write to the file
the current user can execute the file

Negating and grouping tests

! arg negates arg.
arg1 -a arg2 stands for arg1 and arg2
arg1 -o arg2 stands for arg1 or arg2
Parentheses ( … ) group the expressions, but one has to escape them (else the shell sees them as shell special characters).

Exercise 23 Using the && and || syntax to group commands:
      • test whether a directory dirname exists, and if it does exist, enter it,
      • test whether the variable X is larger than the variable Y, and if the test fails, print Y≥X.

Control flow statements

Conditional constructs

if

The if statement has the following syntax:

if command1
then
    statements executed when command1 returned the exit value indicating success
elif command2
then
    statements executed when command2 returned the exit value indicating success
else
    statements executed when neither command1 nor command2 returned the exit value indicating success
fi
The elif and else branches are optional, and elif can appear multiple times.

It is worth underlining that the if and elif require a command and check the exit value returned by the command, and disregard the data written to the standard output/error streams by the command.

A one-liner example:

if grep -q ^user: /etc/passwd; then echo "'user' present"; elif grep -q ^student: /etc/passwd; then echo "'user' absent, 'student' present"; else echo "neither present"; fi

Exercise 24 Write an if statement that tries to remove a file filename and upon success outputs Removed the file while upon failure it outputs Failed to remove the file.

Exercise 25 Create a script that verifies if it was run with exactly two arguments and if the first argument is lesser then or equal to the second argument. If any of the conditions does not hold, print a corresponding text and exit the script.

case

The case statement (in many other languages called the switch statement) has the following syntax:

case value in
  pattern1) statements1 ;;
  pattern2) statements2 ;;
  *) statements3 ;;
esac
The value is consecutively matched against pattern1, pattern2, … and upon first match the corresponding statements are executed. Subsequent patterns are not taken under account.
Notice the double semicolon ;; terminating statements corresponding to a pattern.
There is no special syntax for the default match – simply a pattern of * is used.

A one-liner example:

case $(LANG= date "+%A") in Monday) echo "Ugh...";; S*day) tput bel; echo "Weekend!";; *) echo "a day.";; esac

Exercise 26 Write a script that accepts a filename as an argument, and if the file has an extension:
      • pdf, then run the pdftotext filename - command,
      • zip, then run the unzip -l filename command,
      • any other extension, then run the cat filename command.

Loops

In the loops one can use the break and continue keywords to, respectively, leave the loop and start the next loop iteration.

for

The for loop iterates over a set of values. It has the following syntax:

for VAR in value1 value2   
do
   statements
done
In subsequent iterations, subsequent values from the list are assigned to the VAR variable. Once the loop finishes, the VAR contains the last value assigned in the loop.
When the in value1 value2 part is omitted, the for iterates over the positional parameters.

Usually the list of values is not hand-written, but is a result of an expansion, for instance:
for NUM in $(seq 7)for NUM in 1 2 3 4 5 6 7
for IMG in *.jpgfor IMG in alice.jpg bob.jpg eve.jpg
for USERNAME in $(getent passwd | cut -f1 -d:)for USERNAME in root student ...
for FILE in $(grep -il 'ldap' /etc/*.conf 2>/dev/null)for FILE in /etc/autofs.conf /etc/ldap.conf ...3)

A one-liner example:

for X in `seq 3`; do echo -n "Iteration $X: "; date +%N; done

Exercise 27 Write a script that accepts a list of files as the arguments, and for each file prints its name, its first line and its last line.

Exercise 28 Write a loop that outputs all powers of two up to a given exponent.
(To calculate the power, you can use the expression $((2**exponent)).)

while and until

The syntax of the while loop is as follows:

while command1  
do
   statements2
done
and the syntax of the until loop is as follows:
until command1  
do
   statements2
done
The while executes statements2 as long as the return value of command1 indicates success.
The until executes statements2 as long as the return value of command1 indicates failure.

A one-liner example:

X=7; until [ $X -eq 1 ]; do [ $((X%2)) -eq 1 ] && X=$((3*X+1)) || X=$((X/2)); echo $X; done

Exercise 29 Write a loop that outputs all powers of two up to a given number.

Exercise 30 Write a script that will create a file myProg_1.log if it does not exist, or if such file exists, a file myProg_2.log, or if such file also exists, a file myProg_3.log, and so on. Put in the newly created file the current date.

Functions in shell

To define a function in the shell, one has to use the following syntax: name() body, where the body shall be a compound command such as a list of statements contained within a curly braces (e.g., mkcd() { mkdir -p $1 && cd $1; }).
To call a function, one has just to issue name [argument]... (e.g., mkcd /tmp/dir).
Inside a function the variables $# and $1, $2, … refer to the arguments.
The command return [n] issued inside a function terminates the function returning n (when specified) or the return value of the last executed command.

Consider the following example:

user@host ~ $ colorEcho() {
> case "$1" in
>   red) CCODE=31;;
>   green) CCODE=32;;
>   blue) CCODE=34;;
>   *) CCODE=5;;
> esac
> echo -ne "\033[${CCODE}m";
> echo -n "$2";
> echo -e "\033[0m";
> }
user@host ~ $ colorEcho red riding hood
riding
user@host ~ $ colorEcho green "anne of green gables"
anne of green gables

Exercise 31 Write a function that displays Hello world!

Exercise 32 Write a function that takes two arguments: a number and a text. The function shall output the text the number of times.

Exercise 33 Write a function that resembles the tree program. It should traverse the filesystem depth first and output for each file its name with appropriate indentation.

Signals in shell

The command trap "statements" CONDITION is used to set up either handing a signal (when the CONDITION is a name of a signal) or set the statements to be executed upon terminating the shell (when the CONDITION is EXIT).
Shell runs the commands associated with a signal once the current foreground command terminates or suspends.
Notice that the commands to be executed upon a signal / upon exit must be provided as text.
The trap - CONDITION syntax restores the default handler.

Exercise 34 Handle the INT signal in the current shell by running the fortune command. Trigger a SIGINT and see what happens.

Exercise 35 Create a script with the following contents:

TEMPFILE=$(mktemp)
echo "Created a temporary file $TEMPFILE"
date > "$TEMPFILE"
sleep 12s
date >> "$TEMPFILE"
echo "Cleaning up the temporary file $TEMPFILE"
rm "$TEMPFILE"

Run the script and terminate it with Ctrl+c, or Ctrl+\, or pkill -f scriptname.
Then modify the script so that it cleans up the file regardless of how it was terminated.

.bashrc

Most of the shells read certain configuration files upon start. There is no standard name for a user shell configuration file.

For all interactive shells, the Bash shell executes the commands from ~/.bashrc file upon start.
Therefore, a Bash user that wants to permanently define aliases, functions, set environment variables etc., shall add appropriate commands to the ~/.bashrc file.
The commands in .bashrc must output no text to standard streams when the shell starts in non-interactive mode.

1) Confer with int main(){…; return 0;} and the exit() function
2) POSIX requires only integer arithmetic
3) Warning: this example is not whitespace-proof.
os_cp/scripting.txt · Last modified: 2024/03/25 21:20 by jkonczak