null null

Introduction to Linux
Stefan Becuwe
HPC core facility CalcUA
Vlaams Supercomputer Centrum
based on “Linux: an introduction” (KU Leuven)
„
Contact: [email protected]
„
Example files: http://win.ua.ac.be/~sbecuwe/p/introlinux/
§ input.zip: example input files
§ script.zip: example script files
§ linux-intro.pdf
Larry Ewing, 1996
Linux history
„
Unix: first release in 1970 (Bell Labs (AT&T))
„
Linus Torvalds, CS student Helsinki University
§ develops Unix for PC
§ based on Minix
§ 10K lines of code in 1991 (v0.01), > 16M lines of code today
Hello everybody out there using minix I'm doing a (free) operating system (just a hobby, won't be big and professional like gnu)
for 386(486) AT clones. This has been brewing since april, and is starting to get ready. I'd
like any feedback on things people like/dislike in minix, as my OS resembles it somewhat
(same physical layout of the file-system (due to practical reasons) among other things).
I've currently ported bash(1.08) and gcc(1.40), and things seem to work. This implies that
I'll get something practical within a few months, and I'd like to know what features most
people would want. Any suggestions are welcome, but I won't promise I'll implement
them :-)
Linus ([email protected])
PS. Yes – it's free of any minix code, and it has a multi-threaded fs. It is NOT portable
(uses 386 task switching etc), and it probably never will support anything other than ATharddisks, as that's all I have :-(.
—Linus Torvalds
comp.os.minix, August 25, 1991
1970
1980
1990
2000
Time
2010
10.2
FreeBSD
7.0
NetBSD
BSD family
OpenBSD
BSD (Berkeley Software Distribution)
Bill Joy
SunOS
5.8
4.4
4.1.4
Darwin
NextStep 3.3
OS X
10.11
Apple
Xenix OS
Microsoft/SCO
GNU/Hurd
K16
GNU
Richard Stallman
Minix
Linux
4.3
Linus Torvalds
3.3
Andrew S. Tanenbaum
Research
UNIX
Bell Labs: Ken Thompson,
Dennis Ritchie, et al.
10.5
CommercialUNIX
UnixWare
AT&T
Univel/SCO
11.3
Solaris
Sun/Oracle
System III & V family
11i v 3
HP-UX
7.2
AIX
IBM
IRIX
SGI
6.5.30
https://en.wikipedia.org/wiki/Linux
Linux distributions
„
Many flavors: “distributions”
§ Red Hat, SuSE, Debian, Ubuntu, etc.
§ variants: Fedora, CentOS, OpenSuSE, Kubuntu, Mint, …
„
Distro
§ Linux kernel (core common to all Linux distributions, “uname –a”)
§ installer (text-based or graphic installer with hardware detection, etc.)
§ application software: utilities, compilers, ...
§ tools for configuration and updating
https://en.wikipedia.org/wiki/Linux
Try it!
„
Install a Linux distribution (using a “live cd/dvd”)
§ Ubuntu: www.ubuntu.com/download/
§ OpenSUSE: software.opensuse.org/
§ Fedora: getfedora.org/
„
Install virtual machine software
§ Vmware player: www.vmware.com/products/player/
§ Virtualbox: www.virtualbox.org/
• virtualboxes.org/images/
Linux-like environments
„
Cygwin (Windows)
§ www.cygwin.com/
„
MacPorts (Mac OS X)
§ www.macports.org/
Operating system
“A program that acts as an intermediary between a user of a computer and
the computer hardware.”
§ Execute user programs (and make solving user problems easier)
§ Make the computer system convenient to use
§ Use the computer hardware in an efficient manner
Operating System Concepts Essentials. Silberschatz, Galvin and Gagne
Linux kernel
„
„
kernel is nothing more than a resource manager
resources being managed: process, memory, hardware device
http://www.ibm.com/developerworks/linux/library/l-linux-kernel/
Command line in the 21st
century?
„
text mode, no GUI
control the computer via the keyboard
„
somewhat archaic, but very powerful
„
can be much quicker than a GUI
need to get acquainted with commands, options, syntax, …
„
„
„
„
also in MATLAB, R, …
and in Mac OSX, Microsoft Windows, ...
Purpose of this course
„
learn the basics
become familiar with the “command line”
„
hands on examples
„
What is the shell?
What is the shell?
„
Command line interpreter: accepts input and performs requested task(s)
„
also a scripting language (variables, iterations, …)
„
100s of commands
„
Two basic types of shell prompts:
$ normal user shell
# root user shell (superuser privileges: can do anything)
Most popular shells
$ echo $SHELL
Shell
Prompt
Name
Note
sh
$
Bourne Shell
DefaultonsomeUnixsystems
bash
$
Bourne Again Shell
Enhancedreplacementforthe
Bourneshell.Defaultonmost
LinuxandMacOSXsystems
csh
%
CShell
DefaultonmanyBSDsystems
tcsh
>
TCShell
Enhancedreplacementforthe
Cshell
ksh
$
KornShell
DefaultonAIXsystems
Command line basics
„
Command history: use arrow key
„
Auto-completion: use <TAB> key
$ ls –l /etc/host<TAB>
„
Case and space sensitive (myfile vs. MyFile)
„
No recycle bin or trash can!
„
\ vs. /
§ Linux: / is directory separator, \ is escape character, - is command
argument delimiter
§ Windows: / is command argument delimiter, \ is directory separator
Working with commands
Commands
„
Some simple commands: date, cal, df
„
Some useful commands: clear, logout, exit
Commands?
„
An executable program
A built-in shell function
„
A shell function
„
An alias
Scripts
„
„
Working with commands
„
type – how a command name is interpreted
$ type type
$ type ls
$ type cp
$ type cd
„
which – which executable program will be executed (using current
value of $PATH)
$ which date
$ which cat
Working with commands
„
help for built-in shell functions
$ help cd
„
manual page for commands
$ man ls
„
usage information (if available)
$ cp --help
Working with commands
„
man – Display a command's manual page
$ man ls
§ Search for a word: /word
§ Search forward to next occurence of word: /
§ Show next line: <ENTER>
§ Show previous page: b
§ Show next page: <SPACE BAR>
§ quit from a manpage: q
Working with commands
„
alias – Create an alias for a command
$ alias
$ alias ls='ls --color=tty'
$ unalias ls
Some useful key
sequences
„
look for “Commands for Moving” in “man bash”
beginning-of-line(C-a)
Movetothestartofthecurrentline.
end-of-line(C-e)
Movetotheendoftheline.
forward-char(C-f)
Moveforwardacharacter.
backward-char(C-b)
Movebackacharacter.
forward-word(M-f)
Moveforwardtotheendofthenextword.Wordsare
composedofalphanumericcharacters(lettersanddigits).
backward-word(M-b)
Movebacktothestartofthecurrentorpreviousword.
Wordsarecomposedofalphanumericcharacters(letters
anddigits).
clear-screen(C-l)
Clearthescreenleavingthecurrentlineatthetopofthe
screen.Withanargument,refreshthecurrentlinewithout
clearingthescreen.
Hands on
„
Secrets of man pages:
§ Show the calendar of 2035.
§ Show the directory listing, sorted by date, most recent file at the
bottom.
§ Bash keeps track of all commands you have executed. How to show
and to clear the history?
§ How to search the history from the command line?
Hands on
„
Enter following commands and try to interpret the output.
echo hello world
last
lost
date
finger
clear
hostname
w
cal 2000
arch
top
cal 9 1752
uname -a
echo $SHELL
bc -l
dmesg | more
echo {con,pre}{sent,fer}
echo 5+4 | bc -l
uptime
man "automatic door"
yes please
who am i
man ls
time sleep 5
who
man who
history
id
who is still awake
Navigation
http://www.pinoytux.com/wp-content/uploads/2008/08/linux-dir-tree.jpg
Navigation
http://www.linuxplanet.com/linuxplanet/tutorials/6666/1/screenshot3894/
Navigation
„
Everything is a file:
§ Regular files
§ Directories: files listing a set of files
§ Symbolic links: files referring to the name of another file
„
Windowshasaseparate filesystemtree for each storagedevice(e.g., C-drive,
D-drive, I-drive, …).
„
Linuxhasasinglefilesystemtree, regardless ofhow many drivesorstorage
devices areattached to thecomputer. They areall mounted somewhere in
thetree.
Navigation
„
pwd – print name of current working directory
cd – change directory
„
ls – list directory contents
„
.
current working directory
„
„
..
~
working directory’s parent directory
home directory
„
-
„
previous working directory
Navigation
„
A path is a sequence of nested directories with a file or directory at the
end, separated by the / character
„
Absolute pathname: w.r.t. root directory (/)
§ /home/sbecuwe/courses/introlinux/scripts
„
Relative pathname: w.r.t. current directory
§ introlinux/scripts
Navigation
$
$
$
$
$
$
$
cd
cd ./Documents
pwd
cd ..
cd Documents
cd /bin
pwd
$ cd ~
$ cd -
Navigation
„
Hidden files (begin with a period): configuration and settings files. To
visualize: ls –a
„
File names are case sensitive
„
Although spaces are allowed in file names, it’s better not to use them.
„
Linux has no concept of a “file extension”
„
To determine file type, use file <filename>
ls
$ ls –a
$ ls -l
all files, including hidden files
long listing (type, date, size, owner, permissions)
$ ls –t
sort by modification time
$ ls –S
$ ls –r
sort by file size
reverses the sort order
$ ls -ltr
options can be combined
ls
• $ ls *txt
The shell first replaces *txt by all the files and directory names ending
by txt (including extension .txt), except those starting with ., and then
executes the ls command line.
• $ ls ?.log
Lists all the files which names start by 1 character and end by .log
http://www.thegeekstuff.com/2009/07/linux-ls-command-examples/
ls
• $ ls -d .*
Lists all the files and directories starting with .
-d tells ls not to display the contents of directories.
• bash doesn't expand a filename beginning with a dot unless the dot is
explicitly specified. You can change this default behavior by running
shopt -s dotglob
http://www.thegeekstuff.com/2009/07/linux-ls-command-examples/
less is more
„
Viewing file contents of “ASCII text” files (also valid for reading man
pages)
Page Up or b
Page Down or space
Up Arrow
Down Arrow
G
1G or g
/chars
n
h
q
Scroll back one page
Scroll forward one page
Scroll up one line
Scroll down one line
Move to the end of the text file
Move to the beginning of the text file
Search forward to the next occurrence of chars
Search for the next occurrence
Display help screen
Quit less
Redirection and pipelining
Redirection
„
In Unix, “everything is a file”:
§ ls sends output to a special file called “standard output” (stdout,
“1>”),
§ Status messages go to “standard error” (stderr, “2>”)
§ Input is taken from “standard input” (stdin)
„
By default, stdout and stderr are linked to the screen, whereas stdin is
linked to the keyboard
„
Redirecting I/O = changing where I/O comes from, not necessarily
screen and keyboard
Redirect output
$
$
$
$
ls –la
ls –la > ls-output.txt
ls –la
cat ls-output.txt
$ ls –la /jan > ls-output.txt
error message ?
empty file?
Append output
$
$
$
$
ls –la >> ls-output.txt
ls –la >> ls-output.txt
ls –la
cat ls-output.txt
$ ls –la /jan 2> ls-error.txt
$ cat ls-error.txt
$ ls –la /jan >> ls-output.txt 2>&1
$ ls –la /jan 2> /dev/null
/dev/null is a special device, discarding all data written to it (and a
favorite subject of technical jokes ;-)
cat
„
cat – concatenate files and copy to stdout
$ cat [-A][files]
$ cat
• Type <CTRL d> to indicate end of file (EOF)
$ cat > file
$ cat file1 file2 > filejoin
Pipelines
„
Pipelines: output from one command becomes input for another
command and so on
$ ls /bin /usr/bin | sort | less
„
uniq – report or omit repeated lines
„
„
wc – print line, word, and byte counts
grep – print lines matching a pattern
„
head / tail – print first / last part of files
Hands on
„
Use the file chemistry.txt:
§ How many courses can you find?
§ Which courses are taught by Wouter Herrebout in the first semester?
§ Which are, in alphabetical order, the last 5 course codes starting with
1001WET?
§ Which course can be found twice in the list?
Processes
Processes
„
“Everything in Unix is a file. Everything in Unix that is not a file is a
process.”
„
Processes
§ Instances of a running program
§ Several instances of the same program can run at the same time
§ Processes are assigned a unique identifier which is used to monitor
and control the process (PID)
Processes
„
Startup: init is launched and system services start; many are running
without user interface (daemons)
„
(Parent) process can launch child processes
„
ps – report a snapshot of current processes
$
$
$
$
„
ps
ps -x
# show all processes of current user
ps -aux
# show all processes of all users
ps -u [user] # show all processes of user [user]
top, htop – display tasks, sorted by cpu percentage
Controlling processes
$ xclock
# prompt no longer available
$ <CTRL C>
# interrupt process
$ <CTRL Z>
# stop process without terminating it,
# returns prompt, puts process to sleep
$ bg
# continue execution of the process
# in the “background”
Controlling processes
„
&: instruct shell to start program in the background
§ Run more than one program without having to start multiple terminal
sessions
$ ps
$ xclock &
$ ps
$ fg
$ <CTRL Z>
# return to foreground
# stops process
$ bg
# move to background
Controlling processes
$ kill <pid>
# kill specific process:
# let process do cleanup first
$ kill -9 <pid>
# immediate termination
Hands on
„
Which process is started as the first one?
How many processes are not owned by root?
„
Find two ways to kill the xclock process.
„
Manipulating files and directories
Creating directories
„
mkdir – create directories
$ mkdir dir1 dir2 dir3
„
mkdir -p – create directories and subdirectories
$ mkdir –p dir1/sub1/sub2 dir2/sub3/sub4/sub5
„
rmdir – remove empty directories
$ rmdir dir1 dir2 dir3
Copying files and
directories
„
cp – copy files and directories
$ cp [options] [source] [destination]
$
$
$
$
cp
cp
cp
cp
srcfile targetfile
srcfile1 srcfile2 srcfile3 targetdir
–i …
# asks for confirmation if target exists
–r srcdir targetdir
# recursive copy
Moving and renaming files
„
mv – move and rename files
$ mv [options] [source] [destination]
$
$
$
$
mv
mv
mv
mv
srcfile targetfile
srcfile1 srcfile2 srcfile3 targetdir
–i …
# asks for confirmation
srcdir targetdir
# rename if targetdir does not exist
# move srcdir into targetdir otherwise
Removing files and
directories
„
rm – remove files and directories
$ rm [options] [source]
$ rm srcfile
$ rm –i …
# asks for confirmation
$ rm –r srcfile srcdir
# also deletes contents of srcdir
$ rm –rf srcfile srcdir
# if srcfile or srcdir do not exist, continue silently
(f = force)
Removing files and
directories
„
BE CAREFUL WHEN USING WILDCARDS…
TRY IT FIRST USING ls
$ rm *.html
„
or
$ rm * html
how to remove a file starting with a hyphen (-)?
Soft links
„
ln – create links
$ ln –s source link
$ ln –s file_v123.tex file_final.tex
$ ln –s /usr/local/bin ~/mybin
§ Soft links allow to easily jump to other files or locations on the same
or another file system
§ Editing the symbolic link file == editing the source file
§ Deleting the source file before the symbolic link results in a broken
link.
§ Deleting the symbolic link does not delete the source file.
Wildcards
„
Wildcards make commands very powerful
(and sometimes dangerous ;-)
*
Matches any character
?
[chars]
Matches any single character
Matches any character that is a member of the set chars
[!chars]
Matches any character that is not a member of the set
chars
Matches any character that is a member of the specified
[[:class:]]
class
{w1,w2,w3}
Matches one of the words in the list
Wildcards
[:alnum:]
[:alpha:]
Matches any alphanumeric character
Matches any alphabetic character
[:digit:]
Matches any numeral
[:lower:]
[:upper:]
Matches any lowercase letter
Matches any uppercase letter
Hands on
Give an example for each of the following patterns:
*
[abcdefghijk]*.pdf
[[:digit:]]*.doc
backup.[0-9][0-9][123]
*[[:lower:]123]
Hands on
To check your answer, create a possible_filename and see whether
"ls pattern_above" shows the file or not:
$ touch possible_filename
$ ls pattern_above
Permissions and ownership
Permissions
$ file /etc/shadow
$ cat /etc/shadow
$ ls –l /etc/shadow
$ id
„
uid: id of the user
gid: primary group id of the user
groups: all groups a user belongs to
Permissions
$ ls -la
total 432
drwxr-xr-x
6 sbecuwe
drwxr-xr-x 29 sbecuwe
lrwxr-xr-x
1 sbecuwe
../extra/TLCL-09.12.pdf
-rw-r--r-1 sbecuwe
-rw-r--r-1 sbecuwe
-rw-r--r-1 sbecuwe
permissions
staff
staff
staff
staff
staff
staff
ownership
204
986
23
7 mrt 15:48 .
8 mrt 16:39 ..
7 mrt 10:39 TLCL-09.12.pdf ->
4117 7 mrt 15:07 chemistry.txt
191174 7 mrt 15:45 linux-wro.pptx
14180 27 apr 2009 tux.svg
size
modif. time filename
The number between permissions and ownership is the hardlink count of
the file, when referring to a file, or the number of contained directory
entries, when referring to a directory.
Permissions
d/l rwx
user
rwx
rwx
group other
l = link
d = directory
r = readable
file: open/read
directory: list contents if x also present
file: write/truncate
directory: create, delete, rename files
file: execute
directory: enter directory; list contents if r
also present
w = writable
x = executable
Permissions
„
chmod – change file mode
r = 4, w = 2, x = 1
$ touch file.txt
$ chmod 640 file.txt
rw = 6, r = 4, so
–rw-r-----
$ chmod u=r,g=r,o-rwx file.txt
$ chmod u+w file.txt
Ownership
„
chown – change owner
„
to share files and directories with multiple users belonging to the same
group, provided permissions are ok
$ chown :<group> file
$ chown <user>:<group> file
# root only
Hands on
„
Create a directory testdir containing the file testfile. Change the
ownership of directory and file to a common group. How to prevent
someone belonging to the same group to show the contents of
testdir , but at the same time allowing him to modify testfile?
Searching for files
Searching for files
„
locate – find files the easy way
§ rapid database search, just based on the name
§ depends on updatedb, often run once a day
$ locate [string]
$ locate bin/zip
$ locate zip | grep bin
Searching for files
„
find – find files the hard way
§ searches a given directory (and its subdirectories) for files based on a
variety of attributes: name, file type, size, permissions, modification
time, …
§ unlike ls, find does not produce results in sorted order. Its order is
determined by the layout of the storage device.
$ find ~ -type d | wc -l
§ counts number of subdirectories, starting from the home directory;
other types: f (regular file), l (symbolic link), …
§ contrary to ls, find includes hidden directories
Searching for files
$ find ~ -type f -name "*.JPG" -size +1M | wc –l
$ find ~ -type f -name '*.BAK' -delete
$ find ~ -type f -name 'foo*' -ok ls -l '{}' ';’
$ find ~ -type f -name 'foo*' -exec ls -l '{}' ';’
-ok is like –exec, but ask the user first
Modification times
„
touch – change file access and modification times
$ touch jack
§ If file exists: update modification and access times to current time
§ If file does not exist: create a new empty file
$ touch -t 1203100123 jones
§
March 10, 2012, 01.23h
$ touch –r jack jones
§
set jones’ modification time equal to jack’s modification time
Modification times
„
a file/directory gets created, accessed and modified, so we have ctime,
atime and mtime
„
to show all of them…
find . -maxdepth 1 -printf "%M %u %g %s\t%Ab %Ad
%AH:%AM\t%Cb %Cd %CH:%CM\t%Tb %Td %TH:%TM\t%P\n"
Welcome in the magic world of the
shell
“Each time you type a command line and press the enter key, bash
performs several processes upon the text before it carries out your
command. The process that makes this happen is called expansion.”
Shell
$ echo this is a test
$ echo *
§ * is expanded (non-hidden files in current directory) before echo is
executed
$ echo /usr/*/share
$ echo ~
§ user’s home directory
$ echo ~user2
§ user2’s home directory
Shell
$ echo $((expression))
§ arithmetic expression (only integers!)
§ operators:
+
addition
*
subtraction
multiplication
/
integer division
%
**
remainder
exponentiation
§ single parentheses may be used to group multiple subexpressions:
$ echo $(( (5**2) * (3*4) ))
Shell
„
brace expansion
$ echo Front-{A,B,C}-Back
$ echo {Z..A}
$ echo a{A{1,2},B{3,4}}b
$ mkdir {07..09}-0{1..9} {07..09}-{10..12}
Shell
„
shell variables: $’s!
$ printenv
# display environment variables
$ set
$ echo $SUER
$ echo $USER
# display all variables
$ echo $SHELL
„
command substitution: use output of a command as an expansion
$ echo $(ls)
Shell
$ echo The total is $100.00
„
Text inside double quotes: special characters lose their meaning,
EXCEPT “$”, “\”, “`”
$ touch "two words.txt"
$ ls –l two words.txt
$ ls –l "two words.txt"
$ ls –l two*
$ ls –l two?words.txt
$ mv "two words.txt" two_words.txt
$ echo "$USER $((2+2)) $(cal)"
$ echo "The total is \$100.00"
Shell
„
Text inside single quotes suppresses all expansions
$ echo text ~/*.txt {a,b} $(echo foo) $((2+2)) $USER
$ echo "text ~/*.txt {a,b} $(echo foo) $((2+2)) $USER"
$ echo 'text ~/*.txt {a,b} $(echo foo) $((2+2)) $USER'
Hands on
„
Find the total number of files and directories in your home directory and
show it using echo.
„
How to exclude hidden files / directories?
The environment
Environment
„
Environment variables provide a way to influence the behaviour of
software on the system, e.g., the language being used.
Environment
EDITOR
Thenameoftheprogramtobeusedfortextediting.
SHELL
Thenameofyourshellprogram.
HOME
Thepathnameofyourhomedirectory.
LANG
Definesthecharactersetandcollationorderofyourlanguage.
OLDPWD
Thepreviousworkingdirectory.
PATH
Acolon-separatedlistofdirectoriesthataresearchedwhenyouenter
thenameofaexecutableprogram.
PS1
PromptString1.Thisdefinesthecontentsofyourshellprompt.
PWD
The currentworkingdirectory.
USER
Yourusername.
Environment
$ echo $LANG
$ export LANG="nl_BE.UTF-8"
$ ls –l
$ export LANG="en_US.UTF-8"
$ ls -l
Environment
„
„
interactive login shell:
1. /etc/profile
2.
~/.bash_profile
3.
4.
~/.bash_login
~/.profile
(system wide, applicable for all users)
interactive non login shell (start a shell in a terminal in an existing
session, e.g., screen, nested shells, …):
§
~/.bashrc
Environment
„
non-interactive shell (no input expected, e.g., system tasks, init or
startup scripts): no files read
„
remark: ~/.bashrc is often loaded from ~/.bash_profile
Archiving
Archiving
„
Save and restore multiple files, including directories, to/from a single
file.
$ tar [options] [options_values] [files]
c
v
create a new archive
verbosely list files which are processed.
f
following is the archive file name
z
j
filter the archive through gzip
filter the archive through bzip2
x
extract files from archive
Archiving
$ tar cvzf archive_name.tar.gz dirname/
$ tar xvzf archive_name.tar.gz
$ tar tvzf archive_name.tar.gz
$ tar xvzf archive_file.tar.gz /path/to/file
$ tar xvzf archive_file.tar.gz /path/to/dir/
„
remark: z (.gz) can be replaced by j (.bz2)
http://www.thegeekstuff.com/2010/04/unix-tar-command-examples/
Writing shell scripts
Editing text
„
vi – the default Unix editor
§ vim: “vi improved”
„
nano – simple editor
„
emacs – advanced editor
„
gedit
„
…
vi
„
„
Command line editor, available on every Unix system
Lightweight and fast
„
The program starts up in command mode: it waits for a command to be
executed.
„
To switch to insert mode, press i. To return to command mode, press
<ESC>.
„
See pages 140-159 of TLCL for a “gentle introduction to vi”.
TLCL: http://linuxcommand.org/tlcl.php
Shell scripts
„
shell script = file containing a series of commands
1.
Write a script
2.
3.
Make the script executable
Put the script where the shell can find it (i.e., in a directory that’s in
your $PATH)
Shell scripts
„
#! will show up as first characters: “shebang” notation. It tells the
system the name of the interpreter to be used to execute the script (not
restricted to bash).
#!/bin/bash
#!/usr/bin/env python
§ useful when using Python from a module, not the system wide version
Shell scripts
$ cat script01.sh
#!/bin/bash
# script01
# This is our first script.
echo 'Hello World!'
# comment
$ chmod +x script01.sh
$ script01.sh
$ echo $PATH
$ ./script01.sh
Shell scripts
#!/bin/bash
# script02
currenttime=$(date +"%x %r %Z")
myname=$USER
echo "id: ${myname}, current time: ${currenttime}"
echo "braces are important: $currenttime1"
„
Remarks:
§ user variables can not start with a digit: $1, $2, … are special
variables (command line arguments)
§ creating a variable: without $, e.g., myname=
§ using a variable: with $, e.g., echo ${myname}
functions
#!/bin/bash
function func {
echo "step 2"
# script03
# shell function
return
}
echo "step 1"
func
echo "step 3”
„
Remark: example using functions defined in another file: script03a
and script03b
if
#!/bin/bash
x=5
if [ $x -eq 5 ]; then
echo "x equals 5."
else
echo "x does not equal 5."
fi
if commands; then commands
[elif commands; then commands…]
[else commands]
fi
# script04
test
„
Most frequently used command with if is
test expression
or its equivalent form
[ expression ]
„
Recent versions of bash include an enhanced replacement
[[ expression ]]
which adds an important string expression:
string1 =~ regex
Example (see: “Regular expressions” for more info)
[[ "$var" =~ ^-?[0-9]+$ ]]
test: files
file1 –nt file2
file1 –ot file2
file1 is newer than file2
file1 is older than file2
-d file
file exists and is a directory
-f file
-s file
file exists and is a regular file
file exists and has a length greater
than zero
file exists and is readable
file exists and is writable
file exists and is executable
-r file
-w file
-x file
…
test: strings
-n string
-z string
the length of the string > 0
the length of the string = 0
string1 = string2
strings are equal
string1 != string2
string1 > string2
strings are not equal
string1 sorts after string2
string1 < string2
string1 sorts before string2
test: integers
int1
int1
int1
int1
–eq
–ne
–le
–lt
int2
int2
int2
int2
int1 –ge int2
int1 –gt int2
int1 = int2
int1 ≠ int2
int1 ≤ int2
int1 < int2
int1 ≥ int2
int1 > int2
test: combining
„
Combining expressions:
Operation
[]
[[ ]]
AND
OR
-a
-o
&&
||
NOT
!
!
read
„
read – read values from standard input
#!/bin/bash
echo -n "Please enter an integer -> "
read int
echo -n "Enter one or more values > "
read var1 var2 var3 var4 var5
echo "int = ${int}, var1 = ${var1}, …"
# script05
read
„
Remarks:
§ -n prevents echo from printing a new line
§ extended version: see script05a
while
#!/bin/bash
count=1
while [ $count -le 5 ]; do
echo $count
count=$((count + 1))
done
echo "value of count: $count"
echo "Finished."
while commands; do commands; done
# script06
while
#!/bin/bash
# script06b
while read line; do
jobid=`echo $line | cut -f1 -d" "`
state=`echo $line | cut -f3 -d" "`
echo "$jobid $state"
done < showq.txt
„
combining while and read allows to read lines containing multiple
words, without worrying about IFS etc.
Command line arguments
#!/bin/bash
echo "Number of arguments: $#
\$0 = $0
\$1 = $1
\$2 = $2
\$3 = $3
\$4 = $4
\$5 = $5
\$6 = $6
\$7 = $7
\$8 = $8
\$9 = $9 "
# script07
Command line arguments
„
More than 9 arguments? Either use shift (see next slide) or ${10},
${11}, …
„
Last argument? ${!#} or $BASH_ARGV (Bash only) or ${@: -1}
Command line arguments
#!/bin/bash
echo "first argument in list: $1"
echo "last argument in list: ${@: -1}"
count=1
while [[ $# -gt 0 ]]; do
echo "Nr of arguments left = $#"
echo "Argument $count = $1"
count=$((count + 1))
shift
done
# script08
Command line arguments
„
Each time shift is executed, the value of $# is reduced by one,
the value of $2 is moved to $1, the value of $3 is moved to $2, etc.
for
#!/bin/bash
# script09
for i in A B C D; do
echo $i
done
for variable [in words]; do commands; done
Remark: if [in words] is omitted, for defaults to processing the
positional parameters (command line arguments)
for
#!/bin/bash
# script10
for i; do
if [[ -r $i ]]; then
max_word=
max_len=0
for j in $(strings $i); do
len=$(echo $j | wc -c)
if (( $len > $max_len )); then
max_len=$len
max_word=$j
fi
done
echo "$i: '$max_word' ($max_len characters)"
fi
done
case
#!/bin/bash
read -p "enter
case $REPLY in
[[:alpha:]])
[ABC][0-9])
???)
*.txt)
*)
esac
# script11
word > "
echo
echo
echo
echo
echo
"single alphabetic character." ;;
"A, B, or C followed by digit." ;;
"is three characters long." ;;
"is a word ending in '.txt'" ;;
"is something else." ;; ß
case word in
[pattern [| pattern]...) commands ;;]... esac
Regular expressions
Regular expressions
„
What? Symbolic notations used to identify patterns in text.
„
Caveat: not all regular expressions are the same; they vary slightly from
tool to tool and from programming language to language.
Regular expressions
„
Create some files to test drive regular expressions
$ cd
$
$
$
$
ls
ls
ls
ls
/bin > dirlist-bin.txt
/usr/bin > dirlist-usr-bin.txt
/sbin > dirlist-sbin.txt
/usr/sbin > dirlist-usr-sbin.txt
$ touch .zip 1.zip 1zip 22.zip 2zip
grep
„
grep – global regular expression print
-i
ignore case
-v
-c
invert match
print number of matches
-l
print name of each file that contains a match
-L
-n
print name of each file that does not contain a match
prefix each matching line with the line number
-h
suppresses output of file names
-E
interpret pattern as an extended regular expression
Metacharacters
„
In addition to literals (letters, digits), regular expressions may also
include metacharacters:
§ basic regular expressions (BRE):
^ $ . [ ] * \( \) \{ \}
§ extended regular expressions (ERE):
^ $ . [ ] * ( ) { } ? + |
Note: difference between BRE and ERE for ( ) { }
Metacharacters
.
Match any character
$ grep –h '.zip' dirlist*.txt
vs. ls *zip; ls *.zip; ls | grep .zip
^ $
anchors: beginning (^) or end ($) of line
$ grep –h 'zip$' dirlist*.txt
Character classes
[]
character class
[bg]zip
matches bzip and gzip
[b-g]zip
matches bzip, czip, dzip, …, gzip
[^bg]zip
[^b-g]zip
matches any zip not preceded by b or g
matches any zip not preceded by b, …, g
^[A-Z]
matches any word beginning with an
^[-AZ]
upper case letter
matches any word beginning with –, A or Z
Extended regular
expressions
„
Extended regular expressions: grep –E or egrep
$ cat /usr/share/dict/words | egrep "Et|Ut"
find Et or Ut in /usr/share/dict/words
$ grep –Eh ^(bz|gz|zip) dirlist*.txt
begins with bz or gz or zip
$ grep –Eh ^bz|gz|zip dirlist*.txt
begins with bz or contains gz or contains zip
$ grep –h ^\(bz|gz|zip\) dirlist*.txt
what happens….? why?
Extended regular
expressions
?
*
Match an element zero or one time
Match an element zero or more times
+
Match an element one or more times
{}
{n}
Match an element a specific number of times
exact n times
{n,m}
at least n times, at most m times
{n,}
{,m}
at least n times
at most m times
Hands on
„
Use the file/usr/share/dict/words:
§ How many five letter words do you find?
§ Which words start with chemi?
§ Which words contain both her and bout? (answer using 1 regular
expression)
Hands on
„
Give a regular expression which …
… recognizes phone numbers of the following form:
(03) 265 38 60, with or without brackets and spaces; you can find
some (il)legal examples in phonenr.txt
… finds words in /usr/share/dicts/words starting with
a capital letter and also contain two consecutive a’s
Text processing
sort
„
sort
-b
ignore leading blanks
-f
-n
ignore case
numeric sort
-r
reverse
-k f1[,f2]
sort on a key field located from field f1 to field f2
(default: f2 is end of line)
-t char
field separator character (default: space or tab)
sort
$ ls -l /usr/bin | sort -nr -k 5 | head
$ sort -k 1,1 -k 2n distros.txt
$ sort -k 1,1 -k 2 distros.txt
$ sort -k 3.7nbr -k 3.1nbr -k 3.4nbr distros.txt
$ sort -t ':' -k 7 /etc/passwd | grep –v ^# | head
uniq
„
uniq
-c
output (duplicate) lines, preceded by number
of occurences
-d
only output repeated lines
-i
-u
ignore case
only output unique lines (default)
cut
„
cut
-c
char_list
portion to be extracted: one or more
comma-separated numerical ranges
-f
field_list
same, but for fields
-d
delim_char
in combination with –f;
default: single tab character
$ cut -f 3 distros.txt
$ cut -f 3 –d ' ' distros.txt
$ cut -c15-24 distros.txt
$ cut -f 3 distrostab.txt
$ cut -f 1,3 distrostab.txt
diff
„
diff – detect differences between files
-i
ignore case
-w
-y
ignore all white space
output in two columns
-r
recursively compare directories
$ diff –i file1 file2
$ diff –y file1 file2
$ diff –r dir1 dir2
tr
„
$
$
$
$
tr – transliterate or delete characters
echo
echo
echo
echo
"lowercase letters" | tr a-z A-Z
"secret text" | tr a-zA-Z n-za-mN-ZA-M
"secret text" | tr "sec" " "
"secret text" | tr –d "sec" # delete
Other useful commands
„
paste – combine files, line per line
„
join – join files, based on a shared key field
sed
„
sed – stream editor
§ Editing on a stream of text (standard input or set of files) using BRE
§ typical usage: search and replace
sed 's/regexp/replacement/'
• By default: only first occurence on each line; to replace all
occurences: add ‘g’ at the end
• By default: case sensitive
§ powerful but somewhat complex
§ for larger tasks, you might choose awk, Perl, Python, …
sed
n
linenumber
$
last line
addr1,addr2
from line1 till line2
-n
do not print
s/regex/repl/
p
print current line
a
append text after current line
d
I
delete current line
case insensitive
g
all occurrences
sed
$ echo "front" | sed '1s/front/back/'
$ sed -n '/SUSE/p' distros.txt
$ sed -n '1,5p' distros.txt
$ sed '/Fedora/a from Redhat' distros.txt
$ sed '/Fedora/d' distros.txt
$ sed 's/chemie/scheikunde/I' chemistry.txt
sed
„
Rewrite MM/DD/YYYY in distrostab.txt as YYYY-MM-DD.
1.
Regular expression for MM/DD/YYYY :
[0-9]{2}/[0-9]{2}/[0-9]{4}$
2.
Insert subexpressions:
([0-9]{2})/([0-9]{2})/([0-9]{4})$
3.
Construct replacement: \n refers to the nth subexpression, so we
want:
\3-\1-\2
4.
Result (do not forget the metacharacters), to be entered on one line,
without extra spaces
$ sed 's/\([0-9]\{2\}\)\/\([0-9]\{2\}\)\/
\([0-9]\{4\}\)$/\3-\1-\2/' distrostab.txt
Hands on
„
Use the file showq.txt:
§ How many jobs are “Running”?
§ How many jobs per user are “Running”?
§ How many jobs per user are running, sorted in descending order?
§ Give the number of jobs per number of Procs, sorted.
§ Give, per user, the number of Running jobs and the number of Procs
in use.
Hands on
„
Use the file chemistry.txt:
§ Create one directory per course code (first column).
§ Create in each directory a file where the file name is the family name
of the first teacher (take into account spaces).
§ Put in each file the name of the course.
More information
„
„
www.uantwerpen.be/hpc/
www.vscentrum.be
„
dontfearthecommandline.blogspot.com/
linuxcommand.org/tlcl.php
„
free-electrons.com/doc/unix_linux_introduction.pdf
„
www.ibm.com/developerworks/linux/
„
www.howtogeek.com/tag/linux/
„
„
www.tldp.org
www.lwn.net
„
www.linuxjournal.com
„

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project