Unix Related Data

Why do we Use Shell Scripting

We use Shell Scripts to obtain the following purposes.

1. Customizing your work environment.

2. Automating Your Daily Task.

3.Automating Repetitive Task.

4.Executing Important Procedures like shutting down the system,formatting a disk,Creating a file system on it,mounting the file system,letting the users use the floppy and finally un mounting the disk.

5.Performing the same operation on many files.

When Not to use Shell Scripting :

1. Is too complex,such as writing an entire billing system.
2. Requires a high degree of efficiency
3. Requires a variety of Software tools.

Usage of Shell Scripts with in Informatica:

1.To run an batch via pmcmd.

2. To have header and footer in case we are goanna write to an flat file

3.To run the command task.

4. To update parameter file with session start time and end time.

============================================================================

Basic Commands Part1

Command	Option	Description
1. Uname	uname -r uname -n	To find the version of unix os To find the machine name
2. Man	man grep man -e grep man -k copy	Print the user manual for any command. To find the user manual for grep command. One line definition of grep usage. Helpful when we donot know which command will solve the purpose.e.g :- gives the different option available for copying.
3.Cal	cal 2009 cal dec	Print the calendar year or month Print the calendar 2009 year Print calender for current december month
4. Date	date date +%m date +%h date +'%h%m' date +%d date +%y H,M and S	Tue Mar 09 07:10:10 IST 2009 3 March Mar-03 10 9
5.Who	who who am I who -H who -u	Login details of all the users in system Login details for only myself Login details for all users with headers info Login details for only active users
6.Passwd		To change the password
7.Lock	lock -45	lock the terminal for 45 minutes
8.Bc		To use the calculator
9.ls	ls -x ls -F ls -a ls -r ls -x abc ls -R ls -ld ls -il testfile ls -lt ls -ltr ls -lut	To Print the contents of current directory. Prints Multi column output of files with their names. To Print only directoris and executable files.< * > indicate all files containing executable code and </> indicate the directories. Print all files including hidden files. <a> stands for all. Reverse the sort order of all the files with their names. Print the content of abc directory. Print all files and sub-directories in directory TREE. Print all the directories. Prints the inode number of file. Prints all the file with their last modification time keeping latest modified at top. Prints all the file with their last modification time in reverse order keeping last modified at bottom. prints last access time of files.
10.Cat	cat testfile cat > testfile	Print the content of any file. Create a new file with testfile name.
11.CP	cp -i testfile abc cp -r abc newabc	To copy file and directories. copying testfile to abc directory interactively. copying entire directiry structure abc to newabc recursively.
12.rm	rm -f newabc rm -r newabc	All the options of cp command is applicable for rm command. Removing directory newabc forcefuly. Recursive deletion will not remove the write protected files.
13.lp		Used for Printing the files
14.Split	split testfile split -72 testfile split testfile newfile	To split the big files in different small files of 1000 lines each<default>. Split the files into small files. change the default split line size to 72. Renaming the testfile to new name as <newfile>.
15. Cmp     comm     Diff		These three command are used to find the difference in two files
16.wc	wc -w testfile wc -c testfile wc -l testfile wc testfile	Print the numbers of words, lines and character in the file
17.Path	PATH=$PATH:/home/ gagan:/home/deep	Set the path variable for SHELL to use these directory path to locate the executables commands.
18.stty	stty -a   stty intr \^c stty eof \^a	Print current setting of terminal To delete the character during backspacing To change the interrupt key to Ctrl c instead of 'DELETE' default To change the termination control of input during file creation using cat command from Ctrl d to Ctrl a.
19.Alias	alias alias l ='ls -ltr' alias showdir ='cd $1;ls -l' unalias	Print all the alias set in the system. To create short hand names of command. Passing positional parameter ETL to showdir alias will take us to that directory. To redefine and unset the alias.
20.History	history -5 history 10 15 HISTSIZE 1200	To find the previous command that has been used. Print the last 5 command that has been used. Every command has event number. Display event number between 10 an d15. To change the defalult setting of command history saving feature.
21. r       !!	r-2 r 20	Repeat previous command in Korn Repeat previous command in Bash Repeat previuos to prevoius command Repeat the command with 20 event number.
22.Tilde(~)		Changes control to home directory Changes absolute path home directory of user a_cmdb, the value of home directory can be seen by $HOME environment variable
23.cd -		To switch between current working directory and most recently used directory.
24.chmod	chmod u+rwx newfile chmod g+x newfile chmod 757 newfile chmod 457 newfile chmod -R a+x abc chmod -R 001 .	Changing the permission of file for user <u>, group <g>, others <o> or all <a> with + as assign , - as remove and = as absolute permission.r read<4>, w write<2> and x execute<1>.         Recursively changes the permission to execute for all directory and sub directory in abc directory for all users. . Indicate the current directory.

=================================================================================

Basic Commands Part2

Command	Options	Description
25.Head	head newfile head -5 newfile head -1 newfile | wc -c vi 'ls -t | head -1' head -c 512 newfile head -c 1b newfile head -c 2m newfile	Display top content of file Display top 10 lines of file when used without argument Display top 5 lines of file<newfile> when used without argument To count the number of character present in first line. To open the last modified file for editing. To pick special no of character from file. Print first 512 byte character Print first 2 blocks (1 MB each)
26.Tail	tail -5 tail +10 tail -f 6_load.log	Display end of file Display last 5 lines of file. start dispplaying 10 line onward till the end. Print the gworth of file
27.$$		Prints the Process id of current shell
28.ps	ps -f ps -f -u a_cmdb ps -a ps -e	Prints the Processes associated with current user. Prints the Processes associated with current user with their hierarchy.f stands for full. Prints all the processes associated with a_cmdb user. Prints all the users processes. Prints system process. prints all the ksh process running in the system by first searching for all the process.
29.Background job	ps -f ps -f -u a_cmdb ps -a ps -e	Running job in background Run the search job in background by printing its process id.Shell become the parent of all background process. This command print all the jobs runnung in background with their sequence no of exceution. Running fg will bring most recently started background process <LAST IN FIRST OUT> First job will come into foreground.
30.nohup	nohup ps -eaf | grep 'ksh' &	Shell is parent of all background jobs and it dies when user logs out and ultimately all the child process also dies.To avoid dying of shell even when the user logs out run the background process using nohup command.  System process init having PID as 1 is parent of all SHELL processes and when the user login this PID become the PPID of SHELL prrocess. Now after running the process in nohup mode kernel has reassigned the PPID of ps process with PID of system process which will not die even after SHELL dies through logging out.
31.Kill	kill 520 kill 640 kill 1 kill $! kill 0	To kill the command. Kill the process with 520 process id. Kill the parent process id which inturn kill all the child process. Killing system process init is not possible. A Process having process id as 1 is parent of all SHELL processes. No need to remember Process id of last background procees its in $! Kill all process in the system except login shell process. $$ stores the PID of current login shell
32.Nice	nice who | wc - l & nice -n 10 who | wc -l &	Running the command with low priority Nice value range from 1 to 39, higher the nice value lower the priority.default nice value is 20. nice value becomes 30.
33.At	at 2:10 load.ksh	To Run the job at specied time of the day. Indicate the load.ksh will execute today by 2:10
34.Batch	batch load.ksh	batch command executes the process whenever the CPU is free.
35.Cron		cron executes process at regular intervals it keeps checking the control file at /user/spool/cron/crontabs for jobs.
36.Finger	finger finger a_cmdb	Produces list of all logged users when run without any argument. The 2nd and last field of output is taken fron passwd file from /etc directory.  Prints details about a_cmdb user.
37.Set	set 10 20 30 -x	set the posional parameter. This will set $1, $2, $3 , $* and $# posional parameter When this statement is used at start of ksh script it echoes each statement at the terminal
38.Shift	shift shift 2	shift command transfer the contents of positional parameter to next lower number. shift one position. shift 2 places
39.Chown	chown jaspreet testfile chown -R jaspreet testfile	To change the owner of file can only be done by owner or sys admin.  change the owner from gagan to jaspreet for testfile. Now gagan cannot change the ownership of test file what he can do is just create the similar copy of file and then he can become the owner of file.  Recursively changes the owner all files under current directory.
40.Chgrp	chgrp GTI test file chgrp -R GTI test file	To change the group of file can only be done by owner or sys admin. group changed to GTI from IB as user is still the same and he can again change the group of file to IB. Recursively changes group all files under current directory.
41.Touch	touch 03161430 test file touch -m 04161430 test file touch -a 05161430 test file	Changing the time stamps of file touch without any argument will change both modified time and access time. Changes only modification time. Changes only access time.
42.Linking in	ln 6_load.ksh 7_load.ksh rm 6_load.ksh	Linking two file, doing this both the file will have same inode number and changes in one will reflect in another also.  ls -li 6_load.ksh 7_load.ksh will give us same inode number.  Drop the link between two files.
43.df	df -t /home/oracle	Prints the amount of free space available on the disc. Prints the free and total space under oracle file system.
44.du	du -s du -s /home/*	Prints the disk usage of specific directory tree. By default du prints disk usage for each sub directory. Prints summary of whole directory tree. Prints disk usage for every user

==============================================================================

Zipping Files

Command	Options	Description
1. gzip	gzip etl_code eg:etl_code.gz
	gunzip etl_code.gz eg: etl_code
	zip file*sql eg: file.zip
	unzip file.zip eg:file*sql
2. tar		create backup of files recursively.
	-cvf eg :tar -cvf /home/gagan/sqlbackup ./*.sql
	- x eg:tar-xcvf /home/gagan/sqlbackup ./*.sql	files are restored using -x option.
3. Cut	-c <column start and no> eg : cut -c -5,6-12 test file	column (by specifying position). C stands for column cut.
	-f <field start and end no> eg:cut -f 1,5 test file	field (default deliminator tab). F stands for field cut.
	'-d -f <field start and end no> eg: cut -d "|" -f 1,5 test file | new file	Cut the field b/w 1 and 5 and piped the output to new file.
4. sort	sort test file	By default the sorting starts with first character of each line. Priority1.space, tabs 2. numerals 3.uppercase letters 4. Lower case letters.
	-t eg : sort -t "|" +2 test file	Sorting starts from 3rd field skipping 2nd field, overring the default.delimiter to distinguish b/w start and end of field.
	-r eg : sort -t \| -r +2 test file	Reverse sort starting with 3rd field.
	eg : sort -t \|  +2r test file	The above command can be written in another way.
	-o eg : sort -o abc.txt abc_sort.txt	save sorted data in file.
	eg : sort -t "|" +1 -2 +3 <>	Sorting based on different field as in case of order by clause.Sorting starts with 2nd field then with 4th field, -2 indicate to stop the sorting after 2nd field and resume it with 3rd field.
	-n eg : sort -n <>	Numeric sort.
	-u eg : sort -u <>	Unique sort.
5.paste	-d eg:paste -d "|" <> <>	deliminator.
6.tr	eg : tr '|\' '~-' <test.txt	translate all | with ~ and \ with - .
	eg : tr '[a-z]' '[A-Z] <test.txt	translate to upper case.
	-d eg : tr -d '|' test.txt	delete all occurrence of | .
7.uniq		Unique require sorted file as input.
	-u eg : cut -d "|" -f3 <>|sort|uniq -u	remove duplicate.
	-d eg: cut -d "|" -f3 <>|sort|uniq -d	select only dup records.
	-c eg : cut -d "|" -f3|sort|uniq -c	duplicate  count.
8.Changing time stamp		touch mon date hrs mins <file>.
ls -lt	time of last modification	touch -m 01290430 <file>.
ls -lu	time of last access	touch -a 01290415 <file>.
9.Change Date	date 09181754
10.wall	wall -g dba "hello"	to selectively send msg to dba group.
11.shutdown	shutdown -g2	power down after 2 mins.
	shutdown -y -g0	immediate shutdown.
	shutdown -y -g0 -i6	shutdown and reboot (init level 6).
	shutdown 17:30	shutdown at 17:30.
	shutdown -r now	shutdown immediate and reboot.
12. du		Disk usage.
	du /home/expimp/create_db	tree output for each directory inside.
	du -s //home/expimp/create_db	summary.
13.find	find <loc> <option> <pattern> <action>	find in root dir abc in emp.lst file
	'-mtime = mod time eg: find . -mtime -2 -print	find file modi in less then 2 days.
	'atime = access time eg: find . -atime +365 -print	find the file not accessed in last 1 year.
	! –newer eg: find / -name "*.pl" ! -newer last_backup -print	file modi before last_backup.
	-size. eg: find . -size +2048 -print	files greater then x blocks.
	-a (and) -o (OR) eg: find . \( -name "*.sh" -o -name "*.lst" \) -print	double quotes necessary.
	-exec eg: find . -atime +180 -ok rm -f {} \;	remove the files which are not modi for last 20 days
	-ok eg: find . -atime +180 -ok rm -f {} \;	before removing prompt for confirmation.
	xargs eg: find . -mtime +20 | xargs rm -f	remove all file rm will be executed only once.
	xargs -n -p –t eg:find . -mtime +20 | xargs -n20 -p -t rm -f	remove at max 20 files in batch and in interactive mode.
	-type eg: find / -name log -type f -print	f for file and d for directory.
	-prune eg: find . -name *.log -prune exe –print.	don't descend exe directory.

=============================================================================

Pattern Searching

Commands	Options	Description
/	/Unix	Forward search of Unix keyword in file
?	?Unix	Backward search of Unix keyword in file
n		Repeat the last search
.		Repeat the previous command
:1, $s/<search string>/<Replace String >/g		Pattern search and replacement.
	1,$	Represent all lines in the file.
	g	Stands for globally
:3,10s/gagan/deep/g		search between lines 3 and 10
: .s/gagan/deep/g		Only the current line.
: $s/gagan/deep/g		Only the last line.
: $s/gagan/deep/gc	c	Ask for confirmation for replacement
grep
	-c	counting occurrence.
	-n	display line number for record.
	-v	skip records that contain directory.
	-l	display files containing record.
	-i	ignore case.
	[PQR]	match any single character.
	[c1-c2]	match char with ASCII range
	[^PQR]	match single character which is not PQR.
	^<pat>	beginning with pattern.
	<pat>$	ending with pattern.
	{a-Z A-Z 0-9}	match any single character.
	ls –l |grep “^d”	Prints only directories.
egrep
	egrep '[aA]g+[ar][ar]wal' test1.txt	match one or more occurrence matches ag & agg.
	egrep '[aA]gg?[ar][ar]wal' test1.txt	match zero or one occurrence.
	egrep prashant|director test1.txt	match eap1 or eap2  (finds prashant or line with director)
	egrep (das|sen)gupta test1.txt	match exp x1x3 or x2x3 (like dasgupta & sengupta)
	egrep -f <pattern_file_name> test1.txt	Huge list of pattern search can passed in the form of file name .pattern stored in file eg(prashant|admin|director)
	+	Matches one or more occurrence of previous character.
	?	Matches zero or one occurrence of previous character.
Fgrep		fgrep and egrep accepts multiple pattern both form command line and a file but unlike grep and egrep  does not accept regular expression.
	fgrep -f pattern file emp file	Faster than grep and egrep family

========================================================================

Pattern Matching

 

Command	Options	Description
*		Matches any number of character including NONE. * does not match all files beginning with dot <.>.
	ls -l chap*	Matches all the files which starts with chap.
	ls -x chap*	Matches all the files which starts with chap and prints in multi column way.
?		Matches single character.
	ls -l chap?	Matches all the files with only 5 character name and should start with chap.
[ ijk ]		Matches single character either i or j or k.
[ !ijk ]		Matches single character that is not i or j or k
[ x - z ]		Matches single character that is not within the ASCII range of character x and z.
[ ! x - z ]		Matches single character that is not within the ASCII range of character x and z
	ls -l chap0[1 - 4]	Range specification is also available.
	ls -l [ a – z A-Z]	Matches all file names beginning with alphabet irrespective of case.
	ls -l chap[!0 - 9]	Matches all file names beginning with alphabet chap and not ending with any number.
	ls *.ist	Print all the files which end with ist extensions.
	cp chap?? abc	Copy all files starts with chap to abc directory.
	cmp chap[12]	Compares file chap1 and chap2.
	mv *  ../bin	Moves all the files to bin directory.
	cat chap[!0 - 9]	Concatenates all the files beginning with chap and not ending with number.
	ls -l *	* does not match all files beginning with dot <.>.
	ls -l .???*	The above problem can be solved with specifying first few character using meta character <?> explicitly.
Escaping Backslash ( \ )		Playing with file names which uses meta character in their file name.
	ls - l chap*	Print all files whose names start with chap but not the one whose name is chap*.
	ls - l chap\*	The above problem can be solved by escaping the special character.
Pipe |		To pass the standard output of one command as the standard input to another.
	who | wc -l	Output of who command <three users> passed as input to wc which count the number of lines present.
Tee	who | tee users list	Tee saves the output of who command in user list as well as display it also.
Shell Variable		Shell variables are initialized to null value < by default > so it returns null. All words starts with $ are considered as variable unless single quoted or escaped.
	a=ab, b=cd, z=$a$b. echo $z	shell concatenate two variable.
	echo '$10' eg: $10	All words starts with $ are considered as variable unless single quoted or escaped.
	echo "$10" eg: 0	Shell is looking for $1 variable which is undefined so passes null for this. $1 is part of positional parameter.

=============================================================================

Control Structures