Unix and Linux come with a whole toolkit of utility programs for filtering, sorting and manipulating data. As a "one off" job, you can use them from the command line, and for repeated jobs you can save a series of instructions for these tools in a shell script. Yesterday, I ran a private
shell scripting bootcamp course, and found myself enjoying working with
grep - the filtering tool
sort - the tool that re-orders data
awk - for pattern matching and data manipulation
sed - the stream editor
tr - for transforming characters
Awk is especially worthy of mention. With Awk, you can select rows of data based on what they contain, or what is in certain fields, and manipulate those lines. Instructions may be given to allow different instructions depending on how they're elected, and awk also includes integer and floating point arithmetic capabilities, and the ability to undertake actions before and after the processing of any of the data, in matching clauses called BEGIN and END.
I have added sample awk scripts
here and
here to our web site, and a slightly longer article on awk
here. And, yes, I can deliver training on awk and shell scripting ;-) ...
Linux/Unix Basics will get you started, and we can tailor to meet your needs from there!
(written 2007-09-25)
Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles
A166 - Web Application Deployment - Linux Utilities [63] Almost like old times - (2004-09-26)
[71] Comparators in Linux and Unix - (2004-10-03)
[1361] Korn shell course - (2007-09-22)
[1690] Conversion of c/r line ends to l/f line ends - (2008-06-28)
[2145] Using the internet to remotely check for power failure at home (PHP) - (2009-04-29)
[2320] Helping new arrivals find out about source code examples - (2009-08-03)
[2484] Finding text and what surrounds it - contextual grep - (2009-10-30)
[2638] Finding what has changed - Linux / Unix - (2010-02-17)
[3446] Awk v Perl - (2011-09-18)
[3764] Shell, Awk, Perl of Python? - (2012-06-14)
[3902] Shell - Grep - Sed - Awk - Perl - Python - which to use when? - (2012-10-22)
[4586] Extending your bash shell with aliases, functions and extra commands - (2015-11-28)
[4682] One line scripts - Awk, Perl and Ruby - (2016-05-20)
A101 - Web Application Deployment - Linux -An Introduction For Users [73] vi - full circle - (2004-10-04)
[74] pushd and popd - (2004-10-05)
[152] Aladdin, or careful what you wish. - (2004-12-15)
[249] An easy way out - (2005-03-17)
[430] Linux commands - some basics - (2005-08-31)
[431] File permissions of Linux and Unix systems - (2005-08-31)
[593] Finding where the disc space has gone - (2006-02-06)
[659] Web Application Components - (2006-03-28)
[679] More or less on the edge of the page - (2006-04-11)
[703] Copying files and preserving ownership - (2006-04-28)
[710] Linux training Glasgow, Python programming course Dundee - (2006-05-05)
[711] THE home directory or MY home directory - (2006-05-06)
[749] Cottage industry or production line data handling methods - (2006-06-07)
[984] Cardinal numbers and magic numbers - (2006-12-14)
[1012] Moving files between Windows / DOS and Linux / Unix - (2006-12-30)
[1013] Copy multiple files - confusing error message from cp - (2006-12-30)
[1068] ls -l report, Linux / Unix - types and permssions - (2007-02-06)
[1259] Where am I and how did I get here? - (2007-07-05)
[1287] Work and play at Well House Manor - Football and Shell Shortcuts - (2007-08-02)
[1288] Linux run states, shell special commands, and directory structures - (2007-08-03)
[1408] Wireless hotel tips - FTP and Skype connections failing - (2007-10-26)
[1438] Copy and paste / cut and paste and other vi techniques - (2007-11-20)
[1527] Selecting file names in a shell - one word or another - (2008-02-02)
[1651] ls command - favourite options - (2008-05-23)
[1764] Yank and Push - copy and move in vi - (2008-08-21)
[1803] FTP passive mode - a sometimes cure for upload hangs - (2008-09-20)
[1893] Some Linux and Unix tips - (2008-11-18)
[1897] Keeping on an even keel - (2008-11-21)
[1902] sstrwxrwxrwx - Unix and Linux file permissions - (2008-11-23)
[1904] Ruby, Perl, Linux, MySQL - some training notes - (2008-11-23)
[2201] Running straight from the jar, but not from a tar - (2009-05-26)
[2203] Always use su with minus. And where do programs come from? - (2009-05-27)
[2299] How much space does my directory take - Linux - (2009-07-20)
[2300] What does x on a linux directory mean? - (2009-07-21)
[2479] Accidentally typed ci rather than vi? - (2009-10-27)
[2494] Making Linux Politically correct - (2009-11-06)
[2636] Linux - useful tips including history and file name completion - (2010-02-15)
[2831] Recording (a macro) in vi - (2010-06-27)
[3179] Oops - I typed ci not vi, and have lost my file ... - (2011-02-21)
[3256] Displaying a directory or file system tree - Linux - (2011-04-22)
[3791] The Kernel, Shells and Daemons. Greek Gods in computing - (2012-07-01)
[3819] Packing a tar, jar or war file - best practise - (2012-07-26)
Some other Articles
Apache Tomcat Performance TuningOne business, four different angles!Getting the local voice heardkorn tips - some useful korn shell techniquesawk - a powerful data extraction and manipulation toolKorn Shell scripts on the webKorn shell course - resourcesOne computer per delegate, or pair programming?No Thank You