Text processing with shell utilities (Linux / Cygwin)

This is a small collection of tips for processing text data with the Bash shell and some common utilities. I used these to help process numerical results that my research code had outputted in the form of text files. By automating the processing of text files, I saved a lot of time when preparing and condensing data for publication and thesis writing. Linux users should already have most if not all of the required utilities installed. Windows users should install Cygwin, which gives a Bash shell and all the necessary utilities.

Since Windows and Unix text files differ in the line ending characters used, be aware that there can be some issues when the line ending in a file is not what is expected. The Cygwin tools expect and produce Unix line endings. To convert between Unix and Windows (DOS) line endings, use dos2unix and unix2dos, both tools that come with Cygwin. iconv is another tool that can convert between many different text formats and character encodings.

Concatenate files line by line

If you have two files with the same number of lines and you want to concatenate them line by line, you can use the paste command. For example, you have files a.txt and b.txt, where a.txt contains:

1
2

and b.txt contains:

a
b

and you want to join files a.txt and b.txt line by line like the following (the lines from each file are separated by a tab character):

1	a
2	b

Then use the following command:

paste a.txt b.txt > c.txt

You can specify a delimiter to use between the lines of each file:

paste -d "," a.txt b.txt > c.txt

This will produce:

1,a
2,b

Merge specific CSV columns line by line

If you have two comma separated value (CSV) files and you want to merge specific columns from each file into a new file line by line (equivalent to opening the CSV files in Excel or another spreadsheet program and copying the desired column from each file and pasting them into a new file), you can use the paste command:

paste -d "," <(cut -d \, -f 1 FILE1) <(cut -d \, -f 2 FILE2) > outfile.csv

The two cut commands are performed first due to the <() operation. The cut commands extract column 1 (specified by -f 1, which means “field 1” using cut terminology) from FILE1 and column 2 from FILE2. cut uses the -d option to know what delimiter is used in the files to separate fields (columns). In this case, since FILE1 and FILE2 are CSV files, the delimiter is a comma.

The paste command then pastes the extracted columns together, with a comma between the columns. The output file is therefore also a CSV document. The above takes column 1 from FILE1 and column 2 from FILE2 and pastes them side by side in outfile.csv. I used this method extensively to summarize CSV files that were generated by a research program.

Extracting a range of lines from a text file

To print a specific line range from a file (in this case, print lines 2 to 4 from somefile.txt), use the sed utility:

sed -n 2,4p somefile.txt

You can also print multiple line ranges(lines 1 to 2, then line 4):

sed -n -e 1,2p -e 4p somefile.txt

The above will print lines 1 to 2 and then print line 4.

Of course you can append > outfile.txt to save the output from the above to a file.

Adding delimiters to fixed width data

If a file has lines that are of the format YYYYMMDD (i.e. dates), you can add delimiters so that the output file has lines with the format YYYY MM DD (using a space as a delimiter). To do this, use the cut utility:

cat textfile.txt | cut -c1-4,5-6,7-8 --output-delimiter=' ' > outfile.txt

Cut will print characters 1 to 4 (corresponding to YYYY), 5 to 6 (MM) and 7 to 8 (DD) with the specified delimiter (in this case, a space) between them.

Multiple File Search and Replace

The following replaces 'foo' in all files under the current directory (and all subdirectories, recursively) with 'bar':

find ./ -type f -exec sed -i 's/foo/bar/g' {} \;

You can use any valid option with find. For example, to limit the replacement to HTML files in the directory,

find ./ -type f -name '*.html' -exec sed -i 's/foo/bar/g' {} \;

I have noticed that sed changes the modification date of all files - even files that do not match. If this matters, then I would suggest using grep to get the list of files that contain the match and then make a script that processes each file in the grep result one by one.

Count number of lines matching a pattern

You can count the number of lines matching a certain pattern with the -c option in grep. To show all lines matching the pattern, followed by the count, use the following:

grep PATTERN file; grep -c PATTERN file

There are other ways for doing this with tools other than grep but I have not had to use them.

You can store the count to a Bash variable like this:

count=`grep -c PATTERN file`

Print a text file backwards

The cat command will print the file contents in the normal order. To reverse the order, use the tac command:

tac filename.txt

This will print the lines out one by one, but in reverse order (last line printed first). It does not reverse the lines themselves.

Grep for multiple words

You can grep for two (or more) words at once. For example, create a file containing:

One
Two
Three
Four
Five

Then use the following grep command (the -E option enables extended regular expressions, which lets you use the | as an “or” operator):

grep -E 'One|Two|Three' file.txt

The output is:

One
Two
Three

Discussion

priti, 2011/08/17 06:58
I want to search file1 from column1, 2 of file2. For ex:

File1:
X
Y
Z
File2:
A,B,C,D
X,E,F,L
Y,T,I,o

I need output like:
X,F
Y,I
Please help and revert urgentl!!
Peter Yu, 2011/08/18 21:08
I'm afraid I don't know how to do this offhand. Grep has an option (-f) that takes a list of patterns from a file and finds all matches in a second file. You may be able to use that.
Dave, 2011/09/05 14:56
Your writeup of the grep command was great and provided some useful information.
I was wondering if you could assist me using this command a little more..
I need to search a particular directory and subdirectories for all files beginning with "A," then concatenate all the files contents into one file.
Any guidance as how this can be accomplished?
Peter Yu, 2011/09/11 21:57
Dave - Try the tips at this link: http://superuser.com/questions/119274/grep-beginning-of-file
DavidH, 2013/08/23 03:25
Thank you very much, your syntax for merging CSV columns line by line saved me lot of time .
MIMI, 2015/04/09 08:39
Hi
Thanks for the great understanding of the above topics. But how do i paste column 1 and 2 of file A and column 3 of file B into one output file.
Thanks
Phane, 2015/10/14 10:43
Hi
I have 2 sets of files I want to merge:

file 1 : random text files
file 2 : file containing "full path of file 1:line:column:message to insert"

I want to insert the message just after the line specified in file 2 to file 1, of course all lines will shifted after insertion, and it have to be taken in account.

any idea how to do that ?

ex
file 1
aa
bb
dd


file 2
file 1:2:1:cc
file 1:3:1:ff

on result, file 2 unchanged, file 1 will be
aa
bb
cc
dd
ee


regards,
Phane
harsha, 2015/11/04 14:31
I have a for loop written in shell, at the end of each iteration it generates a string. I want to store all those strings into a single text file. I am using sed to extract that particular line and writing into a text file, but after next iteration it is getting replaced with new string. I want that string to come in the next line.

Thanks in advance
I would love to hear your feedback. Enter your comment below [ Terms of Use ]:
LXTCQ
 

About Peter Yu I am a research and development professional with expertise in the areas of image processing, remote sensing and computer vision. I received BASc and MASc degrees in Systems Design Engineering at the University of Waterloo. My working experience covers industries ranging from district energy to medical imaging to cinematic visual effects. I like to dabble in 3D artwork, I enjoy cycling recreationally and I am interested in sustainable technology. More about me...

Feel free to contact me with any questions about this site at [user]@[host] where [user]=web and [host]=peteryu.ca

Copyright © 1997 - 2017 Peter Yu