Searching and Filtering

Searching In Files

grep

print lines matching a pattern

pattern is expressed must be an exact text chain or a regular expression describing text in a formalised syntax that consists in a sequence of characters that define a search pattern.

• -i (--ignore-case) Ignore case distinctions in both the PATTERN and the input files.
• -v (--invert-match) Invert the sense of matching, to select non-matching lines.
• -c (--count) Do not output matching lines just print number of matches.
• -r (--recursive) Read all files under each directory, recursively, following symbolic links only if they are on the command line.
• -l (--files-with-matches) Do not output matching lines just print the names of files that do contain matching lines
• -L (--files-without-match) Do not output matching lines just print the names of files that do NOT contain matching lines
• -n (--line-number) Prefix each line of output with the 1-based line number within its input file.
• more options described in man grep

FASTA file format

>gi|5524211|gb|AAD44166.1| cytochrome b [Elephas maximus maximus]
LCLYTHIGRNIYYGSYLYSETWNTGIMLLLITMATAFMGYVLPWGQMSFWGATVITNLFSAIPYIGTNLV
EWIWGGFSVDKATLNRFFAFHFILPFTMVALAGVHLTFLHETGSNNPLGLTSDSDKIPFHPYYTIKDFLG
LLILILLLLLLALLSPDMLGDPDNHMPADPLNTPLHIKPEWYFLFAYAILRSVPNKLGGVLALFLSIVIL
GLMPFLHTSKHRSMMLRPLSQALFWTLTMDLLTLTWIGSQPVEYPYTIIGQMASILYFSIILAFLPIAGX
IENY

FASTQ file format

@SEQ_ID
GATTTGGGGTTCAAAGCAGTATCGATCAAATAGTAAATCCATTTGTTCAACTCACAGTTT
+
!''*((((***+))%%%++)(%%%%).1***-+*''))**55CCF>>>>>>CCCCCCC65

Exercises

• count the number of sequences in the file 'abcd.fa' (a multiple fasta file)
• can we have the same approach with a fastq file?

grep with Regular Expressions

• grep -E or egrep

• Regular Expressions: sequences of characters to express a pattern

• useful metacharacters:

  • . any character

  • [] bracket expressions: defines a set of characters

    • [abc] : matches character "a" or "b" or "c"
    • [a-z] : matches on character in the range from "a" to "z", ie "a" or "b" ... or "z"
    • [:alnum:] : matches any letter or numberic characters
    • [:alpha:] : matches any letter
    • [:digit:] : matches any number
    • [:lower:] : matches any letter in range [a-z]
    • [:upper:] : matches any letter in range [A-Z]
    • [:punct:] : matches any punctuation character
    • [:space:] : matches any space character (space, tab, ...)

  • Anchoring.

    • ^ represent start of the string.
    • $ represent end of the string.

  • Repetition.

    • ? match at most once the preceding element. ie A? represent zero or 1 A => "" or "A"
    • + match one or more times the preceding element. ie A+ represent 1 A to any number of successive A => "A" or "AA" or "AAA" etc ...
    • * match zero or more times the preceding element. ie a* represent 0 to any number of successive A => "" or "A" or "AA" or "AAA" etc ...
    • {n} match exactly n times the preceding element. ie A{5} represent exactly 5 successives A => "AAAAA"
    • {m,n} match m to n times the preceding element. ie A{2,5} represent from 2 to 5 successives A => "AA" or "AAA" or "AAAA" or "AAAAA"

  • Logical OR.

    • (one|two) match the strings "one" or "two"

  • more information on regular expression in man grep

sed

Regular expression vs shell special characters.

warning

• remember some characters have a special meaning for the shell, eg >, >>, 2>, *, ?, |, ...
• remember shell evaluation is executed before command execution

example: what do you expect in the following wrong examples:

• grep > file
• grep * file

regular expression and shell share some metacharacters that do not have the same meaning, when you want to search for those characters you have to protect them.

hint: always protect your search pattern by enclosing it in double quotes "

Exercises

• find the names of all the genes in the zipped FASTA file abcd.fa.gz in the unix_training project directory of central-bio.
• in the file abcd_sequences.gb, in the unix_training project directory of central-bio, find the origin organism of all the sequence entries (tip: it's on a line with the ORGANISM keyword).

Filtering/sorting results

cut

• cut - remove columns (sections) from each line of files
• cut OPTION... [FILE]
• -d DELIM use DELIM instead of TAB for field delimiter
• -f select only these fields

who | cut -d ' ' -f 1
find . -atime 0 | cut -d '/' -f 2,3

sort

• sort - sort lines of text files
• sort [OPTION]... [FILE]
• -k --key=KEYDEF sort via a key, KEYDEF gives location and type.
• -n (--numeric-sort) compare according to string numerical value (OBSOLETE).
• -r (--reverse) reverse the result of comparisons.

who | cut -d ' ' -f 1 | sort | uniq -c | sort -k 1n,2

Exercise

We ran a blast with -m8 output. So the following fields are displayed

Exercise (continued)

separated by tab.

1. copy the file blast2_m8.txt in the project of the course in your home.
2. sort the output following the % of identity (the highest identity to the top)
3. display only columns id of hit, percent identity, Expect value, HSP bit score
4. store the results in a new file.

uniq

report or omit repeated lines (Filter adjacent matching lines)

• uniq [OPTION] INPUT
• -c prefix lines by the number of occurrences.

Exercise

• eliminate duplicates from the list of organisms gathered from abcd_sequences.gb.
• from the same blast output than previous exercise, display the sequence ids of the match.