Tips & Tricks

Following are some of the commands (useful in linux terminals) that I often use:

Using awk

• • Delete all the empty lines in a file: awk 'NF' filename

  • Transform rows to a single column: awk '{OFS=RS;$1=$1}1' filename

  • Search only from the 1st column: awk '{if ($1 == "tot") print $0;}' filename

  • Print between two values from a column using variables: awk -v amin=$amin -v amax=$amax ' $1 <= amin || $1 >= amax' filename

  • Print every 10th line: awk 'NR%10==0' filename

  • Print everything a file serially: awk '{OFS=RS;$1=$1}1' filename

  • Print every line multiple (N) times: awk '{for(i=0;i<N;i++)print}' filename

  • Count the number of columns in a file: awk '{ print NF}' filename

  • Print up to 10 decimal place: awk '{printf "%.10f\n", $1} filename

  • Print unique values in column one: awk '!a[$1]++'  filename

  • Print the last column of a file: awk '{print $(NF)}'  filename

  • Logical And commands in awk: awk '{ if( $2>50 && $3>50 ) print $1 }' filename

  • Logical Or commands awk: awk '{ if( $2<50 || $3<50 || $4<50 ) print $1 }' filename

  • Control the number of decimal places to print: awk '{printf "%s %s %.3f %.3e\n", $1, $2, $3, $4}' filename

  • Print the hopping elements of the 0 0 0 unit cell in Wannier: awk '{if ($1 == "0" && $2 == "0" && $3 == "0" ) print $0;}' wannier90_hr.dat 

Using sed

• • Print the 5th line using sed: sed -n "5p" filename

  • Print everything after the match: sed -ne '/pattern/,$ p' filename

  • Delete everything after a particular pattern: sed -i '/particular_pattern/,$d' filename ( -i means it will directly change the file)

  • Delete line number 33: sed -i '33d' filename

  • Insert a string at a specific line number (33): sed -i '33i This is the new line' filename

  • Delete line which contains a particular string: sed -i '/pattern to match/d' filename

  • Print between two line numbers: sed -n '1,134p' filename

  • Use variable inside sed: sed -n "${test1},${test2}p" filename (Always use "  ")

  • Remove the last character from a string: sed 's/.$//' string

  • Grep between two specific strings: sed -n '/test1/,/test2/p' filename

  • Replace one string with another: sed -i "s|EFIELD=0|EFIELD=$i|g" ./INCAR

  • Add all the values in a row: sed 's/ /+/g' |bc filename

  • Add all the values in a column: paste -sd+ | bc filename

  • Replace a special character "*": sed -i 's|\*|0|g' GapPlane.dat

  • Remove spaces from a string: tr -d ' ' filename

  • Remove 5 lines after a pattern with the line that contains the pattern: sed -e '/pattern/,+5d' file.txt

  • Remove all the lines between two matching strings (ELECTRONS): sed -e '/ELECTRONS/,/\//d' scf.in

  • Remove any character from a string : sed 's/[][\/$*.^|@#{}~&()_:;%+"='\'',`><?!-]/ /g'  filename   

  • Remove all white spaces: sed -r 's/\s+//g' filename 

Using grep

• • Use grep in a negative sense: grep -v filename 

  • Count the number of occurrences in grep: grep -c string filename 

  • Using if

    • To ignore a specific loop: continue

    • Check if a regular file is missing  in a directory:
      
      if [ ! -f /tmp/foo.txt ]; then
      echo "File not found!"
      fi

    • Compare floating points using bash:
      
      if (( $(echo "$num1 > $num2" |bc -l) )); then
      Do something
      fi

Using cut

• Cut a string after a character: echo $var | cut -f2 -d":"

• Cut a string before a character: echo $var | cut -f1 -d":"

• Cut a string between two brackets: echo $var | cut -d "(" -f2 | cut -d ")" -f1

• Print everything except the first column: echo $var | cut -d ' ' -f2 proj.1.temp

Miscellaneous

• Transform many columns to a single row: paste -sd" " filename (any text file)

• Change text colors in Linux terminal: tput setaf 3

• Unset the color (tput) in Linux terminal: tput sgr0

• Print Hello world 10 times in termianl: printf 'Hello World\n%.0s' {1..10}

• Make the background of a .png transparent: convert file.png -transparent white your.png

• Compress a file using tar: tar -zcvf xyz.tar.gz xyz

• Inflate a file using tar: tar -zxvf xyz.tar.gz 

• Sort according to the absolute value of the 4th column: sort -k4 -g filename 

• Recursive sorting according to two columns: sort -k4 -k3 filename 

• Gnuplot: Plot multiple files inside a directory at once

  • FILES = system("ls -1 *.dat")
    plot for [data in FILES] data u 1:4 w lp pt 1 lw 1

• Generate sequences like 01,02,03....: seq -w 1 10 or echo {01..10}

• Find any files created between two dates: find . -type f -newermt 2012-11-06 ! -newermt 2014-05-03

• Find a file: find . -name

• Remove hidden elements from a file: sed -i $'s/[^[:print:]\t]//g' filename [can be useful in rare cases for files created using windows and used in linux]

Using python

• • Under preparation.....

Acknowledgements

• Various websites including https://stackoverflow.com, https://stackexchange.com/ etc.

• I am grateful to my friend Suhas Nahas for introducing me to the world of scripting and automation, which has saved me countless working hours!