python - Need for Performance in bash script -


i have 50000 files , each 1 has 10000 lines. each line in form:

value_1 (tab) value_2 (tab) ... value_n 

i wanted remove specific values every line in every file (i used cut remove values 14-17) , write results new file.

for doing in 1 file, wrote code:

file=nameoffile newfile=$file".new" i=0  while read line     let i=i+1     echo line: $i     a=$i"p"     linefirstpart=$(sed -n -e $a $file | cut -f 1-13)     #echo linefirstpart: $linefirstpart     linesecondpart=$(sed -n -e $a $file | cut -f 18-)     #echo linesecondpart: $linesecondpart     newline=$linefirstpart$linesecondpart     echo $newline >> $newfile done < $file 

this takes ~45 secs 1 file, means take about: 45x50000 = 625h ~= 26 days!

well, i need faster, e.g. solution cats whole file, applies 2 cut commands simultaneusly or guess.

also solutions in python accepted + appreciated bash scripting preferable!

the entire while loop can replaced 1 line:

cut -f1-13,18- $file > $newfile 

Comments

Popular posts from this blog

commonjs - How to write a typescript definition file for a node module that exports a function? -

openid - Okta: Failed to get authorization code through API call -

thorough guide for profiling racket code -