0

I'm using a supercomputer which is using the famous #PBS. My walltime is 48 hours and this is not enough to process million files.

My file names are like :

AAAAAA.pdb 
DAAAAA.pdb 
EAAAAA.pdb 
FAAAAA.pdb 
...

All possibles letters are "A D E F G H I K L M N P Q R S T V W Y".

I want to use a script like this :

for file in /dir/*
do
  cmd [option] $file >> results.out
done

But I must use a restart for the wall time. With numbers I would have put a counter but with specific letters I don't know how to write the last file to start from this checkpoink. Like :

if [ -f next.seq ]; then
  seq=`cat next.seq`
else
...
    for file in /dir/seq
    do
      cmd [option] $file >> results.out
    done
...
let seq=seq+1
echo $seq > next.seq
Grego
  • 59
  • 2
  • 9
  • What is the question? Also, what you did write is not clear. Please write in complete sentences using more words saying what you are doing to what. – philipxy Nov 04 '15 at 06:19
  • The question is : I need a way to restart the job I send to the supercomputer. The way I'm trying is to write the file "next.seq" after a file has been processed. Then the loop can restart at this point. I don't know if its the best way and if it can work. – Grego Nov 04 '15 at 06:22
  • Something like this : `startfile=$1 for FILE in ./sedpdb/*; do if ! [ "$FILE" '=' "$next.seq" ] ; then perl -pi -e 's/"*.ext1"/"&.ext1"/g' ./psfgen.inp ; perl -pi -e 's/"*.ext2"/"&.ext2"/g' ./psfgen.inp ; let seq="$FILE+1" echo $seq > next.seq else echo Skipping $FILE fi done` – Grego Nov 04 '15 at 16:00
  • Please edit your question to be clear and complete rather than putting fragments into comments. – philipxy Nov 04 '15 at 18:18
  • If you can choose the order of creation, create them so that your last one is the largest or smallest alphabetically. Then you can restart on ls | sort | head. [Beware of limits](http://stackoverflow.com/a/466596/3404097): your 50M files are more than fit in a linux directory. – philipxy Nov 04 '15 at 18:29

0 Answers0