Capture Tensorflow output in bash script

Question

The training session of a Tensorflow estimator with train and evaluate trainspec gets occasionally killed.

I would like to resume the training session upon reading the output "Killed" (generated by tf.logging.INFO). Ideally by executing the python script again and again. Is there a short way to accomplish this?

duplicate: https://stackoverflow.com/questions/11162406/open-and-write-data-to-text-file-using-bash-shell-scripting `python script.py > output.txt` — keithpjolley, Sep 06 '18 at 14:10
Possible duplicate of [Open and write data to text file using bash/shell scripting](https://stackoverflow.com/questions/11162406/open-and-write-data-to-text-file-using-bash-shell-scripting) — Albin Paul, Sep 06 '18 at 14:15

score 0 · Answer 1 · answered Sep 06 '18 at 14:49

Not too much experience about it, but according to my limited knowledge, you can turn to the use of pipe in Linux. Like this,

tail -f xxx.log | grep --line-buffered killed_information | while read msg ; do python train.py ; done

Note: killed_information should be replaced by actual error outputs of the train.py

score 0 · Accepted Answer · answered Sep 07 '18 at 08:06

0

while [ 1 ]; do

    if grep -Fxq "killed" logFile; then
       # code if found (Run your script again from here)
    fi

    #check every 5 minutes
    sleep 300

done

(Code adopted from https://stackoverflow.com/a/4749368/10008499 )

answered Sep 07 '18 at 08:06

xxx374562

226
1
8

The `[ 1 ]` happens to work but doesn't do what you think. You are probably looking for `while true; do` – tripleee Sep 10 '18 at 04:28
@tripleee yeah, true, but 1 is faster to type than true :) – xxx374562 Sep 10 '18 at 04:33
On my keyboard, `[ 1 ]` is significantly more cumbersome to type than `true` and one character longer. – tripleee Sep 10 '18 at 04:35

Capture Tensorflow output in bash script

2 Answers2