0

The setup is the following: I am working with Kaldi tools. They are called in bash scripts of a certain form. Those bash scripts are called by a wrapper program written in python that submits the tasks to a Sun Grid Engine.

The command I want to execute is the following 

feat-to-dim 'ark:copy-feats scp:train.scp ark:- |' -

Executing this in the command line, yields the correct result '40' together with the warning 

WARNING (feat-to-dim[5.0.23-f7b2f]:Close():kaldi-io.cc:501) Pipe copy-feats scp:train.scp ark:- | had nonzero return status 13

However, if I wrap this in the following way:

python -c "import os; os.system(\"feat-to-dim 'ark:copy-feats scp:train.scp ark:- |' -\")"

the program copy-feats fails with the error:

ERROR (copy-feats[5.0.23-f7b2f]:Write():kaldi-matrix.cc:1240) Failed to write matrix to stream

After various stack traces and following errors, the following warning is printed additionally: 

WARNING (feat-to-dim[5.0.23-f7b2f]:Close():kaldi-io.cc:501) Pipe copy-feats scp:train.scp ark:- | had nonzero return status 134

This is what I've found out: feat-to-dim closes the pipe in advance whereas copy-feats tries to continue writing output. Since it is not possible, copy-feats is terminated. 13 probably indicates the broken pipe error.

For Python, this is a serious issue why it turns it into an error and terminates. But in this case, it's not Python which yields this error, but copy-feats. Therefore, stuff like try/catch in python or trapping the pipe seem not to have any success in this case.

Moreover, the following lines work perfectly fine without any warnings or errors:

python -c "import os; os.system(\"copy-feats scp:train.scp ark:-\")" > cp
python -c "import os; os.system(\"feat-to-dim ark:cp -\")"

and the following line yields the simple error message cat: write error: Broken pipe and exit status 256:

python -c "import os; os.system(\"feat-to-dim ark:'cat cp |' -\")"

Do you have any further suggestions here?

Community
  • 1
  • 1
Green绿色
  • 1,620
  • 1
  • 16
  • 43

1 Answers1

0

13 probably indicates the broken pipe error.

13 is an answer (feature dimension), you can use it as a result, you can ignore all other errors like this

with open(os.devnull, "w") as devnull:
    subprocess.call("feat-to-dim 'ark:copy-feats scp:feats.scp ark:- |' -", shell=True, stderr=devnull)

This is what I've found out: feat-to-dim closes the pipe in advance whereas copy-feats tries to continue writing output.

This is a kaldi design, it tries to read just first feature and simply dumps the rest but since the pipe has no mean to terminate writer child, it has to exit in such a bad way. An option is to read full child output in feat-to-dim, but that will be slower.

You can probably open Kaldi bug about that.

Nikolay Shmyrev
  • 24,897
  • 5
  • 43
  • 87
  • Actually, 13 is the return status of `feat-to-dim 'ark:copy-feats scp:train.scp ark:- |' -` executed in the command line. The answer (feature dimension) should be 40. Thank you for your suggestion. It works, but it's hard to integrate into my setup because ignoring all errors by default is not what I want. Is it also possible to catch this broken pipe error somehow? - it would be best if I could handle it in the bash code. – Green绿色 Jan 28 '17 at 15:22
  • No, 13 is a feature dimension. You can use add-deltas instead of copy-feats and then you'll see 39. – Nikolay Shmyrev Jan 28 '17 at 15:29
  • Using `add-deltas` instead of `copy-feats` yields 120 and return status 13 again. – Green绿色 Jan 28 '17 at 16:10
  • Nevertheless, I guess given my setup, I could call python -c with the above code within the bash script which is called by the python program to invoke `feat-to-dim` and suppress the error. That would work fine for me. Thank you! – Green绿色 Jan 28 '17 at 16:15