Python wait until the os.system end

Question

I need to run an external script (is not possible modify this) in order to manag a CLOB data and insert in table. But if I use the os.system in a loop, only 1 record is inserted in the table, if I add a time.sleep(20) between call, all works fine.

I try to use a subprocess, but the instruction is very big and the command return Errno 36 filename too long.

In this case only the last record has been inserted:

for element in list:
   query = 'INSERT IN TABLE1 VALUES(<clob>)'
   rc = os.system('externalScript.py -q "' + query.encode('utf-8') + '"')

In this case the subprocess return filename too long:

for element in list:
   query = 'INSERT IN TABLE1 VALUES(<clob>)'
   proc = subprocess.Popen(['python', 'externalScript.py -q "' +  query.encode('utf-8') + '"' ], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

Is there a solution without adding time.sleep(20) after os.system?

Would you be able to `import externalScript` and use it without calling subprocess? — ex4, Mar 04 '21 at 10:51
`os.system()` by definition already waits until the external process has finished. — tripleee, Mar 04 '21 at 11:04
The correct syntax would be `subprocess.Popen(['python', 'externalScript.py', '-q', query], text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)` but really don't use `Popen()` here; switch to `subprocess.check_call` or perhaps `subprocess.call` or `subprocess.run(..., check=True)` ... except of course running Python as a subprocess of itself is usually also the wrong thing to do. Just `import externalScript` (maybe you'll need to refactor it) and call its main function directly from the current script. — tripleee, Mar 04 '21 at 11:08
The problem is the same with a subprocess.call : "[Errno 36] File name too long" — SirLancillotto, Mar 04 '21 at 11:17
I am no able to import externalScript, is a 3PP script and I can use only with a call — SirLancillotto, Mar 04 '21 at 11:17
Just switching to `subprocess.call` fails because you still have a syntax error which needs to be fixed. If your argument is long enough to trigger that warning, it would probably be better if you could make the script read a file, or standard input. — tripleee, Mar 04 '21 at 11:20

score 1 · Answer 1 · answered Mar 04 '21 at 11:09

For your subprocess.Popen call (which, as others have noted, should be a subprocess.call), you should be passing the script name and the arguments separately and without the other quotes, like this:

for element in list:
   query='INSERT IN TABLE1 VALUES(<clob>)'
   proc = subprocess.call(['python', 'externalScript.py', '-q',  query.encode('utf-8')], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

However, this probably won't actually be an improvement over os.system; both should wait until the external script has finished. Something is wrong either in that external script, or in whatever database it connects to, and it can't be called in quick succession.

score 0 · Answer 2 · answered Mar 04 '21 at 10:54

0

subprocess.Popen is nonblocking call. You need to use subprocess.call which is blocking, meaning it will block processing code until the call is finished.

Excellent post about the topic:

Blocking and Non Blocking subprocess calls

answered Mar 04 '21 at 10:54

ex4

2,289
1
14
21

2

The weird thing is that `os.system` is also blocking, so it should also wait until the external script has finished. Something is wrong somewhere else... – Jiří Baum Mar 04 '21 at 11:05
@sabik how to wait untile os.system finished? Because the loop working only if I add a time.sleep(x), but I don't like that... – SirLancillotto Mar 04 '21 at 11:31
1

It has been pointed out to you in several comments already: `os.system()` already does exactly that. If the script you run in a subprocess continues to run something in a background sub-subprocess after it finishes and reports back to Python, there is no simple way to know when it's _really_ done. As far as the question is at all well-defined, the subprocess you explicitly invoke has finished when `os.system()` returns. – tripleee Mar 04 '21 at 11:49

Python wait until the os.system end

2 Answers2