4

I have a python script main.py that needs to start the scrapy executable, so I use Popen to do that (using subprocess.call() has same result). To simplify things I just run the help argument of scrapy.

import subprocess
...
p = subprocess.Popen(['scrapy', '-h'])

The script needs to be run inside a vitualenv which has scrapy installed on it. When I activate the virtualenv and run the script as python main.py the command scrapy -h is executed as expected.

Now, I need this script to be run as a systemd service. I have created the systemd unit for the script as follows:

[Unit]
Description=Scrapy test
After=network.target

[Service]
User={{ scrapy_user }}
WorkingDirectory={{ test_path }}
ExecStart={{ virtualenv_path }}/bin/python {{ test_path }}/main.py

[Install]
WantedBy=multi-user.target

When I enable and start the service with sudo systemctl start scrapy-test the code runs normally until p = subprocess.Popen(['scrapy', '-h']) when I get an exception raised

File "{{ test_path }}/main.py", line 52, in launch_spider
   p = subprocess.Popen(['scrapy', '-h'])
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
   errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
   raise child_exception
OSError: [Errno 2] No such file or directory

I have also tried to change the working directory by adding os.chdir('{{ virtualenv_path }}/bin') just before Popen, but I get the exact same exception. I am sure the script is run through the virtualenv because (1) it is set so in the ExecStart attribute and (2) the script would raise exceptions earlier when importing modules that are not installed in the system's python.

Gallaecio
  • 3,620
  • 2
  • 25
  • 64
Mr Alexander
  • 339
  • 1
  • 5
  • 13
  • I have the same problem, but in my case your solution does not work, as it is a third-party lib (pygatt) which calls hcitool or gatttool. – Arturo Ribes Jan 21 '18 at 19:33
  • Could you move your answer from an edit in your question to a proper, accepted response? – Gallaecio Jan 22 '19 at 13:25

3 Answers3

1

You must specify the current working directory on your call to Popen to ensure that scrapy is executed from the right directory.

Gallaecio
  • 3,620
  • 2
  • 25
  • 64
0

The clue is here:

ExecStart={{ virtualenv_path }}/bin/python {{ test_path }}/main.py

The scrapy command is on $VIRTUAL_ENV/bin, which is not on the systemd $PATH.

You could try:

import os
import subprocess
p = subprocess.Popen(['%s/bin/scrapy' % os.environ['VIRTUAL_ENV'], '-h'])
Apalala
  • 9,017
  • 3
  • 30
  • 48
0

Another issue you may encounter later, you may want to make python to wait for the end of subprocess run with p.wait(): 

import os
import subprocess
p = subprocess.Popen(['%s/bin/scrapy' % os.environ['VIRTUAL_ENV'], '-h'])
p.wait()

Otherwise systemctl may terminate your processes too early.

AI Mechanic
  • 631
  • 7
  • 8