Waiting subprocess to execute openvpn commands. Sourcing vars issue

Question

I have a problem with waiting sub-process running from within python.I've read tons of info about it, here as well. Sorry to lift this question ones again but still no solution for me.

My code

cmds = "cd /etc/openvpn/easy-rsa && . ./vars && ./clean-all && ./pkitool --initca && ./pkitool --server && ./build-dh"
runCmds = subprocess.Popen(cmds, shell=True)
# run = os.system
# runCmds = run(cmds)
# runCmds.wait()
# runCmds.call()

works perfect, but I need it wait subprocesses ending to run next portion of code. Commented lines not work for me. If I run something from commented I get error

Please source the vars script first (i.e. "source ./vars") .......

One time it seems wait() worked but after some time don't. Method call() runs commands but never ends. Why methods not working for me, especially wait()? I suggest my problem is somewhere in sourcing openvpn vars script in my environment. Please help me!
UPDATED: console log with

set -x

  [25/Jan/2016 18:30:42]"POST /run-step3-process/ HTTP/1.1" 200 49
+ cd /etc/openvpn/easy-rsa
+ . ./vars
+ pwd
+ export EASY_RSA=/etc/openvpn/easy-rsa
+ export OPENSSL=openssl
+ export PKCS11TOOL=pkcs11-tool
+ export GREP=grep
+ /etc/openvpn/easy-rsa/whichopensslcnf /etc/openvpn/easy-rsa
+ export KEY_CONFIG=/etc/openvpn/easy-rsa/openssl-1.0.0.cnf
+ export KEY_DIR=/etc/openvpn/easy-rsa/keys
+ echo NOTE: If you run ./clean-all, I will be doing a rm -rf on /etc/openvpn/easy-rsa/keys
NOTE: If you run ./clean-all, I will be doing a rm -rf on /etc/openvpn/easy-rsa/keys
+ export PKCS11_MODULE_PATH=dummy
+ export PKCS11_PIN=dummy
+ export KEY_SIZE=1024
+ export CA_EXPIRE=3650
+ export KEY_EXPIRE=3650
+ export KEY_COUNTRY=SS
+ export KEY_PROVINCE=FFFFFFF
+ export KEY_CITY=AAAA
+ export KEY_ORG=GGGG
+ export KEY_EMAIL=qq@qq.yy
+ export KEY_EMAIL=mail@host.domain
+ export KEY_CN=ccccccc
+ export KEY_NAME=changeme
+ export KEY_OU=changeme
+ export PKCS11_MODULE_PATH=changeme
+ export PKCS11_PIN=1234
+ ./clean-all
+ ./pkitool --initca
Using CA Common Name: ccccccc
Generating a 1024 bit RSA private key
.................................++++++
............................................++++++
writing new private key to 'ca.key'
-----
+ ./pkitool --server
Using Common Name: ccccccc
Generating a 1024 bit RSA private key
............++++++
...........................++++++
writing new private key to 'ccccccc.key'
-----
Using configuration from /etc/openvpn/easy-rsa/openssl-1.0.0.cnf
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'SS'
stateOrProvinceName   :PRINTABLE:'FFFFFFF'
localityName          :PRINTABLE:'AAAA'
organizationName      :PRINTABLE:'GGGG'
organizationalUnitName:PRINTABLE:'changeme'
commonName            :PRINTABLE:'ccccccc'
name                  :PRINTABLE:'changeme'
emailAddress          :IA5STRING:'mail@host.domain'
Certificate is to be certified until Jan 22 18:30:42 2026 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated
+ ./build-dh
Generating DH parameters, 1024 bit long safe prime, generator 2
This is going to take a long time
.....................................+...+..................................................................................................................................+................+..................................++*++*++*

UPDATE 2 I have researched my code's behaviour:

Before run:
1.apache(or localhost)restart
2.etc/openvpn/easy-rsa/keys clean
3.start browser with new incognito window
I give permission for etc/openvpn/easy-rsa/ as 47777

success == generating process run, new keys create
error == “please source ./vars.....”
CN == variable for server name in vars 
wait() == subrpocess.wait() code following string with bash commands

code ALWAYS work as below:

orig vars -> edit -> wait() -> error
orig vars -> edit (without CN, ./pkitool --server SERVER )-> wait() -> error

orig vars -> NONedit ->wait() -> success

orig vars -> edit ->WITHOUT_wait() -> success
edited vars -> edit ->WITHOUT_wait() -> success
edited vars -> edit(without CN, ./pkitool --server SERVER) -> WITHOUT_wait() -> success
orig vars -> edit(WITH_ CN, ./pkitool --server) -> WITHOUT_wait() -> success
edited vars -> edit(WITH_ CN, ./pkitool --server) -> WITHOUT_wait() -> success

I edit vars in python:

from django.shortcuts import  HttpResponse, HttpRequest
import subprocess
from subprocess import Popen, PIPE
import json
import os.path

def pass3Cmds():
''' run commands on step3  to generate keys and cert in '/etc/openvpn/easy-     rsa/keys'
'''
    cmds = "cd /etc/openvpn/easy-rsa && . ./vars && ./clean-all && ./pkitool --initca && ./pkitool --server && ./build-dh"
    runCmds = subprocess.Popen(cmds, shell=True)


def runStep3Process(request):
    '''collect data from step3 user form and insert 
    them in '/etc/openvpn/easy-rsa/vars'
    '''
    path = '/etc/openvpn/easy-rsa/vars'
    data = json.loads(request.body)

    key_cn = 'export KEY_CN="%s"' % data['key_cn']
    if request.method=='POST' and request.user.is_authenticated():
        with open(path) as varsfile:
            data = varsfile.readlines()
        try:
            data[69] = key_cn +'\n' 
            with open(path, 'w') as newvarsfile:
                newvarsfile.writelines(data)
                pass3Cmds()
                pem = '/etc/openvpn/easy-rsa/keys/dh1024.pem'
                if os.path.exists(pem):
                    return HttpResponse(successMsg2)
                return HttpResponse(dangerMsg)
        except IndexError:
            return HttpResponse(warnMsg2)
     return HttpResponse(warnMsg)

Ones again: code perfectly works with this way edited vars until I want run any code to wait subprocess. If I run e.g. subprocess.wait() I get "please source ./vars" error

Question is: why editing vars in my case force error?

Your code certainly *does* source `vars` inside the relevant shell. Are you sure you've actually edited the vars script per documentation? — Charles Duffy, Jan 25 '16 at 17:39
BTW, I'd strongly suggest redirecting `stdin` from `/dev/null`. Otherwise, openssl is potentially going to be trying to prompt from the stdin file descriptor it inherits from the parent (Python) process. — Charles Duffy, Jan 25 '16 at 17:41
...to ask it a different way: Do you get the same behavior when you run the same command interactively, without Python involved? — Charles Duffy, Jan 25 '16 at 17:43
when I run root@localhost:/etc/openvpn/easy-rsa# bash -x . ./vars it print .: .: is a directory — Andriy, Jan 25 '16 at 18:16
No, not like that. `cmds = "set -x; cd /etc/openvpn/easy-rsa && . ./vars && ./clean-all && ./pkitool --initca && ./pkitool --server && ./build-dh"` — Charles Duffy, Jan 25 '16 at 18:24
...or, if you really want to do it as a one-off: `bash -x -c '. ./vars'` — Charles Duffy, Jan 25 '16 at 18:24
If you aren't in a position to track stderr, by the way, consider: `cmds = "exec 2>/tmp/easy-rsa.log; set -x; cd /etc/openvpn/easy-rsa && . ./vars && ./clean-all && ./pkitool --initca && ./pkitool --server && ./build-dh"` — Charles Duffy, Jan 25 '16 at 18:27
As updated, there's no `Please source the vars script first` error message involved. — Charles Duffy, Jan 25 '16 at 18:39
Timing out in `build-dh` probably just means you don't have enough entropy in the pool. — Charles Duffy, Jan 25 '16 at 18:39
Consider installing and using `rngd`, from the `rng-tools` package, to leverage any hardware random number generator your system may have available. — Charles Duffy, Jan 25 '16 at 18:40
yes , "Please source the vars script first" not involved when I run two shown commands but if only i try force wait in any way it ever give this error — Andriy, Jan 25 '16 at 18:43
"it" is not useful in this context. Use the `set -x` log to determine *exactly* where the error comes from. If the error is on stdout rather than stderr, you may need to make the `exec` line `exec >/tmp/easy_rsa.log 2>&1`. — Charles Duffy, Jan 25 '16 at 19:30
But seems no error in commands itself. Claim **Please source the vars script first** occurs just if I try run subprocess.wait() for example. Why it influence . ./vars? What wait() does to "disturb" vars? — Andriy, Jan 25 '16 at 19:47
It doesn't "disturb" vars at all. Something else is going on, and figuring out what that something else is a debugging exercise in discovering root cause. — Charles Duffy, Jan 25 '16 at 19:48
BTW, your code doesn't run at all with easy-rsa 3, and I have no interest in installing an ancient version to be able to test it. — Charles Duffy, Jan 25 '16 at 19:53
...well, I say that, but then I *did* install an ancient version, and the only bug I could reproduce in the code given had nothing at all to do with `vars`. — Charles Duffy, Jan 25 '16 at 20:00
...edit: May actually have something to do with vars, if you're setting a CN there, but that's a reproduction step not given in the question. A question needs to contain a **complete** reproducer to be answerable. — Charles Duffy, Jan 25 '16 at 20:26
This is an improvement, but it's still not a MCVE: It's not **complete** unless it can be run by someone without no other setup, but the `request.body` reference refers to code not provided. Better to just hardcode these values. — Charles Duffy, Jan 26 '16 at 21:18
Also, this code is fragile for no good reason: You can just set the environment variables directly with the `env=` argument to `subprocess.Popen` and not bother with the `vars` file at all -- neither reading it or rewriting it. — Charles Duffy, Jan 26 '16 at 21:20
Thanks for your comments!Could you give an example of using *env=* argument in my case? — Andriy, Jan 26 '16 at 21:25
Would it be too much for you to figure out things on your own for once? But... `env={'KEY_OU': 'changeme', 'KEY_CN': 'changeme', 'KEY_DIR': '/etc/openvpn/easy-rsa', 'KEY_CONFIG': '/etc/openvpn/easy-rsa/openssl-1.0.0.cnf', ...}` -- the `KEY_DIR` and `KEY_CONFIG` variables are the ones it looks for to determine whether to display the message saying to source `vars`, so if you set them yourself it's not going to happen. — Charles Duffy, Jan 26 '16 at 21:30
As far as I understand I can place these arguments as dic to `subprocess.Popen ` to avoid editing vars? Which exactly these ones can be? I see some variables from vars and `KEY_DIR` and `KEY_CONFIG` from pki-tool... — Andriy, Jan 26 '16 at 21:42
Yes -- see the documentation for the subprocess module. As for what the values can be, they're exactly the same as the values set by the `vars` script. — Charles Duffy, Jan 26 '16 at 21:43
Only variables I need to edit vars? No `KEY_DIR` and other stuff? — Andriy, Jan 26 '16 at 21:45
If you aren't sourcing vars -- and I suggest that you don't -- then you need to include `KEY_DIR`, `KEY_CONFIG`, and anything else you *would* have gotten by sourcing vars. — Charles Duffy, Jan 26 '16 at 21:46
So in that case I don't need source vars?it is possible to avoid it? It would be good ! — Andriy, Jan 26 '16 at 21:49
Of **course** it's possible. If you look at the code that's throwing the error message (and if you want to avoid an error message, it's always good to look at exactly how the conditional that triggers it is implemented), it's looking for variables inherited through the environment. Sourcing a file doesn't have any monopoly as the sole way to set environment variables. — Charles Duffy, Jan 26 '16 at 21:52
I'd suggest spending more time figuring out how and why things work before trying to make them work differently. If you'd looked at *why* you were getting the `vars` prompt (asked yourself the question, for instance, "how does it know that `vars` hasn't been sourced?"), then much of the fumbling around might have been avoided. — Charles Duffy, Jan 26 '16 at 22:02
Sorry but I didn't catch what do you mean: "how does it know that vars hasn't been sourced?" — Andriy, Jan 26 '16 at 22:11
Just that. The error message you're getting, "Please source the vars script first" -- did you consider looking into exactly under what circumstances that error message is printed, ie. how the script decides whether or not to exit with that error? — Charles Duffy, Jan 26 '16 at 22:12
Sourcing as far as I know run commands inside the same process and when i add new string of code ,i.e. Popen.wait() , it runs other process that interrupt previous one? — Andriy, Jan 26 '16 at 22:16
Yes, sourcing runs commands inside the same process, but if it goes on to run other commands, you know that it already finished that step. But here's the important part: The other commands, like `clean-all` or `pkitool --init-ca`, are *children* of that shell, not part of it -- and they were created by executing separate scripts, not as forks of the parent. There are only so many ways to communicate state (ie. whether something like `vars` was run) from a parent process to a child across an `execve()` boundary, and all those are things you can do from Python too. — Charles Duffy, Jan 26 '16 at 22:24
...so, this is part of why it's important to actually understand the fundamentals of how UNIX works, rather than guessing: If you know that there are only a limited number of ways for state to be communicated, then it's a natural idea to convey that state a different way. If you know that `source` blocks until it's completed, you won't come up with outlandish guesses like having that start a separate process that can somehow be interrupted. Etc. — Charles Duffy, Jan 26 '16 at 22:25
...btw, need I point out that you **still** haven't provided a standalone reproducer for the bug in question -- that is, code someone else can copy-and-paste from your question to see the same behavior without any prior setup required? — Charles Duffy, Jan 26 '16 at 22:27
Nuh-uh. Read http://stackoverflow.com/help/mcve and/or http://sscce.org/ -- "full view.py code" is not a standalone reproducer. — Charles Duffy, Jan 26 '16 at 22:57
This is closer, but still doesn't run standalone; for instance, it uses but doesn't import the `json` module -- which, if you were testing that it would run in a raw Python interpreter, you should have caught yourself. Test that the code in your question can be run by someone just copying-and-pasting with no changes. And on that point -- consider including a copy-and-pasteable shell script to check out a relevant copy of easy-rsa under /tmp rather than requiring folks to have code installed in /etc. — Charles Duffy, Jan 26 '16 at 23:17
similarly, it's also not standalone because it requires but doesn't provide a `request` dictionary. — Charles Duffy, Jan 26 '16 at 23:17

Charles Duffy · Answer 1 · 2016-01-25T19:56:47.243

1

Your log shows the process waiting indefinitely in ./build-dh.

This is, as the associated log message indicates, a slow process. You shouldn't expect it to be instant.
However, if your system's entropy pool is low, it can be not merely slow but indefinite. Consider using rngd from the rng-tools package to use your system's hardware random number generator (presuming its CPU provides one) to populate the kernel's entropy pool.

Beyond that, your usage is incorrect:

./pkitool --server

...is not a valid command: You need to provide a CN to generate a server certificate, for instance:

./pkitool --server server

edited Jan 25 '16 at 19:56

answered Jan 25 '16 at 18:42

Charles Duffy

280,126
43
390
441

I try exclude ./build-dh command from common command string and add runCmds.wait() in sequence but error about source vars immediately occurs – Andriy Jan 25 '16 at 18:53
See update -- actually did do a checkout of easy-rsa 2.0 and run your exact code. Only reproduced one bug, and it had nothing to do with `vars`. – Charles Duffy Jan 25 '16 at 20:01
./pkitool --server by default pick up CN value from vars that I previously edited so no need to run --server server thus in etc/openvpn/easy-rsa/keys i have e.g. 1111.crt and so on – Andriy Jan 25 '16 at 20:21
Not in the version I'm running it doesn't. – Charles Duffy Jan 25 '16 at 20:23
I use 2.0 and it does. Seems you are right as for point 2. in your answer. Every time I run ./build-dh it lasts different time and sometimes hang up. Could you give some more details for point 2? – Andriy Jan 25 '16 at 20:24
*shrug*. Still, your bug is a CNR. Check out easy-rsa 2.0 from git, use your exact code, and it doesn't reproduce. Until you edit your question to provide a reproducer tested to work straight from a git checkout, it's a waste of time for anyone else to try to debug. – Charles Duffy Jan 25 '16 at 20:24
OK, perhaps, but I can edit to run command >./pkitool --server server without changes but it for sure will not solve the problem of "Please source the vars script first"... – Andriy Jan 25 '16 at 20:34
Doesn't matter. Until you can give reproduction steps that let me or anyone else get the "Please source the vars script first" bug, we can't help you with it. – Charles Duffy Jan 25 '16 at 20:36
Start from scratch: Check easy-rsa out from git, switch to the 2.0 branch, and record every single step needed to get the "Please source the vars script first" question. And next time, **test that your code samples in your questions actually are complete enough to produce your bug starting from scratch**. – Charles Duffy Jan 25 '16 at 20:36
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/101726/discussion-between-andriy-kravchenko-and-charles-duffy). – Andriy Jan 26 '16 at 21:31

score 0 · Answer 2 · answered Jan 25 '16 at 14:01

0

I don't know "/etc/openvpn/easy-rsa" but if it establishes a vpn connection, the program will hang while the connection remains. In order to terminate it, you can use a timeout param to force a timeout on it. Use of subprocess.call.

cmds = "cd /etc/openvpn/easy-rsa && . ./vars && ./clean-all && ./pkitool --initca && ./pkitool --server && ./build-dh"
timeout = 60 * 10 # 10 minutes

try:
   ret_code = subprocess.call(cmds, shell=True, timeout=timeout)
   # return_code
except subprocess.TimeoutExpired: 
   # do_something()

call, check_call and check_output accept a timeout parameter.

answered Jan 25 '16 at 14:01

Ali SAID OMAR

6,404
8
39
56

I get 'module' object has no attribute 'TimeoutExpired' error – Andriy Jan 25 '16 at 14:24
According to the doc https://docs.python.org/3/library/subprocess.html#subprocess.TimeoutExpired, New in version 3.3. – Ali SAID OMAR Jan 25 '16 at 14:34
if I run "ret_code = subprocess.call(cmds, shell=True, timeout=timeout)" without try/exept i receive __init__() got an unexpected keyword argument 'timeout' – Andriy Jan 25 '16 at 14:37
http://stackoverflow.com/questions/1191374/using-module-subprocess-with-timeout, in fact timeout is a python3 support. – Ali SAID OMAR Jan 25 '16 at 14:38
No, it doesn't establish a VPN connection; the given commands generate a SSL CA and a server certificate, and then exit. Thus, the premise of this answer is incorrect, and there should be no need at all for a `timeout` argument. – Charles Duffy Jan 25 '16 at 17:40

score -1 · Answer 3 · answered Jan 25 '16 at 12:32

-1

Import the time module and use time.sleep(amount_of_seconds).

answered Jan 25 '16 at 12:32

L_Pav

281
1
3
11

I try time.sleep, it makes pause but I get the same error about sourcing vars again... – Andriy Jan 25 '16 at 12:44
Sleeping for a preset time isn't an acceptable solutions for this anyhow. The amount of time these commands take will vary wildly based on the amount of entropy in the system's RNG pool, so the time to wait can't be reliably predicted. – Charles Duffy Jan 25 '16 at 17:42

Waiting subprocess to execute openvpn commands. Sourcing vars issue

3 Answers3