'cleartool find -print' inside Python3 'subprocess.check_output' returns empty string

Question

I am trying to run the following command using Python subprocess

/usr/atria/bin/cleartool find <path> -follow -name '*.elf' -version "lbtype(CXC1111-111)" -print

Code snippet:

cmd = [clt, 'find', path, '-follow', '-name', '\"*.elf\"', '-version', lb, "-print"]
try:         
    output = subprocess.check_output(cmd)

    print("--%s--" % output)
    print("===DONE===")
except subprocess.CalledProcessError as e:
    print("CALLEDPROCESSERROR")
    print(e)
except subprocess.TimeoutExpired as e:
    print(e)

o/p after running the snippet

--b''--

===DONE===

When the cleartool find command is executed on shell, the filename is printed

Ideally in python script output should print the filename. Any ideas where this is going wrong?

score 2 · Accepted Answer · answered Jul 11 '13 at 08:22

By default, subprocess.Popen has shell=False. The check_output function passes its arguments along to Popen, so you're getting shell=False here too. Without invoking the shell, each argument is passed uninterpreted to cleartool. When you run, from the shell, the command:

cleartool find ... -name '*.elf' -version "lbtype(CXC1111-111)" ...

(bits snipped to make this fit better in the window), the single and double quotes are stripped by the shell, so that cleartool just gets *.elf and lbtype(CXC1111-111). Those are the byte-sequences you need to pass with shell=False. (Keeping shell=False is probably best; if you set it to True you'll have to paste the command up into a single string and quote shell metacharacters.)

Your explanation on cleartool and shell makes me clear. Thanks for the detailed explanation. — Hemanth, Jul 11 '13 at 11:55

score 1 · Answer 2 · answered Jul 11 '13 at 08:10

I think i figured out the problem.

Before solution, here is how my lb and cmd looks

lb = '\"lbtype(%s-%s)\"' % (tmp_prod_no, rev)
cmd = [clt, 'find', lm_path, '-follow', '-name', '\"*.elf\"', '-version', lb, "-print"]

\" tags are the culprit for the problem After following modifications (lb and *.elf), it works fine

lb = 'lbtype(%s-%s)' % (tmp_prod_no, rev)
cmd = [clt, 'find', lm_path, '-follow', '-name', '*.elf', '-version', lb, "-print"]

Can some one explain how subprocess deal with quotes in command. Here are different combinations i tried and the errors

Case 1 - Double quotes for lb and elf

lb = '\"lbtype(%s-%s)\"' % (tmp_prod_no, rev)
cmd = [clt, 'find', lm_path, '-follow', '-name', '\"*.elf\"', '-version', lb, "-print"]

o/p:
--b''--
===DONE===

Case 2 - Double quotes for elf

lb = 'lbtype(%s-%s)' % (tmp_prod_no, rev)
cmd = [clt, 'find', lm_path, '-follow', '-name', '\"*.elf\"', '-version', lb, "-print"]    

o/p:
cleartool: Error: Syntax error in query (near character 1).
cleartool: Error: Invalid query: ""lbtype(CXC1727075-R78A12)""
cleartool: Warning: Skipping   \vobs/cello/babs/control_test_dm/jpre_test_lm/bin/jpre_test.ppc.elf".
CALLEDPROCESSERROR
Command '['/usr/atria/bin/cleartool', 'find',   '/vobs/cello/babs/control_test_dm/jpre_test_lm', '-follow', '-name', '*.elf', '-version', '"lbtype(CXC1727075-R78A12)"', '-print']' returned non-zero exit status 1

Case 3 - No Double quotes gives correct answer

lb = 'lbtype(%s-%s)' % (tmp_prod_no, rev)
cmd = [clt, 'find', lm_path, '-follow', '-name', '*.elf', '-version', lb, "-print"]

o/p:
--b'\vobs\asd\asd\adasd'--
===DONE===

Why the clearcase complaining about lbtype in Case 2 but not in Case 1.

score 0 · Answer 3 · edited May 23 '17 at 11:49

0

As I mention in "cleartool find" prerequisite, a find must be executed from within a ClearCase view (snapshot or dynamic);

You need to make sure the Python subprocess:

keep the current path
has the vob mounted in that process (if you are using a dynamic view).

torek mentions in the comments a simpler cause for the issue:

The default for the various subprocess invocations is shell=False

See his answer.

That is especially true if your path (on the shell) is /vobs: that is only valid for a view which has been set, which spawns its own process. See "Python and ClearCase setview" for more.
It wouldn't allow an alias like 'clt' to work either ('cleartool' might work, 'clt' not: see "Clearcase running commands from a script (error: Bad phone number)" as an example)

Try it in a dynamic view using the full path of said dynamic view:

/view/aView/vobs/aVob/...

A good test would be for your Python script to do first:

a print of the cwd (current working directory)
a cleartool lsvob (to see, if you are within a dynamic view, if those vobs are still mounted in the python subprocess).

edited May 23 '17 at 11:49

Community

1
1

answered Jul 10 '13 at 14:20

VonC

1,262,500
529
4,410
5,250

I replaced cmd with the below line and i got the correct output cmd = [clt, "ls", vobfile] So i believe settings related to vlearcase view should be fine. – Hemanth Jul 10 '13 at 14:53
@Hemanth or `cleartool find` somehow prints on stderr instead of stdout? – VonC Jul 10 '13 at 15:43
You are right (of course) about all the cleartool restrictions, but the problem in this case is much simpler. :-) The default for the various `subprocess` invocations is `shell=False`. – torek Jul 11 '13 at 08:17
BTW in previous $orkplace I had a side project of attempting to write something like cvsps for ClearCase. I found a lot of your info on CC here on SO really useful. (Side project never got that far, and it's now "previous $orkplace". But still, tracking down original checkins from multisite zippy merges, ugh.) – torek Jul 11 '13 at 22:54
@torek Thank you. Funnily enough, I was in charge to write a ClearCase to Git import tool. That workspace is also referred as "previous" ;) – VonC Jul 12 '13 at 05:26

'cleartool find -print' inside Python3 'subprocess.check_output' returns empty string

3 Answers3