python regex to parse the output of a command

Question

The below python script executes a command. The command returns an output. Output is listed below.

I need to parse the following from the output.

1) HTTP response code 2) Contents of points array. Array could be empty, or non empty array (last line of the output)

OUTPUT :

Matched line for host akaa-wcb54zu2flskjhx4-szsapxzmalbot2fr.luna-dev.akamaiapis.net
{'access_token': 'akaa-r6gnip3mmql3evrj-mgiex4eajadrjltu', 'max-body': 131072, 'secret': 'yBc1LSGKBd6+sSP5tTR2o+YxlNYvlYNjWGdCFDl6Xt0=', 'signed-header': [], 'host': 'akaa-wcb54zu2flskjhx4-szsapxzmalbot2fr.luna-dev.akamaiapis.net', 'client_token': 'akaa-eyaa6fvcfngdlrb6-uxvn3hkpzxx33oxc'}
[]
Canonicalized header:
Auth data: EG1-HMAC-SHA256 client_token=akaa-eyaa6fvcfngdlrb6-uxvn3hkpzxx33oxc;access_token=akaa-r6gnip3mmql3evrj-mgiex4eajadrjltu;timestamp=20180410T16:38:24+0000;nonce=265e61d8-813c-4c3a-9edb-91caaa027420;
String-to-sign: POST    https   akaa-wcb54zu2flskjhx4-szsapxzmalbot2fr.luna-dev.akamaiapis.net  /prolexic-analytics/v2/metrics      YSycIdG2e9nY8KVLrPvMfzrBNMZEI8ncRxtcUP4h1J0=    EG1-HMAC-SHA256 client_token=akaa-eyaa6fvcfngdlrb6-uxvn3hkpzxx33oxc;access_token=akaa-r6gnip3mmql3evrj-mgiex4eajadrjltu;timestamp=20180410T16:38:24+0000;nonce=265e61d8-813c-4c3a-9edb-91caaa027420;
args = ['curl', '-H', 'Authorization: EG1-HMAC-SHA256 client_token=akaa-eyaa6fvcfngdlrb6-uxvn3hkpzxx33oxc;access_token=akaa-r6gnip3mmql3evrj-mgiex4eajadrjltu;timestamp=20180410T16:38:24+0000;nonce=265e61d8-813c-4c3a-9edb-91caaa027420;signature=p6y+oDzaTfh3nj32R8jveDx8EGmjpHGybL7LJ0L+Adk=', '-S', '-i', '-k', '-X', 'POST', '-H', 'Host:akaa-wcb54zu2flskjhx4-szsapxzmalbot2fr.luna-dev.akamaiapis.net', '-H', 'Content-Type: application/json', '-H', 'Accept: application/json', '-d', '{"contract": "amade", "start": 1522420500, "end": 1522424147, "samples": 20, "type": {"connect": ["bandwidthIn"]}}', 'https://akaa-wcb54zu2flskjhx4-szsapxzmalbot2fr.luna-dev.akamaiapis.net/prolexic-analytics/v2/metrics', '-H', 'Expect:']
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: application/json
Content-Length: 151
X-Trace-Id: c4db5acce87c20f3
Date: Tue, 10 Apr 2018 16:38:26 GMT
Connection: keep-alive

{"status":true,"currentContract":"amade","statusMsg":"Metrics acquired successfully","data":[{"service":"connect","metric":"bandwidthIn","points":[]}]}

SCRIPT :

import subprocess
import sys
import re

n = sys.argv[1]
for i in range(int(n)):
 # command to be executed 
 cmd = "python egcurl --eg-config .egcurl -Sik -X POST --eg-verbose -H 'Host:akaa-wcb54zu2flskjhx4-szsapxzmalbot2fr.luna-dev.akamaiapis.net' -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{'contract': 'amade', 'start': 1522420500, 'end': 1522424147, 'samples': 20, 'type': {'connect': ['bandwidthIn']}}'  https://akaa-wcb54zu2flskjhx4-szsapxzmalbot2fr.luna-dev.akamaiapis.net/prolexic-analytics/v2/metrics"


 cmd_exec = subprocess.call(cmd, shell=True)
 r1 = re.search("HTTP(.*)",cmd_exec)
 print(r1)
 print(cmd_exec)

error:

File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/re.py", line 146, in search
    return _compile(pattern, flags).search(string)
TypeError: expected string or buffer

See my answer to a similar question, https://stackoverflow.com/a/49570796/7811673. But in this case i'd use `urllib` instead of calling `curl`. — t.m.adam, Apr 10 '18 at 17:06
Why don't you try printing cmd_exec? call() returns the return code of the process. — Taku, Apr 10 '18 at 17:06
I think you need to rethink this. You are using Python to call another instance of Python in the shell to run curl then using a regex to parse that? — dawg, Apr 10 '18 at 17:12

score 1 · Answer 1 · answered Apr 10 '18 at 17:08

1

the problem is you are doing

 cmd_exec = subprocess.call(cmd, shell=True)

and subprocess.call return the result code, not the output. You want to use

 cmd_exec = subprocess.check_output(cmd, shell=True)

answered Apr 10 '18 at 17:08

Freddy

779
1
6
20

Good call! It should be noted this only works with python 2.7+ though. – Aaron N. Brock Apr 10 '18 at 17:10
Thanks Aaron, I'll take in count next time :) – Freddy Apr 10 '18 at 17:18
there's nothing wrong with your answer, I'm just commenting for future people looking at this answer :) – Aaron N. Brock Apr 10 '18 at 17:19

score 0 · Answer 2 · answered Apr 10 '18 at 17:06

You're issue here is that subproccess.call() returns the process return code, not the output of the command. From the docs:

Run the command described by args. Wait for command to complete, then return the returncode attribute

So, you're trying to use re.search() on cmd_exec (the return code) which isn't a string so re doesn't know how to handle it and throws an error.

See this answer on how to get the output of a subproccess command!

python regex to parse the output of a command

2 Answers2