run awk date conversion in python 3.6 subprocess

Question

I am trying to write the following awk line into python

awk -F, '{cmd="date -d \""$2"\" +%s"; cmd | getline date; printf ("%s,%d,%d\n",$1,date,$3)}' counts.csv > counts_epoch.csv

v1

import subprocess

with open("counts_epoch.csv", 'wb') as f:
    subprocess.Popen(["awk", "'{cmd="date -d \""$2"\" +%s"; cmd | getline date; printf ("%s,%d,%d\n",$1,date,$3)}'", 'counts.csv'], stdout=f)

This will return:

    subprocess.Popen(["awk", "'{cmd="date -d \""$2"\" +%s"; cmd | getline date; printf ("%s,%d,%d\n",$1,date,$3)}'", 'counts.csv'], stdout=f)
                                    ^
SyntaxError: invalid syntax

v2:

with open("counts_epoch.csv", 'wb') as f:
    subprocess.call(["awk", '{cmd="date -d \""$2"\" +%s"; cmd | getline date; printf ("%s,%d,%d\n",$1,date,$3)}', 'counts.csv'], stdout=f)

returns:

awk: cmd. line:1: {cmd="date -d ""$2"" +%s"; cmd | getline date; printf ("%s,%d,%d
awk: cmd. line:1:                                                            ^ unterminated string
awk: cmd. line:1: {cmd="date -d ""$2"" +%s"; cmd | getline date; printf ("%s,%d,%d
awk: cmd. line:1:                                                            ^ syntax error

Any suggestion? Thanks!

Why don't you want to use python native date/time manipulation functions/libraries? — Mikhail Zakharov, Aug 08 '18 at 09:01
no need for all this complex stuff. If you are to tag this "python", use python. — Jean-François Fabre, Aug 08 '18 at 09:01
Hey folks, please stop delegitimizing the question! Even though I myself would also consider using pure Python in a situation like this, it doesn't make this question any less valid. There are many legitimate reasons as to why someone would want to do this. — Elias Dorneles, Aug 08 '18 at 09:26
From python you're calling awk to call shell to call date? Seems like there's got to be a simpler way.... — Ed Morton, Aug 08 '18 at 10:47

score 1 · Accepted Answer · answered Aug 08 '18 at 09:23

1

My trick for these situations is to copy the exact command using triple quotes to quote all the quotes inside of it, prefix with r to get a raw string and then use shlex.split to do split the arguments as the shell would do:

import shlex
cmdline = r"""awk -F, '{cmd="date -d \""$2"\" +%s"; cmd | getline date; printf ("%s,%d,%d\n",$1,date,$3)}' counts.csv"""
cmd = shlex.split(cmdline)

Then you can do your thing:

with open("counts_epoch.csv", 'wb') as f:
    subprocess.Popen(cmd, stdout=f)

WARNING: Tbh I haven't tested the code I just pasted here, but it should work. =)

answered Aug 08 '18 at 09:23

Elias Dorneles

22,556
11
85
107

The correct code to call date from awk would be `cmd="date -d \047$2\047 +\047%s\047"; date=((cmd | getline line) > 0 ? line : -1); close(cmd); printf "%s,%d,%d\n",$1,date,$3`. See http://awk.freeshell.org/AllAboutGetline and google shell quoting rules. idk about wrapping that in python. – Ed Morton Aug 08 '18 at 10:58
Thank you! It does work! A quick question: the number of quotes should be the same as something? Or 3 is just the magic number here? @elias – Chubaka Aug 09 '18 at 08:06
@Chubaka yes, triple quotes is the Python syntax for multiline strings: https://stackoverflow.com/questions/10660435/pythonic-way-to-create-a-long-multi-line-string/10660443#10660443 – Elias Dorneles Aug 09 '18 at 08:53

run awk date conversion in python 3.6 subprocess

1 Answers1