0

I'm trying to retrieve and work with data from historical versions of files in a git repo. I'd like to have something like a dictionary that holds <hash>, <time of commit>, <value retrieved from contents of a file revision>, <commit message> for each entry.

I figured the data I retrieve from each file revision, and any calculations done with them, would be best handled using python. And the subprocess module appeared to be the best fit to integrate my git commands.

Below I show how I'm defining a function getval(key, filename) that I had hoped would output <SHA-1 hash>:<Value> to console, but would like to have a dict with more info... also with <time>, and <commit message>.

I help operate an ion accelerator, where we store 'savesets'--or values relevant to a given accelerator tune--using git. Of the values in these files, are things like charge(Q) and mass(A). Ultimately, I want to retrieve both values, get the ratio (Q/A), and display a list of file revision hashes sorted by the charge:mass ratio of the ion we delivered with the settings in that file's revision.

Sample of file (for 56Fe17+):

# Date: 2018-12-21 01:49:16.888 PV,SELECTED,TIMESTAMP,STATUS,SEVERITY,VALUE_TYPE,VALUE,READBACK,READBACK_VALUE,DELTA,READ_ONLY REA_EXP:LINE,0,1544047322.881066957,NO_ALARM,NONE,enum,"JENSA~[UDF;AT-TPC;GPL;JENSA]",,"---",,true REA_BTS19:BEAM:OPTICSFILE,0,1541798820.065952460,NO_ALARM,NONE,string,"BTS19_test3.data",,"---",,true REA_BTS19:BEAM:A_BOOK,0,1545322510.562031883,NO_ALARM,NONE,double,"56.0",,"---",,true REA_BTS19:BEAM:Z_BOOK,0,1545322567.544226340,NO_ALARM,NONE,double,"26.0",,"---",,true REA_BTS19:BEAM:Q_BOOK,0,1545322512.701768974,NO_ALARM,NONE,double,"17.0",,"---",,true

So far--and with the help of others here--I've figured out a git one-liner that greps the revision history of a given file for a key[a string] and uses sed and awk to output <hash>:<val associated with the key>.

Git Oneliner I'm Starting with:

git grep 'BTS19:BEAM:A_BOOK' $(git rev-list --all) -- ReAccelerator/Snapshots/RFQ-JENSA_Setpoints.snp | sed 's/:/,/' | awk -F, '{print $1 ":" $8}'

Oneliner's Output

e78f73fe6f90e93d5b3ccf90975b0e540d12ce09:"56.0" 4b94745bd0a6594bb42a774c95b5fc0847ef2d82:"56.0" f2d5e263deac1d9112be791b39f4ce1b1b34e55d:"56.0" c03800de52143ddb2abfab51fcc665ff5470e363:"56.0" 4a3a564a6d87bc6ff5f3dc7fec7670aeecfe6a79:"58.0" d591941e51c4eab1237ce726a2a49448114b8f26:"58.0" a9c8f5cdf224ff4fd94514c33888796760afd792:"58.0" 2f221492beea1663216dcfb27da89343817b11fd:"58.0"

I've also started playing with the subprocess python module. But I'm struggling to figure out how to handle my more complicated git commands. Generally, I'll want to be able to pass a key, and a file.. something like getval(key, filename).

When my cmd string was ['git', 'grep', str, '$(git rev-list --all)', '--', pathspec], it returned errors stating that '$(git rev-list --all)' was ambiguous. Thinking it wasn't being expanded, I added a separate process to execute the nested command, but I'm not sure I'm doing this correctly.

My Python file (gitfun.py): which I'm currently running the function from

import sys, os
import subprocess

def getval(str, pathspec, repoDir='/mnt/d/stash.projects/rea'):
    p1 = subprocess.Popen(["git", "rev-list", "--all"], stdout=subprocess.PIPE)
    output, err = p1.communicate()

    cmd = ['git', 'grep', str, output, '--', pathspec]        
    p2 = subprocess.Popen(cmd, cwd=repoDir)
    p2.wait()

cwd = '/mnt/d/stash.projects/rea'
filename = 'ReAccelerator/Snapshots/RFQ-JENSA_Setpoints.snp'
os.chdir(cwd)
getval('BTS19:BEAM:A_BOOK', filename)

Currently it is returning 'file name too long' so (even though I'm not convinced it really is too long) I tried changing my core.longpaths in git config to true, however this had no effect. Again why I suspect I'm not handling my replacement of the $(git rev-list --all) expansion correctly.

For this code, I expect something that looks like this:

522628b8d3db01ac330240b28935933b0448649c:ReAccelerator/Snapshots/RFQ-JENSA_Setpoints.snp:REA_BTS19:BEAM:A_BOOK,0,1545240215.74320185 5,NO_ALARM,NONE,double,"58.0",,"---",,true 2557c599d2dc67d80ffc5b9be3f79899e0c15a10:ReAccelerator/Snapshots/RFQ-JENSA_Setpoints.snp:REA_BTS19:BEAM:A_BOOK,0,1545240215.74320185 5,NO_ALARM,NONE,double,"58.0",,"---",,true 7fc97ec2aa76f32265196c42dbcd289c49f0ad93:ReAccelerator/Snapshots/RFQ-JENSA_Setpoints.snp:REA_BTS19:BEAM:A_BOOK,0,1545240215.74320185 5,NO_ALARM,NONE,double,"58.0",,"---",,true ...

But I ultimately want an output to console that looks identical to the git one-liner above, or better yet, a dict that I can print to console or do other things with.

Daniel Crisp
  • 107
  • 11

1 Answers1

2

Remember that your shell tokenizes the command line using white space.

When you run git rev-list --all, you get output like:

2a4be2748fad885f88163a5b9b1b438fe3cb2ece
c1a30c743eb810fbefe1dc314277931fa33842b3
b2e5c75131e94a3543e5dcf9fb641ccd553906b4
95718f7e128a8b36ca93d6589328cc5b739668b1
87a9ada188a8cd1c13e48c21f093be7027d61eca

When you substitute that into your git grep command...

git grep 'BTS19:BEAM:A_BOOK' $(git rev-list --all) -- \
ReAccelerator/Snapshots/RFQ-JENSA_Setpoints.snp

...each line is a separate argument. That is, if the output of git rev-list --all was exactly what I've shown above, then your one-liner would be tokenized into the following arguments, which I have listed one per line for clarity:

git
grep
BTS19:BEAM:A_BOOK
2a4be2748fad885f88163a5b9b1b438fe3cb2ece
c1a30c743eb810fbefe1dc314277931fa33842b3
b2e5c75131e94a3543e5dcf9fb641ccd553906b4
95718f7e128a8b36ca93d6589328cc5b739668b1
87a9ada188a8cd1c13e48c21f093be7027d61eca
--
ReAccelerator/Snapshots/RFQ-JENSA_Setpoints.snp

But you're not doing this in your Python code! You're pasing the entire output of git rev-list --all as a single argument. That means the command you're trying to execute has a fixed number (6) of arguments:

git
grep
BTS19:BEAM:A_BOOK
2a4be2748fad885f88163a5b9b1b438fe3cb2ece c1a30c743eb810fbefe1dc314277931fa33842b3 b2e5c75131e94a3543e5dcf9fb641ccd553906b4 95718f7e128a8b36ca93d6589328cc5b739668b1 87a9ada188a8cd1c13e48c21f093be7027d61eca
--
ReAccelerator/Snapshots/RFQ-JENSA_Setpoints.snp

All those revisions are getting bundled together in a single argument, which is where the "filename too long" error comes from. You need to split that output into multiple arguments just like the shell does:

p1 = subprocess.Popen(["git", "rev-list", "--all"], stdout=subprocess.PIPE)
output, err = p1.communicate()

cmd = ['git', 'grep', str] + output.splitlines() + ['--', pathspec]        
p2 = subprocess.Popen(cmd, cwd=repoDir)
p2.wait()
larsks
  • 277,717
  • 41
  • 399
  • 399
  • 1
    Side note: if you want to split like the shell, `shlex.split` gets a lot closer. In this case, the output really is lines, so `output.splitlines()` is fine, of course. (Also, Python 3.x was mentioned in tags, so be aware the subprocess output defaults to bytes, rather than strings.) – torek Jan 03 '19 at 09:05
  • Ah hah! Thank you both! This is fun. @larsks, output.splitlines() makes this work as expected. @Torek The tag was no mistake, I am using python 3. I probed type(output) once I suspected output was an issue, it is indeed bytes. I wasn't quite sure what this meant though... as it turns out, output.splitlines() works with the bytes type, but shlex.split does not. shlex.split returns `Attribute Error: 'bytes' object has no attribute 'read'`. – Daniel Crisp Jan 03 '19 at 16:39
  • Also, to give credit where it's due, and provide some continuity to my questions if anybody finds this useful. @Torek was the one who helped me figure out my one-liner. https://stackoverflow.com/questions/53951431/searching-and-handling-git-objects. Thanks again! – Daniel Crisp Jan 03 '19 at 16:42
  • 1
    @DanielCrisp: dealing with mixed `bytes` and `str` in Py3k is a pain. The trick to remember here is that external programs (`git` and other things run by `subprocess`) really do work with *byte* streams; if you want characters, they tend to be UTF-8 encoded, so that you can spell "agréable" and the like, where é and ö and so on take more than one byte. Py3k `str` characters are each a Unicode point. – torek Jan 03 '19 at 16:44
  • @torek: On that point... I've started trying to capture output of p2 the same way I was with p1. But I'd like to have it printed out as well. The issue is that print(output2) is not converting occurences of \n into actual new lines. So I've tried to handle the encoding... but none of this is working: `print(output.decode('utf-8'))`, then `print(output.format(output.decode('utf-8')), end='')`, and `print(str(output, 'utf-8'))`. [etc. from https://stackoverflow.com/questions/606191/convert-bytes-to-a-string]. Do you have further suggestions? – Daniel Crisp Jan 03 '19 at 18:26
  • There are two ways to deal with this. One is to use `.decode(encoding)`, as you're trying to do. The other is to use the `subprocess` support that does this with the stream directly, through the encoding, text, or "universal newlines" options (see https://docs.python.org/3/library/subprocess.html). Since you're on Windows, if I recall correctly, you probably really want the latter. – torek Jan 03 '19 at 18:51
  • Ah, thank you. As it turns out, I had mixed up which versions of my code I was running; so `.decode("UTF-8")` does work as expected, but I also tried the "universal_newlines=True" option for Popen, which is working as well. – Daniel Crisp Jan 03 '19 at 19:19