How do I git add only lines matching a pattern?

Question

I'm tracking with git some configuration files. I usually do an interactive git add -p but I'm looking at a way to automatically add all new/modified/deleted lines that match a pattern. Otherwise it's going to take me ages to do all the interactive split and add. git add has a pattern matching for filenames, but I can't find anything about the content.

Answer 1

here's a way:

1. use git diff > patch to make a patch of the current diff.

2. use gawk to make a second patch only of +/- lines matching the pattern: remove - from deleted lines not matching the pattern, delete + lines not matching the pattern, modify the hunk header line numbers, output each modified hunk, but don't output any modified hunks that no longer have any changes in them.

3. use git stash save, apply patch, add -u, and stash pop to apply and stage the modified patch and leave the rest of the changes unstaged.

this worked for several test cases, it works on the entire diff at once (all files), and it's quick.

#!/bin/sh

diff=`mktemp`
git diff > $diff
[ -s $diff ] || exit

patch=`mktemp`

gawk -v pat="$1" '
function hh(){
  if(keep && n > 0){
    for(i=0;i<n;i++){
      if(i==hrn){
        printf "@@ -%d,%d +%d,%d @@\n", har[1],har[2],har[3],har[4];
      }
      print out[i];
    }
  }
}
{
  if(/^diff --git a\/.* b\/.*/){
    hh();
    keep=0;
    dr=NR;
    n=0;
    out[n++]=$0
  }
  else if(NR == dr+1 && /^index [0-9a-f]+\.\.[0-9a-f]+ [0-9]+$/){
    ir=NR;
    out[n++]=$0
  }
  else if(NR == ir+1 && /^\-\-\- a\//){
    mr=NR;
    out[n++]=$0
  }
  else if(NR == mr+1 && /^\+\+\+ b\//){
    pr=NR;
    out[n++]=$0
  }
  else if(NR == pr+1 && match($0, /^@@ \-([0-9]+),?([0-9]+)? \+([0-9]+),?([0-9]+)? @@/, har)){
    hr=NR;
    hrn=n
  }
  else if(NR > hr){
    if(/^\-/ && $0 !~ pat){
      har[4]++;
      sub(/^\-/, " ", $0);
      out[n++] = $0
    }
    else if(/^\+/ && $0 !~ pat){
      har[4]--;
    }
    else{
      if(/^[+-]/){
        keep=1
      }
      out[n++] = $0
    }
  }
}
END{
  hh()
}' $diff > $patch

git stash save &&
  git apply $patch &&
  git add -u &&
  git stash pop

rm $diff
rm $patch

refs:

git diff apply

unified diff format

gawk match groups to array

git add -u

Answer 2

I cranked out this experimental and poorly tested program in TXR:

Sample run: first where are we at in the repo:

$ git diff
diff --git a/lorem.txt b/lorem.txt
index d5d20a4..58609a7 100644
--- a/lorem.txt
+++ b/lorem.txt
@@ -2,10 +2,14 @@ Lorem ipsum dolor sit amet,
 consectetur adipiscing elit,
 sed do eiusmod tempor
 incididunt ut labore et dolore
-magna aliqua. Ut enim ad minim
+minim
+minim
 veniam, quis nostrud
 exercitation ullamco laboris
+maxim
+maxim
 nisi ut aliquip ex ea commodo
+minim
 consequat.  Duis aute irure
 dolor in reprehenderit in
 voluptate velit esse cillum

And:

$ git diff --cached  # nothing staged in the index

The goal is to just commit the lines containing a match for min:

$ txr addmatch.txr min lorem.txt
patching file .merge_file_BilTfQ

Now what is the state?

$ git diff
diff --git a/lorem.txt b/lorem.txt
index 7e1b4cb..58609a7 100644
--- a/lorem.txt
+++ b/lorem.txt
@@ -6,6 +6,8 @@ minim
 minim
 veniam, quis nostrud
 exercitation ullamco laboris
+maxim
+maxim
 nisi ut aliquip ex ea commodo
 minim
 consequat.  Duis aute irure

And:

$ git diff --cached
diff --git a/lorem.txt b/lorem.txt
index d5d20a4..7e1b4cb 100644
--- a/lorem.txt
+++ b/lorem.txt
@@ -2,10 +2,12 @@ Lorem ipsum dolor sit amet,
 consectetur adipiscing elit,
 sed do eiusmod tempor
 incididunt ut labore et dolore
-magna aliqua. Ut enim ad minim
+minim
+minim
 veniam, quis nostrud
 exercitation ullamco laboris
 nisi ut aliquip ex ea commodo
+minim
 consequat.  Duis aute irure
 dolor in reprehenderit in
 voluptate velit esse cillum

The matching stuff is in the index, and the nonmatching +maxim lines are still unstaged.

Code in addmatch.txr:

@(next :args)
@(assert)
@pattern
@file
@(bind regex @(regex-compile pattern))
@(next (open-command `git diff @file`))
diff @diffjunk
index @indexjunk
--- a/@file
+++ b/@file
@(collect)
@@@@ -@bfline,@bflen +@afline,@aflen @@@@@(skip)
@  (bind (nminus nplus) (0 0))
@  (collect)
@    (cases)
 @line
@      (bind zerocol " ")
@    (or)
+@line
@      (bind zerocol "+")
@      (require (search-regex line regex))
@      (do (inc nplus))
@    (or)
-@line
@      (bind zerocol "-")
@      (require (search-regex line regex))
@      (do (inc nminus))
@    (or)
-@line
@;;    unmatched - line becomes context line
@      (bind zerocol " ")
@    (end)
@  (until)
@/[^+\- ]/@(skip)
@  (end)
@  (set (bfline bflen afline aflen)
        @[mapcar int-str (list bfline bflen afline aflen)])
@  (set aflen @(+ bflen nplus (- nminus)))
@(end)
@(output :into stripped-diff)
diff @diffjunk
index @indexjunk
--- a/@file
+++ b/@file
@  (repeat)
@@@@ -@bfline,@bflen +@afline,@aflen @@@@
@    (repeat)
@zerocol@line
@    (end)
@  (end)
@(end)
@(next (open-command `git checkout-index --temp @file`))
@tempname@\t@file
@(try)
@  (do
     (with-stream (patch-stream (open-command `patch -p1 @tempname` "w"))
       (put-lines stripped-diff patch-stream)))
@  (next (open-command `git hash-object -w @tempname`))
@newsha
@  (do (sh `git update-index --cacheinfo 100644 @newsha @file`))
@(catch)
@  (fail)
@(finally)
@  (do
     (ignerr [mapdo remove-path #`@tempname @tempname.orig @tempname.rej`]))
@(end)

Basically the strategy is:

• do some pattern matching on the git diff output to filter the hunks down to the matching lines. We must re-compute the "after" line count in the hunk header, and preserve the context lines.

• output the filtered diff into a variable.

• obtain a pristine copy of the file from the index using git checkout-index --temp. This command outputs the temporary name it has generated, and we capture it.

• Now send the filtered/reduced diff to patch -p1, targetting this temporary file holding the pristine copy from the index. Okay, we now have just the changes we wanted, applied to the original file.

• Next, create a Git object out of the patched file, using git hash-object -w. Capture the hash which this command outputs.

• Lastly, use git update-index --cacheinfo ... to enter this new object into the index under the original file name, effectively staging a change for the file.

If this screws up, we can just do git reset to wipe the index, fix our broken scriptology and try again.

Just blindly matching through + and - lines has obvious issues. It should work in the case when the patterns match variable names in config files, rather than content. E.g.

Replacement:

-CONFIG_VAR=foo
+CONFIG_VAR=bar

Here, if we match on CONFIG_VAR, then both lines are included. If we match on foo in the right hand side, we break things: we end up with a patch that just subtracts the CONFIG_VAR=foo line!

Obviously, this could be made clever, taking into account the syntax and semantics of the config file.

How I would solve this "for real" would be to write a robust config file parser and re-generator (which preserves comments, whitespace and all). Then parse the new and original pristine file to config objects, migrate the matching changes from one object to the other, and generate an updated file to go to the index. No messing around with patches.

Answer 3

I don't think that is possible; since git add -p always shows you hunks of changes; but that hunk might contain some line that you wanted to add (and matches your pattern) and a line containing changes you don't want to add.

Sometimes I face a similar problem when I did two changes and want to commit them separately:

• rename of a variable
• add some functionality

There is a workaround I use:

• Put my changes aside (using git stash or just copying the files)
• rename the variable (so I redo the easy part of my work; since renaming a variable is usually taken care of by the IDE)
• commit these changes
• reapply my changes (using git stash pop or copying the files back)
• commit the rest of my changes

Answer 4

This is, of course crazy. But you know, we have some crazy workflows at my work that I occasionally ask about and there's usually some good reason for the craziness.

Ok, is the pattern a line-by-line pattern or a "if the chunk contains it" pattern? If it is line by line, perhaps you could do something like this. This won't work exactly, but it's a start

git diff <file> | egrep '^[^+]|<pattern' > file.patch
git stash
git apply file.patch

If you have to apply any chunk that has the pattern you're looking for, then you will need a longer, more stateful script to parse your diff. In fact, that is probably necessary anyway. Crawl through the diff looking for '@@' characters indicating the beginning of a diff section. Buffer up that section till you get to the end. If you've run into the pattern in question, output the section, if not, then throw it away. Then apply that new diff as a patch.

git diff <file> | parse_diff_script.sh > file.patch
git stash
git apply file.patch

Answer 5

If you're using VSCode, you can select all lines matching the pattern in a file and stage them:

1. Open a diff in the Source Control pane.
2. Select an example line, then hold ctrlD to select all other matching lines. You can enable regex-based search using the floating panel in the top-right after hitting ctrlD once.
3. ctrlK+ctrlaltS to stage all selections.

Answer 6

You can start with git ls-files to obtain a list of files of interest for a given path. Then you can pipe that list into grep and restrict based on a regex match. Finally, this reduced list of files can be piped into git add via xargs git add:

git ls-files [path] | grep '^some regex goes here$' | xargs git add -p

This approach will allow you to apply a smart regex which hopefully can reduce the number of files for your interactive git add session. By definition, doing git add -p requires human interaction, so if after applying the pattern you still have too many files, then you should find another approach.

If you want to add whole files, not lines, try this answer.