How to automagically remove commits that cancel themselves out?

Question

I have a development branch with many commits. These commits include "real" changes (e.g. add a feature) as well as temporary changes (e.g. add test code in one commit, then remove it in a later commit).

The real changes from this branch are being gradually added into the master. After each addition, I rebase the develoment branch on the new master. With each cycle, difference between the development branch and the master is getting smaller. However, the rebased development branch contains all original (now rebased) commits. Some of these commits now have zero net effect.

Simple example:

DEV_BRANCH
    Commit ...
    Commit D: Remove all test code from f.cpp
    Commit ...
    Commit C: Add new feature to f.cpp
    Commit ...
    Commit B: Add more test code to f.cpp
    Commit A: Add some test code to f.cpp
    Commit ...
MASTER
    Commit X: Add new feature to f.cpp
    Commit ...

At this point, change to f.cpp from commit C is already in the master as commit X, and commits A+B+D when combined have no change to f.cpp. In other words, diff between the branch and the master shows nothing for file f.cpp.

(In reality, commits A, B, C, and D may include changes to other files as well.)

Is there any way to automagically simplify commits in the development branch either during rebase or otherwise?

In the above simple example, is it possible to remove commit C (with change already merged to master hence now "empty") and also commits A, B, and D (no change when combined) automatically?

In a more complicated scenario when the commits to f.cpp modify other files as well, is it possible to remove changes to file f.cpp automatically from the commits in the development branch (to simplify them) but leave the commits present if they include changes to other files?

Rephrased, if applying all changes to a file from all commits in the development branch results in a file identical to the one in the master, is it possible to prune changes to this file from the development branch commits? (I do realize this may lead to side-effects if changes to other files need to be done in sync with those to the file that has no net effect. However, this is not a problem in my scenario because I do not need to cherry-pick any intermediate state from the development branch.)

Answer 1

I think that there are a couple of fairly simple solutions for your particular use-case that take into account how the dupes are generated in the first place.

Solution 1: For a small number of commits on diverging branches

Let's say you have the following set of commits in dev shown in git rebase --interactive mode:

pick bf45b13 Add feature X
pick b1f790f Cleanup feature X
pick 40b299a Add feature Y

Now you decide to cherry-pick 40b299a into master.

The problem is that the new commit when you rebase dev will be empty. I would recommend doing git rebase -i master for future rebases. You can then delete the lines that you cherry picked from the selection, and be left only with the commits that you want.

Since you want to do this "automagically", you can make your own git script to do a simultaneous cherry-pick and rebase, eliminating the correct commit from dev while adding it to master.

If you call the following script something like git-transfer, and add it to your PATH somewhere, you can invoke it as git transfer dev master commit [...]:

#!/bin/bash

usage() {
    echo "Usage: git transfer from-branch to-branch commits [...]"
    if [ -n "${1}" ]
    then
        echo
        for line; do echo "${line}"; done
    fi
    exit 1
}

FROM="$(git rev-parse --abbrev-ref "${1}")"
[ -z "${FROM}" ] && usage 'from-branch must be a valid branch name' || shift
TO="$(git rev-parse --abbrev-ref "${1}")"
[ -z "${TO}" ] && usage 'to-branch must be a valid branch name' || shift

ORIGINAL="$(git rev-parse --abbrev-ref HEAD)"

if [ "${ORIGINAL}" == "${TO}" ]
then
    echo "Already on branch ${TO}"
else
    echo "Switching from ${ORIGINAL} to ${TO}"
    git checkout "${TO}" || exit
fi

while [ $# -gt 0 ]
do
    for COMMIT in "$(git rev-parse "${1}" | grep '^[^^]')"
    do
        echo "Moving ${COMMIT} to ${TO}"
        git cherry-pick "${COMMIT}"
        echo "Removing ${COMMIT} from ${FROM}"
        EDITOR="sed -i '/^pick $(git rev-parse --short ${COMMIT})/ d'" git rebase -i "${TO}" "${FROM}"
    done
    shift
done

if [ "${ORIGINAL}" != "${TO}" ]
then
    echo "Switching back to ${ORIGINAL} from ${TO}"
    git checkout "${ORIGINAL}"
fi

The script accepts the name of the "from" branch that will be rebased, the name of the "to" branch that will be cherry-picked into, and a list of commits, commit ranges, etc. It is based heavily on the ideas proposed in this question and answer (Also this answer for the loop).

This solution is suitable for small numbers of commits (one at a time most likely) as well as for branches that have diverged significantly.

Solution 2: For larger numbers of commits on a single timeline

In the use case you described, you periodically rebase dev onto master, so dev is effectively ahead of master all the time. Next time, instead of picking individual commits onto master, do the following:

1. Start an interactive rebase
2. Move the commits you want to the beginning of the list
3. Complete rebase
4. Move master to the last commit you want.

This approach will save you from having any duplication at all and is probably the simplest thing you can do. Note that while dev will constantly be rebased, master will only fast forward with this approach.

Answer 2

I see people have, in the comments, addressed the complexity issues.

It's not possible in general since the problem is too hard. If you define the problem down to "empty" (no changes from immediately preceding) commits, it's easy, and is in fact already in Git: rebase already does it unless you ask for --keep-empty.

(For that matter, git filter-branch, which is sort of like rebase on steroids, can also do it, although its default is the opposite: it keeps such commits unless given --prune-empty. If you are using a commit filter, you may use git_commit_non_empty_tree, which simply compares the current tree to the previous before actually invoking git commit-tree. In this case, git filter-branch is probably heavier-weight than you would want, though.)

In a comment, you wrote:

I considered to create a new branch from the master and copy all changed files from my original dev branch over. However, this creates one new lump commit and looses all granularity (which is useful to have until I am done picking all real changes.)

Git is designed for this: just "detach" HEAD (git checkout --detach or check out by raw SHA-1 ID) and make whatever temporary commits you like. Once you switch back to a named branch, the temporary commits will remain only as long as they are protected by the HEAD reflog (default 30 days for unreachable-from-tip commits). (Well, if they're loose objects, as they will be, they also get a 14-day grace period, but 14 is less than 30.)

You don't even have to make a commit though: just git diff (or git diff-tree) the tree at the tip of master vs the tree at the tip of dev_branch. In other words, both forms of the complete source tree are already in Git, as two individual commits. They have whatever history (of other commits) in between them:

          o-------o    <-- master
         /
...--o--o
         \
          o--o--o--o   <-- dev_branch

but that cannot stop you from comparing them directly. The detached HEAD trick is useful for doing this:

          o-------o    <-- master
         /
...--o--o------*       <-- detached HEAD
         \
          o--o--o--o   <-- dev_branch

where * is some proposed commit (actually committed!), which you can then discard if you don't like it by using git reset or git checkout to move HEAD back to some other commit, or back to pointing to a branch name.

(In fact, this is how git rebase does its work: it detaches HEAD at the point on which the new branch will grow, adds commits until the new branch is done, then erases the arrow going from the branch name to its original tip-most commit and makes the arrow point to the new tip-most commit.)