Edit: it turns out the question was not "how do I do this particular operation" but, apparently, a completely different question. Here's my rephrase of the different question.
Given one starting point commit identifier R, one tip-of-branch commit identifier T, and one onto-commit identifier O, how would one come up with a commit sequence that identifies some commits that can be squashed?
Unfortunately, I can't tell from this revised question (which I'm not sure is properly revised in the first place) what qualifies some commits as squash-able and others as non-squashable. If we allow arbitrary amounts of graph to occur between R and T, we might have:
R -- A - B ------ C - D -- E - T
\ / /
F - G -- H -- I
for instance, and it's not clear to me whether A-B should be squashed, or C-D-E, or D-E; or H-I, and so on. It seems likely that commits that are themselves merges and/or serve as branch-points should remain, but they themselves could be squashed-into.
Edit some more: I see the question has also been updated: the commits to squash are those that were themselves on a branch—but given my example above, all of F-G-H-I were probably "on a branch" that was merged back in more than once, and perhaps not all of those should be squashed. So I'm still at a loss for choosing "squash candidates". If we constrain the problem further, they might be "any commit that's on the R..T chain that is not on the --first-parent segment of the R..T chain, perhaps.
[original answer follows]
Here's how I would do this semi-manually (note that if git rebase -i could do it, this is pretty much how it would do it: interactive rebase is basically a series of cherry-picks, with some special casing). Below, O stands for the SHA-1 of commit O, or a branch or tag name that resolves to it.
$ git checkout -b rewrite O # get a new "rewrite" branch
$ git cherry-pick A # copy commit A
$ git cherry-pick --no-commit B # get B, no commit yet
$ git cherry-pick --no-commit C # add C: this is your S
$ git diff # check on it, do any fixing up needed
$ git commit # commit S, write new commit msg
# (might grab messages from B&C)
$ git cherry-pick D # add commit D
You can omit the --no-commit on the pick of C if its commit message is what you want and you're sure there's nothing to fix up. Or, use git cherry-pick --edit to start with C's commit message and edit it (or, as usual, git commit --amend or whatever).
This does depend on the existing merge M being trivial or mostly-trivial. If it's an "evil merge" with a lot of changes hidden in it, one or two more git cherry-picks of M, with -m 1 and/or -m 2, might be required before picking D.
The "rewrite" branch is then your desired result. If you start it with a better name than "rewrite" you won't have to rename it either. :-)
