2

I have scripts which run some git commands and try to push changes. Those changes are not conflicting changes, since each one works with a different file.

Now what is happening is that when one script pulls and tries to push. in that brief moment if history changes, it is not able to push.

How to make sure that if push fails? it rebases or just patches the code changes?

I dont want to force push since that will rewrite the history and i will lose changes.

Note: these all are working on the same branch.

Tilak Raj
  • 1,369
  • 5
  • 31
  • 64
  • 1
    did you try: "git pull --rebase " then push ; every one working on same branch isn't a good idea – jo_ Apr 24 '22 at 16:26
  • so u think everyone should work on a different branch and then how do i make sure, everyone merges properly back in main? if i can do that, i am happy to do it. I have main permissions to push/force push. – Tilak Raj Apr 24 '22 at 16:29
  • also even git pull and git push. During this window. someone can makes changes on remote, making the current history invalid. – Tilak Raj Apr 24 '22 at 16:30
  • i think i need NFS or something to make sure, commands run while taking a lock. i can't think of any other solution. – Tilak Raj Apr 24 '22 at 16:31
  • 2
    The bigger frame challenge here is why do you have scripts making changes to a git repo, and is git definitely the right tool for the job? Should you be looking for a data store that offers exclusive locks and automatic versioning or change tracking? For instance, an SQL database with "on update" triggers to record the changes. – IMSoP Apr 24 '22 at 16:58
  • we use argo cd which use github as a truth source. We have scripts for some use case. – Tilak Raj Apr 24 '22 at 17:10
  • How much time elapses between the last fetch/pull before the commits are created, and when they are (attempted to be) pushed? – TTT Apr 25 '22 at 03:30
  • git pull --rebase=false --no-edit -X ours origin "$BRANCH" followed by a push . this works for me with a retry logic. – Tilak Raj May 01 '22 at 00:42

1 Answers1

2

i think i need NFS or something to make sure, commands run while taking a lock

Actually, since Git 2.23 (Q3 2019), git push has a --atomic option, which reinforces what was already a quasi-atomic operation before.

So as long as the server is set to reject any divergent push (unless --force is used), a git pull --rebase, as commented, will be enough.

# repeat until git push works:
git pull --rebase
git push --force      # which is quasi atomic

Note: go-git does not support rebase, so you might need to wrap that command (git pull --rebase) in a exec.Command call, using go-git/go-git plumbing/transport/file/client.go.

The OP Tilak Raj proposes in the comments:

retry 5 git update-ref -d refs/remotes/origin/main && git pull --rebase=false --no-edit -X ours origin "refs/heads/$BRANCH" && git push -u origin "$BRANCH" &> /tmp/gitplan

Tilak Raj
  • 1,369
  • 5
  • 31
  • 64
VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • i cant use git more than 2.17 – Tilak Raj Apr 25 '22 at 01:25
  • you mean to say, a git pull --rebase followed by git push will do the trick? . i assume this answer doesn't hold true without atomic? – Tilak Raj Apr 25 '22 at 01:26
  • @TilakRaj It does, considering git push is quasi atomic already. – VonC Apr 25 '22 at 05:55
  • I am using some go routines to push something as well. We had our product designed around github. Thats why we have go routines and scripts doing things. – Tilak Raj Apr 25 '22 at 08:25
  • go-git doesnt have a git pull rebase supported. – Tilak Raj Apr 25 '22 at 08:25
  • i was thinking of distributed locking for the same case. But i really like what u mentioned. I will try it. see how it works. On the go git side, i can only use pull. – Tilak Raj Apr 25 '22 at 08:26
  • # repeat until git push works: could u tell me small snippet for it? i am assuming. Having some sleep at the end, in a loop or something? how do i check if push fails? – Tilak Raj Apr 25 '22 at 08:28
  • @TilakRaj "repeat until git push works" just means: "if the push is rejected (because somebody else has pushed first), then `git pull --rebase`, test locally, and push again. You repeat that sequence as long as the push is rejected. – VonC Apr 25 '22 at 15:10
  • makes sense. i can use this way of doing thing. ```so you might need to wrap that command (git pull --rebase) in a exec.Command call, using go-git/go-git plumbing/transport/file/client.go.``` could u elaborate this more? – Tilak Raj Apr 26 '22 at 10:25
  • @TilakRaj the idea is to wrap the git pull --rebase command in an exec.Command call, as coded in https://github.com/go-git/go-git/blob/db2bc57350561c4368a8d32c42476699b48d2a09/plumbing/transport/file/client.go#L38-L41, since go-git does not support directly any rebase operation. – VonC Apr 26 '22 at 12:14
  • so if i have my stuff started with worktrees in go, i can directly in bw go to the same branch and run git pull --rebase and it should work right? – Tilak Raj Apr 26 '22 at 16:34
  • That's the idea, yes. – VonC Apr 26 '22 at 16:41
  • great. i am trying out the whole thing, i will let u know if it works out or not :). thanks for pointing out this. Even my seniors also asked me to do the same stuff. I still don't understand why this would workout? is it because of the quasi atomic operation? – Tilak Raj Apr 26 '22 at 17:32
  • where can i read more about this quasi-atomic operation ? i cant find good sources. – Tilak Raj Apr 26 '22 at 17:34
  • @TilakRaj The first two links of the answer are the sources I have on that "atomic" aspect. – VonC Apr 26 '22 at 20:04
  • this doesn't work. it stilll shows the same error – Tilak Raj Apr 27 '22 at 21:00
  • @TilakRaj And with a `git push --force`? – VonC Apr 27 '22 at 22:42
  • i cant do force push. that defeats the purpose os the question – Tilak Raj Apr 27 '22 at 23:01
  • its says history changed – Tilak Raj Apr 27 '22 at 23:01
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/244268/discussion-between-vonc-and-tilak-raj). – VonC Apr 27 '22 at 23:08
  • git pull --rebase=false --no-edit -X ours origin "$BRANCH" with a git push. This wrapped in a retry logic works fine for now. – Tilak Raj May 01 '22 at 00:43
  • could u update the answer? `retry 5 git update-ref -d refs/remotes/origin/main && git pull --rebase=false --no-edit -X ours origin "refs/heads/$BRANCH" && git push -u origin "$BRANCH" &> /tmp/gitplan` this is the actual soln. – Tilak Raj May 07 '22 at 20:43