0

So we have a job that will generate some content that we need for our builds. It runs like 5 different scripts in succession, these scripts will generate the content, commit, then push to the repo on a specific branch. We are running these scripts against 3 different branches.

We are running these script in parallel with different jenkins machines (no shared workspaces).

The flow is like this:

  1. Find repo in home directory (again, this directory is not shared with other machines running in parallel) and checkout/pull/reset to desired branch
  2. copy repo into workspace to perform build actions (we copy into workspace because the repo is so large it takes 10m for jenkins to clone it from scratch)
  3. run script #1
  4. git commit -m "<constructed msg>"
  5. git push
  6. git clean -df
  7. Repeat starting from step 3 until all scripts are completed.

Here is the error we're getting when they run in parallel. Note, the job works perfectly fine if we do one branch at a time.

Command: git push Exitcode: 1 Standard Output: Error Stream: To <repo> ! [rejected] release/<branch> -> release/<same branch> (fetch first) error: failed to push some refs to '<repo>' hint: Updates were rejected because the remote contains work that you do hint: not have locally. This is usually caused by another repository pushing hint: to the same ref. You may want to first integrate the remote changes hint: (e.g., 'git pull ...') before pushing again. hint: See the 'Note about fast-forwards' in 'git push --help' for details.

The error makes it seem like because they are all pushing at the same time there is some kind of conflict, but I would think this should only be the case if we're pushing to the same branches? These are all 3 diff branches. It seems one branch succeeded but the rest fail, so I think its obvious that the parallel nature of this is causing the issue, but I fail to understand why when its running on different machines, on different branches.

krazyito65
  • 95
  • 1
  • 1
  • 9
  • Oh, please, don't tell me you **not** making a shallow clone in step 2. – eftshift0 Oct 21 '22 at 19:13
  • If you are _pushing_ against the same repo, there would be some locking going on when a push takes place, I would expect, right? – eftshift0 Oct 21 '22 at 19:15
  • > Oh, please, don't tell me you not making a shallow clone in step 2. < We are not doing a shallow clone. It was a full clone of the repo, we are just updating it to the branch we want then copying it into the workspace, which gets deleted after the run ends. > If you are pushing against the same repo, there would be some locking going on when a push takes place, I would expect, right? < We are pushing to the same repo, yes, but does a lock like this happen regardless of what branch was pushed to? I would expect it to only really lock the branch that was changed. – krazyito65 Oct 21 '22 at 19:19
  • It's a tricky question about locking to which I don't know the answer to... It's obvious to me that it's just not about operating on 3 different branches because it's really related to the _objects_ that each push operation will need to... well, push.... you might have duplicates and so on.... and so if 3 pushes are potentially pushing the same objects, to avoid problems in this scenario, I would expect some locking to be going on there under the hood, regardless of what references (branch/tag) you are trying to move around. – eftshift0 Oct 21 '22 at 20:06
  • This is the part I am referring to when I say that I do not expect that you are doing a full clone: `it takes 10m for jenkins to clone it from scratch`. That sounds like a _full_ clone to me. – eftshift0 Oct 21 '22 at 20:07
  • https://stackoverflow.com/a/52668468/2437508 – eftshift0 Oct 21 '22 at 20:11
  • >and so if 3 pushes are potentially pushing the same objects, to avoid problems in this scenario, I would expect some locking to be going on there under the hood, regardless of what references (branch/tag) you are trying to move around.< i think this is probably the answer i was looking for, and it seems likely, i just wasn't sure how it worked to begin with. – krazyito65 Oct 21 '22 at 20:14
  • >This is the part I am referring to when I say that I do not expect that you are doing a full clone: it takes 10m for jenkins to clone it from scratch. That sounds like a full clone to me.< sorry i may be articulating it wrong, but we have the repo already checked out on the machine, its there before the job even starts. All we do is pull/checkout/reset it prior to copying it into the workspace. So yes, its a full clone, but we're only updating the history thats missing instead of cloning the full thing all at once – krazyito65 Oct 21 '22 at 20:16
  • Hey, this is an _educated_ guess.... wait for more feedback to have a clearer idea of what is going on as there are people around here that are way more knowledgeable than I am that could give you the gory details or point in a completely different unrelated direction. – eftshift0 Oct 21 '22 at 20:17

1 Answers1

0

Ok... here's a thought...avoid the 3 pushes and make a single push after you finish your 3rd commit... like this:

git push some-remote @~2:branch-of-first-commit @~:branch-of-second-commit @:branch-of-third-commit

Does that make sense?

eftshift0
  • 26,375
  • 3
  • 36
  • 60
  • For now we're just running the branches in series instead of in parallel It works like this. This would be a good solution if I was doing it on the same machine. Sadly, different machines are being used so each machine would not know about the other commits. – krazyito65 Oct 22 '22 at 02:34