7

Is it possible to have a Jenkins Job with has been triggered by the Join plugin copy artifacts from multiple upstream jobs?

I'm trying to set-up a Jenkins configuration with a "diamond" of jobs: my-trigger runs and spawns two jobs, my-fork1 and my-fork2, that can run concurrently and take varying amounts of time, and the Join plugin sets off the job my-join once both forks have completed.

Each of my-trigger, my-fork1 and my-fork2 creates and fingerprints artifacts (say, text files).

I want to copy the artifacts from each of the upstream jobs in my-join using the "Copy artifacts from another project" tool, with the "Which build" parameter set to "Upstream build that triggered this job". However, I see output like this in the console of my-join:

Building remotely on build-machine in workspace /path/to/workspace/my-join

Copied 1 artifact from "my-trigger" build number 63

Copied 1 artifact from "my-fork1" build number 63

Unable to find a build for artifact copy from: my-fork2

and the job fails. In this case, my-fork2 finished first, so my-fork1 triggered the join step. I believe that that means that my-join only has record of my-fork1 and my-trigger as being upstream. If my-fork1 finishes first, then my-fork2 kicks off the join, and the job fails when trying to copy from my-fork1.

If I change the configuration to copy the artifact from the build "Latest successful build" then the build succeeds, but my-trigger may run many times in succession so there would be no guarantee that my-join is joining related artifacts.

How can I get the join step to copy artifacts from multiple forks upstream?

Note: the second point of this question seems to be asking the same thing, but the only answer there doesn't address it, and has been accepted.

Thanks tensorproduct

Community
  • 1
  • 1
laffoyb
  • 1,540
  • 3
  • 22
  • 35

2 Answers2

9

If your builds are parameterized with a unique parameter for each run of the join-diamond, you can use that parameter in the CopyArtifact plugin to determine which build to copy from. You would want to specify "Latest successful build" and qualify it with the parameter and value.

We have a similar situation where I work; multiple simultaneous runs of a join-diamond. The parameter in the build allows the downstream jobs to get the correct artifacts from the upstream jobs.

Jason Swager
  • 6,421
  • 6
  • 41
  • 56
  • Yup, thanks for that. In the end, I used the BUILD_TAG of the triggering build to set a UNIQUE_ID parameter of all downstream builds and used that as the identifying marker for the join step. – laffoyb May 30 '13 at 09:21
  • Can you provide the exact string to use in order to achieve that? I get problems... – dstj Aug 27 '14 at 16:10
  • I'm afraid that I don't work at the place where I put this fix in place anymore, so I don't have the exact string to hand. To the best of my memory, I used the Parameterized Trigger Plugin to kick off the downstream builds. I set a parameter manually with `UNIQUE_ID=${BUILD_TAG}`. – laffoyb Oct 01 '14 at 15:36
  • When triggering the Join build, specify this parameter also. Then, in join-build, I used the CopyArtifact plugin, as Jason suggests. It's important to make sure that "Last successful build" is chosen, and specify `UNIQUE_ID=${UNIQUE_ID}` (though I can't remember the name of the field that goes into). – laffoyb Oct 01 '14 at 15:42
  • 1
    That field would be "Parameter filters". – sschuberth Jan 15 '15 at 13:38
  • @laffoyb -- This method only works with one build at a time. When doing concurrent builds, its not consistently grabbing the same artifacts. Any idea how to remedy it? Asked here: http://stackoverflow.com/questions/39257228/concurrent-jenkins-jobs-always-copying-last-successful-of-last-build-artifacts – kayleeFrye_onDeck Aug 31 '16 at 23:33
  • @kayleeFrye_onDeck I'm afraid that question you linked goes nowhere, so I'm not exactly sure what the issue is. However, in my experience the answer given here (unique parameters defining the upstream job specified in Copy Artifacts step) should work. – laffoyb Sep 01 '16 at 08:12
  • Hi @laffoyb -- I ended up deleting it. It does not work consistently when running multiple concurrent jobs with downstream tasks. I replicated the behavior last night after reading the Bug Report on it. I made a job called delegator which then called on a downstream job that would run `dir /s /d C:\Windows>%WORKSPACE%\node.log` across half a dozen nodes of varying hardware and software configs, and only 4 of the 6 returned the correct artifacts. Be wary of this solution. – kayleeFrye_onDeck Sep 02 '16 at 00:19
  • Could you link the bug report on this issue? Is it a known Jenkins bug? – laffoyb Sep 02 '16 at 08:27
0

Step by Step settings of the provided solution from Jason Swager:

Project dependencies: diamond->fork->diamond_ready

Project "fork": String parameter "UNIQUE_ID" (only dummy not used inside) (Creates an artifcat and Archive the artifacts)

Project "diamond_ready" String parameter: UNIQUE_ID Copy artifacts from another project Project name: fork Parameter filters: UNIQUE_ID=${UNIQUE_ID}

Project "diamond": Trigger parameterized build on other project Projects to build: fork Predefinded parameters: UNIQUE_ID=${BUILD_TAG} Join Trigger: Post-Join Actions: Trigger parameterized build on other projects Projects to build: diamond_ready Predefined Generator parameters: UNIQUE_ID=${BUILD_TAG}

pulp
  • 698
  • 1
  • 6
  • 13