COPR fedoraproject.org builder refuses to download sources specified in my .spec file

Question

I have a package build in https://copr.fedoraproject.org. My spec for rpkg is here. The build is failing with the following

Generated rpkg config:
[rpkg]
preprocess_spec = True

[git]
anon_clone_url = https://github.com/%(repo_path)s

[lookaside]
download_url = https://${git_props:remote_netloc}/repo/pkgs/%(repo_path)s/%(filename)s/%(hashtype)s/%(hash)s/%(filename)s

Writing config into /var/lib/copr-rpmbuild/workspace/workdir-ce16hwto/.config/rpkg.conf
Running: rpkg srpm --outdir /var/lib/copr-rpmbuild/results --spec /var/lib/copr-rpmbuild/workspace/workdir-ce16hwto/skupper-router

cmd: ['rpkg', 'srpm', '--outdir', '/var/lib/copr-rpmbuild/results', '--spec', '/var/lib/copr-rpmbuild/workspace/workdir-ce16hwto/skupper-router']
cwd: /var/lib/copr-rpmbuild/workspace/workdir-ce16hwto/skupper-router
rc: 1
stdout: Wrote: /var/lib/copr-rpmbuild/results/skupper-router.spec
stderr: git_dir_pack: archiving /var/lib/copr-rpmbuild/workspace/workdir-ce16hwto/skupper-router:
commit cb2a29004b1660168cc4a6894e521b0cf71cb25d (HEAD -> jd_2022_03_15_rpm, origin/jd_2022_03_15_rpm)
Date:   Sat Apr 9 01:04:32 2022 +0200

    add static build rpm
git_dir_pack: Wrote: /var/lib/copr-rpmbuild/results/skupper-router-cb2a2900.tar.gz
error: Bad source: /var/lib/copr-rpmbuild/results/qpid-proton-0.37.0.tar.gz: No such file or directory

Copr build error: git_dir_pack: archiving /var/lib/copr-rpmbuild/workspace/workdir-ce16hwto/skupper-router:
commit cb2a29004b1660168cc4a6894e521b0cf71cb25d (HEAD -> jd_2022_03_15_rpm, origin/jd_2022_03_15_rpm)
Date:   Sat Apr 9 01:04:32 2022 +0200

    add static build rpm
git_dir_pack: Wrote: /var/lib/copr-rpmbuild/results/skupper-router-cb2a2900.tar.gz
error: Bad source: /var/lib/copr-rpmbuild/results/qpid-proton-0.37.0.tar.gz: No such file or directory

(full log at https://download.copr.fedorainfracloud.org/results/jdanek/skupper-router/srpm-builds/04126008/builder-live.log.gz)

On my machine, running rpkg srpm succeeds.

I searched around and it seems to me that COPR should be willing to download sources if they are distributed over https, which mine are.

Can I use the lookaside cache feature, somehow, for my sources?

Answer 1

Your build fails on the git_dir_pack (rpkg-util, aka rpkg failure). This happened before the rpmbuild process even started.

A few weeks ago we migrated the Copr builders to Fedora 35, and thus also to rpkg-util v3. Feel free to take a look how to fix similar issues. Or perhaps contact the rpkg-util upstream for a guidance.

Answer 2

Normally, downloading the qpid-proton file by https would work in Copr. It's done by rpmbuild (being invoked by rpkg) if there is %_disable_source_fetch 0 defined in rpm macros somewhere.

The problem is that one of the lines in the spec file is: %undefine _disable_source_fetch which disables this behavior for the given spec file.

Answer 3

The problem is that the sources file apparently must be downloaded from the lookaside cache. So, let's create ourselves one. Or at least, the appearance of one.

First, download the required source archive to your computer, to the git repo checkout dir, and do an "offline upload" using rpkg

wget https://www.apache.org/dist/qpid/proton/0.37.0/qpid-proton-0.37.0.tar.gz
rpkg upload --offline qpid-proton-0.37.0.tar.gz

This creates sources file. Commit that to the repo. Undo the edit to .gitignore that this also did. (If that rpkg command fails, ignore that as long as the sources file got created.)

Create new file rpkg.conf with the following content, and commit it to repo.

[lookaside]
download_url = https://www.apache.org/dist/qpid/proton/0.37.0/%(filename)s

Now push that and build it in COPR.

(Don't get confused by the obviously wrong download_url printed below; our rpkg.conf in the repo directory is overriding that.)

Running: git checkout jd_2022_03_15_rpm_cstdint

cmd: ['git', 'checkout', 'jd_2022_03_15_rpm_cstdint']
cwd: /var/lib/copr-rpmbuild/workspace/workdir-rmzys1po/skupper-router
rc: 0
stdout: branch 'jd_2022_03_15_rpm_cstdint' set up to track 'origin/jd_2022_03_15_rpm_cstdint'.
stderr: Switched to a new branch 'jd_2022_03_15_rpm_cstdint'

Generated rpkg config:
[rpkg]
preprocess_spec = True

[git]
anon_clone_url = https://github.com/%(repo_path)s

[lookaside]
download_url = https://${git_props:remote_netloc}/repo/pkgs/%(repo_path)s/%(filename)s/%(hashtype)s/%(hash)s/%(filename)s

Writing config into /var/lib/copr-rpmbuild/workspace/workdir-rmzys1po/.config/rpkg.conf
Running: rpkg srpm --outdir /var/lib/copr-rpmbuild/results --spec /var/lib/copr-rpmbuild/workspace/workdir-rmzys1po/skupper-router

cmd: ['rpkg', 'srpm', '--outdir', '/var/lib/copr-rpmbuild/results', '--spec', '/var/lib/copr-rpmbuild/workspace/workdir-rmzys1po/skupper-router']
cwd: /var/lib/copr-rpmbuild/workspace/workdir-rmzys1po/skupper-router
rc: 0
stdout: Downloading qpid-proton-0.37.0.tar.gz from jiridanek/skupper-router at www.apache.org:

Wrote: /var/lib/copr-rpmbuild/results/skupper-router.spec
Wrote: /var/lib/copr-rpmbuild/results/skupper-router-0.0.git.3560.d7714b9c-2.0.0.fc35.src.rpm
stderr: git_dir_pack: archiving /var/lib/copr-rpmbuild/workspace/workdir-rmzys1po/skupper-router:
commit d7714b9cac0641d4e34c0774b875fcb67cee8793 (HEAD -> jd_2022_03_15_rpm_cstdint, origin/jd_2022_03_15_rpm_cstdint)
Author: Jiri Daněk <jdanek@redhat.com>
Date:   Wed Apr 13 17:13:22 2022 +0200

    add autopack
git_dir_pack: Wrote: /var/lib/copr-rpmbuild/results/skupper-router-d7714b9c.tar.gz

Output: ['skupper-router.spec', 'qpid-proton-0.37.0.tar.gz', 'skupper-router-0.0.git.3560.d7714b9c-2.0.0.fc35.src.rpm', 'skupper-router-d7714b9c.tar.gz']

Success!