On a server, we run continuous integration jobs as runscripts of singularity image, e.g.
wget url_to_mysingularityimage.sif
chmod +x ./mysingularityimage.sif
./mysingularityimage.sif
In case this is relevant: in our specific case, the runscript is a bash script. It works great.
After executing the job, the image is deleted.
rm ./mysingularityimage.sif
But it looks like a corresponding loop device still exists, i.e.
losetup -a | grep mysingularityimage.sif
outputs something like:
/dev/loop27: []: (/path/to/mysingularityimage.sif (deleted)), offset 40960, sizelimit 831307776
As we run jobs, the number of loop devices increases. At some points, the limit number of loop devices is reached, and singularity fails with the error:
FATAL: container creation failed: mount /proc/self/fd/3->/usr/local/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/3: failed to find loop device: could not attach image file to loop device: no loop devices available
Is there something we are doing wrong ? Is there a command we are expected to run at the end of each job to ensure the loop device is "freed" ?
(note: we use bamboo for continuous integration, and bamboo always performs the job in a new directory, which may explain why the same loop device is not reused over. Bamboo: https://www.atlassian.com/software/bamboo).
