losetup -d not working

Question

I have setup 5 loop devices with backup files as raw 40 MB files in /tmp folder.

$ losetup -l
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE
/dev/loop1         0      0         1  0 /tmp/1
/dev/loop2         0      0         0  0 /tmp/2
/dev/loop3         0      0         0  0 /tmp/3
/dev/loop4         0      0         0  0 /tmp/4
/dev/loop5         0      0         0  0 /tmp/5

I create a raid array for first 4 devices using lvm. Hence first 4 devices become busy as follows:

$ lsblk 
NAME                    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                       8:0    0 931.5G  0 disk 
├─sda1                    8:1    0   100M  0 part /boot/efi
├─sda2                    8:2    0   900M  0 part 
├─sda3                    8:3    0   128M  0 part 
├─sda4                    8:4    0 372.6G  0 part 
├─sda5                    8:5    0 140.5G  0 part /
├─sda6                    8:6    0    15G  0 part 
├─sda7                    8:7    0 186.3G  0 part 
├─sda8                    8:8    0  37.3G  0 part [SWAP]
├─sda9                    8:9    0  93.1G  0 part 
└─sda10                   8:10   0  40.8G  0 part 
loop1                     7:1    0  78.1M  0 loop 
├─vol_vg-raid4_rmeta_0  252:0    0     4M  0 lvm  
│ └─vol_vg-raid4        252:8    0    48M  0 lvm  
└─vol_vg-raid4_rimage_0 252:1    0    16M  0 lvm  
  └─vol_vg-raid4        252:8    0    48M  0 lvm  
loop2                     7:2    0  78.1M  0 loop 
├─vol_vg-raid4_rmeta_1  252:2    0     4M  0 lvm  
│ └─vol_vg-raid4        252:8    0    48M  0 lvm  
└─vol_vg-raid4_rimage_1 252:3    0    16M  0 lvm  
  └─vol_vg-raid4        252:8    0    48M  0 lvm  
loop3                     7:3    0  78.1M  0 loop 
├─vol_vg-raid4_rmeta_2  252:4    0     4M  0 lvm  
│ └─vol_vg-raid4        252:8    0    48M  0 lvm  
└─vol_vg-raid4_rimage_2 252:5    0    16M  0 lvm  
  └─vol_vg-raid4        252:8    0    48M  0 lvm  
loop4                     7:4    0  78.1M  0 loop 
├─vol_vg-raid4_rmeta_3  252:6    0     4M  0 lvm  
│ └─vol_vg-raid4        252:8    0    48M  0 lvm  
└─vol_vg-raid4_rimage_3 252:7    0    16M  0 lvm  
  └─vol_vg-raid4        252:8    0    48M  0 lvm  

I now try deleting the loop device 1. I do not get any warnings/errors either on console or in dmseg:

$ sudo losetup -d /dev/loop1 
shehbaz@donjaffer:~$ 

Now, I list all devices again as follows:

$ losetup -l
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE
/dev/loop1         0      0         1  0 /tmp/1
/dev/loop2         0      0         0  0 /tmp/2
/dev/loop3         0      0         0  0 /tmp/3
/dev/loop4         0      0         0  0 /tmp/4
/dev/loop5         0      0         0  0 /tmp/5
shehbaz@donjaffer:~$

I also see the lsblk output same as I provided above.

I try running strace on the command that I tried earlier (to see if there is some permission denied or some other form of error that is not being reported), I get the following output:

$ sudo strace losetup -d /dev/loop1
execve("/sbin/losetup", ["losetup", "-d", "/dev/loop1"], [/* 17 vars */]) = 0
brk(0)                                  = 0x1985000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc13f271000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=108331, ...}) = 0
mmap(NULL, 108331, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc13f256000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libsmartcols.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260R\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=134536, ...}) = 0
mmap(NULL, 2233952, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc13ee2e000
mprotect(0x7fc13ee4e000, 2093056, PROT_NONE) = 0
mmap(0x7fc13f04d000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1f000) = 0x7fc13f04d000
mmap(0x7fc13f04f000, 1632, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fc13f04f000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\v\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1869392, ...}) = 0
mmap(NULL, 3972864, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc13ea64000
mprotect(0x7fc13ec24000, 2097152, PROT_NONE) = 0
mmap(0x7fc13ee24000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c0000) = 0x7fc13ee24000
mmap(0x7fc13ee2a000, 16128, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fc13ee2a000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc13f255000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc13f253000
arch_prctl(ARCH_SET_FS, 0x7fc13f253740) = 0
mprotect(0x7fc13ee24000, 16384, PROT_READ) = 0
mprotect(0x7fc13f04d000, 4096, PROT_READ) = 0
mprotect(0x60f000, 4096, PROT_READ)     = 0
mprotect(0x7fc13f273000, 4096, PROT_READ) = 0
munmap(0x7fc13f256000, 108331)          = 0
brk(0)                                  = 0x1985000
brk(0x19a6000)                          = 0x19a6000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=7216688, ...}) = 0
mmap(NULL, 7216688, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc13e382000
close(3)                                = 0
stat("/sys/block", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
uname({sys="Linux", node="donjaffer", ...}) = 0
stat("/dev/loop-control", {st_mode=S_IFCHR|0660, st_rdev=makedev(10, 237), ...}) = 0
stat("/dev/loop1", {st_mode=S_IFBLK|0660, st_rdev=makedev(7, 1), ...}) = 0
open("/dev/loop1", O_RDONLY|O_CLOEXEC)  = 3
ioctl(3, LOOP_CLR_FD)                   = 0
close(3)                                = 0
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++
shehbaz@donjaffer:~$ 

I still see all the devices using losetup command. Is -d option not working properly? I tried unmounting /dev/loop1 as well, but it tells me /dev/loop1 is not mounted.

$ umount -l /dev/loop1
umount: /dev/loop1: not mounted
shehbaz@donjaffer:~$

Please help. thank you.

Answer 1

Important note on the above code snippet from Tom hale:

the losetup command used in bash scripts is sensitive to the value if IFS.

try this to see for yourself:

line="/dev/loop8 /somefile"
losetup $line     # this works
IFS=""
losetup $line     # this one will fail

If your variable ($line) contains multiple parameters, the IFS value allows losetup to detect them. By clearing it, the last losetup thinks that the complete variable is only 1 parameter and that obviously fails.

Answer 2

I would say that the driver keeps the file opened until you close the last file descriptor that is opened on it, thus the file remains in use and using the loop device until you close it.

That is quite common on Linux, actually, if you create a file and open it, the space it takes on the file system will be available only when all file descriptor opened on it will be closed.

Thus here, will remain used until you stop the lvm raid your created on it.

As a consequence, you cannot reuse the loopX until it was released; I guess you get an error if you try to use it, and losetup -f does not propose it neither.

Answer 3

It seems to be caused by a process with working directory on the old FS mountpoint.

Frustratingly, it seems there is no way of telling which processes they are after the unmount.

I raised util-linux issue "losetup -d exits 0 when it doesn't succeed" #484, and it the maintainer shows that it's a kernel issue.

I wrote the following code to print the prior usage of the mountpoint if the loopback device isn't automatically removed.

#!/bin/bash

set -euo pipefail
shopt -s failglob

# Unmount the entries given in $1, (one per line)
unmount() {
  # https://unix.stackexchange.com/questions/9784/how-can-i-read-line-by-line-from-a-variable-in-bash
  # You need printf '%s\n' "$var" here because if you use printf '%s' "$var" 
  # on a variable that doesn't end with a newline then the while loop will
  # completely miss the last line of the variable.
  # printf '%s\n' "$1" | while IFS= read -r dir; do # each mount point
  IFS= printf '%s\n' "$1" | while read -r dir; do # each mount point
    ! /usr/bin/mountpoint -q -- "$dir" && continue

    # Bug: can only find processes using mountpoint before unmount
    # https://github.com/karelzak/util-linux/issues/484
    open_files=$(sudo lsof "$dir" 2>/dev/null || true)
    # Exits non-zero if object not in use
    # Avoid "WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs"

    mountpoints=$(awk -v dir="$dir" 'BEGIN{dir="^" dir "$"} $2 ~ dir {print $1}' /etc/mtab)

    while sudo "/bin/umount" -Rdl -- "$dir"; do # returns false when already unmounted
      # -R  Recursive unmount
      # -d  Remove the associated loop device
      # -l  Lazy remove filesystem references immediately
      # -v  Verbose
      sleep 0.01;
    done

    if loop_dev=$(losetup -a --list | grep --fixed-strings "$mountpoints"); then
      printf "WARNING: loop device remains after unmount:\n%s\n" "$loop_dev" 2>&1
      printf "%s\n" "$open_files" 2>&1
    fi
  done
}

# # List of mount points to umount before checking /etc/mtab
# # List any contained loopback files here
# sub_mounts='/media/backup
# /media/backup-NEW'
# unmount "$sub_mounts"

# sort -r to unmount subiretories before parents:
mtab_mounts=$(/bin/awk '$2 ~ /^\/media/ {print $2}' /etc/mtab | sort -r)
unmount "$mtab_mounts"

Answer 4

The reason you don't get an error message when you try to delete the loop, is that the kernel doesn't return it as an error. Now, many years later, that seems to still be the case.

ioctl(3, LOOP_CLR_FD)                   = 0

Even though lsof doesn't show any processes keeping the loop device open, the kernel will not release the loop until you also get lvm to release the device. (Without RAID, but still relevant, I imagine,) I only managed to get rid of my own loops after deactivating the LVs using the PV. In my case it disappears as soon as I do:

lvchange -an computer-vg/root

No need to even run losetup -d again.