5

I am running a daemon which analyses files in a directory and then deletes them. In case the daemon is not running for whatever reason, files get stacked there. Today I had 90k files in that directory. After starting the daemon again, it processed all the files.

However, the directory remains large; "ls -dh ." returns a size of 5.6M. How can I "defragment" that directory? I already figured out that renaming that directory, and creaing a new one with the same name and permissions solves the problem. However, as files get written in there at any time, there doesn't seem to be a safe way to rename the directory and create a new one as for a moment, the target directory does not exist.

So a) is there a way/a (shell) program which can defragment directories on an ext3 filesystem? or b) is there a way to create a lock on a directory so that trying to write files blocks until the rename/create has finished?

Bada
  • 59
  • 2
  • Btw. apparently you can use the "filefrag" command to check whether a directory is fragmented. – oliver Apr 27 '15 at 12:39
  • Also see http://serverfault.com/questions/264124/shrink-reset-directory-size and http://unix.stackexchange.com/questions/38639/how-to-compact-a-directory – oliver Jan 20 '17 at 13:47

3 Answers3

1

Not really applicable for Ext3, but maybe useful for users of other filesystems:

I haven't tried either of these solutions, though.

oliver
  • 6,204
  • 9
  • 46
  • 50
1

"Optimize directories in filesystem. This option causes e2fsck to try to optimize all directories, either by reindexing them if the filesystem supports directory indexing, or by sorting and compressing directories for smaller directories, or for filesystems using traditional linear directories." -- fsck.ext3 -D

Of course this should not be done on a mounted filesystem.

Shi
  • 4,178
  • 1
  • 26
  • 31
0

I'm not aware of a way to reclaim free space from within a directory.

5MB isn't very much space, so it may be easiest to just ignore it. If this problem (files stacking up in the directory) occurs on a regular basis, then that space will be reused anytime the directory fills up again.

If you desperately need the ability to shrink the directory here's a (ugly) hack that might work.

Replace the directory with a symbolic link to an empty directory. If this problem reoccurs, you can create a new empty directory, and then change the symlink to point to the new directory. Changing the symlink should be atomic, so you won't loose any incoming files. Then you can safely empty and delete the old directory.

[Edited to add: It turns out that this does not work. As Bada points out in the comments you can't atomically change a symlink in the way I suggested. This leaves me with my original point. File systems I'm familiar with don't provide a mechanism to reclaim free space within directory blocks.]

Keith Smith
  • 3,611
  • 3
  • 19
  • 12
  • 1
    How can the symlink be changed? If the target already exists, the symlink gets created *inside* the target pointing to itself, try "mkdir haha hehe && ln -vs haha hoi && ln -vs hehe hoi". And according to symlink(2), it returns EEXIST in case the target already exists. I do not yet see it being atomic. – Bada Nov 29 '10 at 19:51
  • @bada: You're right. I misremembered the how symlinks are created. So my hack is both ugly and wrong! Sorry about that. I'm afraid that leaves me without any suggestions about how to solve your problem... – Keith Smith Nov 30 '10 at 20:53
  • 1
    Actually, rename is atomic. So create a new link, and then move it in place, either using the OS call rename (python os.rename) or the command "mv -f -T newlink oldlink". -T tells not to put in the directory, but to replace the link instead. – Mathieu Longtin Jan 20 '17 at 16:47