1

Suppose a threaded program is doing filesystem operations (traversing, reading, creating, removing, moving directories and files).

According to an answer, ftw() is unsafe in this situation. (Also, ftw() requires global variables, which doesn't seem too elegant.)

Should I use fts_open() / fts_children() / fts_read() instead? Or perhaps scandir()?

What is it that is unsafe about a threaded program using ftw() (where some thread is using ftw() to aggredate data and another thread is deleting/moving some data), precisely, and why?

Jonathan Leffler
  • 730,956
  • 141
  • 904
  • 1,278
étale-cohomology
  • 2,098
  • 2
  • 28
  • 33
  • 2
    If one thread executes `chdir()` while another thread is executing `ftw()`, all hell may break loose. Removing a directory while it is being scanned might cause problems, too (but that could happen from another process, let alone other threads). Is portability an issue? It may not be available everywhere (BSD/macOS marks it as "expected to be included in a future IEEE Std 1003.1-1988 ('POSIX.1') revision", which improves its chances of being portable, but doesn't guarantee it — it isn't in POSIX 2018. – Jonathan Leffler May 24 '18 at 08:03
  • 2
    You'll probably need to use `FTS_NOCHDIR` if you're in a threading application, or be diligent about using the `*at()` functions carefully to access files. – Jonathan Leffler May 24 '18 at 08:07
  • 1
    Portability is not an issue. So, changing the working directory of the process (from different threads) is the only thing that can lead to inconsistencies? Are absolute paths a magic fix? Is `FTS_NOCHDIR` guaranteed to solve this issue? I'm really struggling the understand what could possibly go wrong...! – étale-cohomology May 24 '18 at 08:43
  • 1
    I don't know whether changing directory is the only way to run into problems; it is just one rather important way in which things could get screwed up. If you have two threads, or two processes, at work, one trying to scan the file system via `fts_read()` et al and the other making arbitrary changes (like moving a directory from one position in the tree to another), then I can see all sorts of opportunities for confusion in the accounting being done by the scanning code — omitting things that should have been counted, counting things twice which should have been counted just once. – Jonathan Leffler May 24 '18 at 15:43
  • 1
    I'm going to mention [`nftw()`](http://pubs.opengroup.org/onlinepubs/9699919799/functions/nftw.html) because it hasn't showed up yet otherwise, It has several advantages over [`ftw()`](http://pubs.opengroup.org/onlinepubs/9699919799/functions/ftw.html), but thread-safety isn't one of them. Consequently, it is mostly an aside as the issues that afflict `fts_read()` and `ftw()` also afflict `nftw()`. – Jonathan Leffler May 24 '18 at 15:46
  • 1
    Note that [`scandir()`](http://pubs.opengroup.org/onlinepubs/9699919799/functions/scandir.html) reads and selects the contents of a single directory without traversing the hierarchy. It is designed for a different job from `ftw()` and friends. It is not a simple replacement for `ftw()` et al. – Jonathan Leffler May 24 '18 at 15:54

1 Answers1

1

Converting comments into an answer.

If one thread executes chdir() while another thread is executing ftw() (or nftw()), all hell may break loose. Removing a directory while it is being scanned might cause problems, too (but that could happen from another process, let alone other threads).

Is portability an issue? The fts(3) suite of functions may not be available everywhere (BSD/macOS marks it as "expected to be included in a future IEEE Std 1003.1-1988 ('POSIX.1') revision", which improves its chances of being portable, but doesn't guarantee it — it isn't in POSIX 2018(see below).

You'll probably need to use FTS_NOCHDIR if you're in a threading application, or be diligent about using the *at() functions carefully to access files.

Portability is not an issue. So, changing the working directory of the process (from different threads) is the only thing that can lead to inconsistencies? Are absolute paths a magic fix? Is FTS_NOCHDIR guaranteed to solve this issue?

I don't know whether changing directory is the only way to run into problems; it is just one rather important way in which things could get screwed up. If you have two threads, or two processes, at work, one trying to scan the file system via fts_read() et al and the other making arbitrary changes (like moving a directory from one position in the tree to another), then I can see all sorts of opportunities for confusion in the accounting being done by the scanning code — omitting things that should have been counted, counting things twice which should have been counted just once.

I'm going to mention nftw() because it hasn't showed up yet otherwise. It has several advantages over ftw(), but thread-safety isn't one of them. Consequently, it is mostly an aside as the issues that afflict fts_read() and ftw() also afflict nftw().

Note that scandir() reads and selects the contents of a single directory without traversing the hierarchy. It is designed for a different job from ftw() and friends. It is not a simple replacement for ftw() et al.

It looks like fts(3) was considered but rejected for POSIX: http://www.opengroup.org/platform/ballots/1003.1a.d14.crb.txt

Jonathan Leffler
  • 730,956
  • 141
  • 904
  • 1,278