24

I have a trouble with NFS client-side attribute caching. I'm using some servers, one is an NFS server and the others are NFS client servers.

All servers are Debian(lenny, 2.6.26-2-amd64 of Linux) and versions are following.

 % dpkg -l | grep nfs
ii  libnfsidmap2                        0.20-1                     An nfs idmapping library
ii  nfs-common                          1:1.1.2-6lenny1            NFS support files common to client and server
ii  nfs-kernel-server                   1:1.1.2-6lenny1            support for NFS kernel server

In the NFS server, /etc/exports is written as following:

/export-path   192.168.0.0/255.255.255.0(async,rw,no_subtree_check)

In the NFS clients, /etc/fstab is written as following:

server:/export-path     /mountpoint   nfs rw,hard,intr,rsize=8192,async 0 0

As you can see, "async" option is used for multi-clients access performance. However, sometimes this can cause false-caching errors.

Since I am maintaining many servers (and I have not so strong permission to change the mount options), I don't want to modify /etc/exports nor /etc/fstab. I think it is sufficient if I have a command-line tool that "cleans" NFS client-side attribute cache with a user permission.

Please let me know if there such commands.

Thanks,

(Appended)

I mean by "false-caching errors",

 % ls -l /data/1/kabe/foo                  
ls: cannot access /data/1/kabe/foo: No such file or directory
 % ssh another-server 'touch /data/1/kabe/foo' 
 % ls -l /data/1/kabe/foo
ls: cannot access /data/1/kabe/foo: No such file or directory

Sometimes such cases happen. The problem is not a file content but file attributes(=dentries information) since NFS says it guarantees Close-to-Open consistency.

Tomoya Kabe
  • 1,107
  • 2
  • 10
  • 15
  • I found this question while investigating possible solutions to this problem (+1 for that). I'd prefer either no local caching or very short expiry time of local cache if NFS cannot provide consistency otherwise. Gigabit LAN shouldn't be much of an obstacle so I don't expect too much performance loss of doing so. An ideal solution would be the server monitoring changes to the filesystem and notifying clients when their caches need to be flushed but I don't think that NFS supports this. – Tronic Jan 05 '13 at 04:47
  • Since bandwidth is not significant here but *latency*, Gigabit LAN still has some performance impact. FWIW, `lookupcache=none` raised the time for `git clone` from 2.7 sec to 20 sec for me. – Torsten Bronger Aug 27 '15 at 08:43

5 Answers5

26

Depending on what you mean by "false-caching errors", running sync may get you what you need. This will flush all filesystem buffers.

If needed, you can also clear out the VM caches in the kernel using /proc/sys/vm/drop_caches. 

# To free pagecache
echo 1 > /proc/sys/vm/drop_caches

# To free dentries and inodes
echo 2 > /proc/sys/vm/drop_caches

# To free pagecache, dentries and inodes
echo 3 > /proc/sys/vm/drop_caches
JimB
  • 104,193
  • 13
  • 262
  • 255
  • Thank you for reply. I appended what I mean by "false-caching errors" to my question. This problem has not been solved by only `sync`. I think the second of your proposal commands, so I will try later. – Tomoya Kabe Nov 30 '11 at 00:29
  • 1
    I have the same problem, I want to sync on specific folder, lets assume I don't have root, so I can't run the proposed commands – Zamir Dec 30 '12 at 11:55
15

Within a given process, calling opendir() and closedir() on the parent directory of a file invalidates the NFS cache. I used this while programming a job scheduler. Very, very helpful. Try it!

This is the line number of the relevant code (showing the use in context): https://github.com/earonesty/grun/blob/master/grun#L820

It was the only way I could fix the issue of job #1 completing and job #2, which needed some output files, firing off in a context where those files were visible,

famzah
  • 1,462
  • 18
  • 21
Erik Aronesty
  • 11,620
  • 5
  • 64
  • 44
  • Thank you so much for this suggestion – mob Mar 02 '16 at 18:08
  • 2
    `opendir()` definitely works while `sync` doesn't. A simple `ls -la /data/1/kabe/ >/dev/null` would do the trick. – famzah May 04 '21 at 08:40
  • 4
    I have just confirmed this through experiment and it is indeed a very useful piece of knowledge so thanks @erik-aronesty. I can't find any mention of this in the mount options for the nfs client. Do you happen to know how you discovered it? – nickform May 04 '22 at 17:06
  • 1
    And for the benefit of any future visitors wondering about Windows as I was - the same experiments that confirmed this works on Linux came back negative on Windows, i.e. it doesn't work with Windows's NFS client (tested on a fully patched Windows Server 2016 Datacenter today). – nickform May 18 '22 at 09:27
  • @nickform yep, this was tested on linux. be curious if you get the windows version to work, which api does the trick – Erik Aronesty May 20 '22 at 17:38
  • 1
    This almost works for me, but not quite under stress testing. The sequence `d = opendir(...)`, `(void)readdir(d)`, `closedir(d)` appears to be fully reliable. Of course, the `readdir` does add a round-trip call to the NFS server. – Nemo Jul 31 '23 at 22:10
7

AFAIK, the sync and async options aren't the source of attribute caching. Async allows the server to delay saving data to server filesystem, e.g. it affects the write durability in case of NFS server failures, but if the NFS server is stable then async does not affect the NFS clients.

There is a lookupcache=positive NFS mount option that might be used to prevent negative lookup caching, e.g. the NFS returning "No such file or directory" when the file actually exists on the server. See Directory entry caching in man nfs.

ArtemGr
  • 11,684
  • 3
  • 52
  • 85
2

You're seeing the effects of NFS's attribute cache. See man nfs, and check out DATA AND METADATA COHERENCE.

NFS by default caches attributes for a minimum of 30 seconds (acregmin and acdirmin) and a maximum of 60 seconds (acregmax and acdirmax). You can override all of these together with actimeo, or disable the attribute cache entirely with noac. With the noac mount option, the behaviour described by the OP goes away, but hits performance.

lookupcache=positive is useful if you're just looking for the appearance of new files, but deletions will still go through the attribute cache.

Jeff Taylor
  • 306
  • 2
  • 10
  • lookupcache=positive works a little bit different: If pos or positive is specified, the client assumes positive entries are valid until their parent directory's cached attributes expire, but always revalidates negative entries before an application can use them (from the doc: https://linux.die.net/man/5/nfs). – slava Nov 13 '18 at 10:46
-2

clear /var/lib/nfs/rmtab file on nfs server.

The below commands are used to clear memory related problems. and it is very dangerous too. soem times it will crash ur application hosted on the box

# sync

# To free pagecache
echo 1 > /proc/sys/vm/drop_caches

# To free dentries and inodes
echo 2 > /proc/sys/vm/drop_caches

# To free pagecache, dentries and inodes
echo 3 > /proc/sys/vm/drop_caches
Anthon
  • 69,918
  • 32
  • 186
  • 246
Bin
  • 13
  • 1