Windows file share: why sometimes newly created files aren't visible for some period of time?

Question

We were faced with very strange issue that made us crazy. Sometimes newly created files on our File Share PC were "absent" for some period of time. To reproduce a problem you should have at least two computers, call them alpha and beta. Create file share on beta PC (\\beta\share\bug) and run this PowerShell script from alpha PC:

param(
  $sharePath="\\beta\share\bug"
)
$sharePC = ($sharePath -split '\\')[2]
$session = New-PSSession -ComputerName $sharePC
$counter = 0
while ($true) {
  $fileName = $sharePath + "\$counter.txt"
  Invoke-Command -Session $session -ScriptBlock {
    param(
      $fileName
    )
    "" > $fileName
  } -ArgumentList $fileName
  if (Test-Path $fileName) {
    Write-Host "File $fileName exists" -fore Green
  } else {
    Write-Host "!!! File $fileName does NOT exist!" -fore Red
  }

  $counter = $counter + 1
  Start-Sleep 2
}

After starting this script you should be able to see these messages:

File \\beta\share\bug\1.txt exists
File \\beta\share\bug\2.txt exists
...

And now: Open cmd.exe and run this command:

if exist \\beta\share\bug\foo.txt echo 1

After this during approx 10 seconds you'll see following messages:

!!! File \\beta\share\bug\3.txt does NOT exist!
!!! File \\beta\share\bug\4.txt does NOT exist!

We've discovered that bug is caused by enumerating shared directory where new files are being created. In Python call os.listdir('//beta/share/bug') to reproduce a bug. In C#: Directory.GetDirectories(@"\\beta\share\bug"). You can even simply navigate to share directory by shell and call ls or dir.

Bug were found on Windows Server 2008 R2

Note, that you cannot watch directory content on alpha PC in Windows Explorer in real time, because if you open this directory in Explorer bug would not occur! So ensure to close all such windows before attempts to reproduce a bug. After each script restart you should manually remove all already created files from share (because script is rather stupid and always starts from 0.txt).

We currently have 2 workarounds for this issue:

If client sees this situation, it creates some temporary file in problematic directory - after this files magically appear.
Disable SMB 2.0: http://www.petri.co.il/how-to-disable-smb-2-on-windows-vista-or-server-2008.htm

Does anybody have ever discovered similar problem and can explain why it occurs and how "correctly fix" it?

Thanks

I don't think a network trace is going to help. This seems to be a caching issue on the server side. We have the exact same problem moving from Windows 2003 to 2012R2. It seems SMB shows files for the server that created them, but then shows the files do not yet exist to other servers that query for them. It's pretty bad. — Brain2000, Jan 04 '15 at 02:10

Justin Michael · Accepted Answer · 2017-06-05T23:11:00.227

I was experiencing a similar problem and, eventually, I found the cause of this issue. The specific problem is the SMB2 Directory Cache, which is one of the SMB2 Client Redirector cache components:

This is a cache of recent directory enumerations performed by the client. Subsequent enumeration requests made by client applications as well as metadata queries for files in the directory can be satisfied from the cache. The client also uses the directory cache to determine the presence or absence of a file in the directory and uses that information to prevent clients from repeatedly attempting to open files which are known not to exist on the server. This cache is likely to affect distributed applications running on multiple computers accessing a set of files on a server – where the applications use an out of band mechanism to signal each other about modification/addition/deletion of files on the server.

The default value for this wonderful little cache is 10 seconds, which is producing the behavior you're seeing. When your code asks the system about that directory/file it's getting the cached result, which is 10 seconds old, so it says the file does not exist. Setting HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Lanmanworkstation\Parameters\DirectoryCacheLifetime (DWORD) to a value of 0 will disable the cache and resolve the file doesn't exist issue. Surprisingly this change does not require a restart of the client machine! This will also allow you to keep SMB2 enabled, which should be better for a number of reasons vs. forcing SMB1.

Furthermore, the cache is not used when the share in question is opened in Windows explorer, as having it open tells the system to bypass the cache to keep a live view going. Modifying something in the share via code, however, does not.

~~I think this whole issue is fixed in Windows 2008 R2/7 and higher, but I cannot absolutely confirm it.~~ This is still an issue in modern versions of Windows. See the comments below for details.

It's not fixed in 2008R2 and higher. In fact, it's still an issue in Windows 2012R2. — Brain2000, Jan 04 '15 at 02:11
By the way, this solution fixed out issue for machines accessing Windows 2012R2 through SMB shares. Thank you Justin, your information here has been extremely valuable! — Brain2000, Jan 04 '15 at 02:24
@Brain2000 Thanks for confirming that this is still an issue in later versions. — Justin Michael, Jan 04 '15 at 21:15
Recently started having this issue in Windows 10. This fixed it for me — Edd, Jun 05 '17 at 14:37
Beautiful! Now all I needed is 2 files with .reg extension, one that turns caching on and another to turn it off. Thanks, man! Saved my mental sanity! — dkellner, Mar 11 '19 at 14:19

score 4 · Answer 2 · answered Mar 31 '11 at 18:01

4

A month passed and no answer...

So, we stayed with "Disable SMB 2.0" solution. At least it works.

http://www.petri.co.il/how-to-disable-smb-2-on-windows-vista-or-server-2008.htm

answered Mar 31 '11 at 18:01

Roman

4,531
10
40
69

score 3 · Answer 3 · answered Feb 17 '15 at 19:24

3

There is also a bug in Windows 7 SP1 for which there is a hotfix available

File that a user adds to a remote folder is not displayed in Windows Explorer on a computer that is running Windows 7 or Windows Server 2008 R2

answered Feb 17 '15 at 19:24

user281806

1,020
9
14

score 3 · Answer 4 · answered Dec 09 '16 at 10:08

3

You can use magical suffix $NOCSC$ , instead of disabling SMB or caching via registry keys, as some others suggested. This will allow you to leave all Windows settings intact, but at the same time files will not get cached.

Here is a question specific example: \\beta$NOCSC$\share\bug\1.txt

Check this link if you want more details:

http://blog.wisefaq.com/2016/01/26/nocscno-client-side-caching/

answered Dec 09 '16 at 10:08

zmechanic

1,842
21
27

However, it seems that Win Server 2012 R2 doesn't understand this hint. – satnhak Mar 22 '18 at 07:51

score 2 · Answer 5 · answered Feb 02 '16 at 16:02

2

The easiest way to work around this (as suggested by the OP) is to create a temporary file or subfolder in the folder where you expect the files to appear and remove it right away. This triggers the change to become visible.

We noticed that having a FileSystemWatcher in the folder also helps, even when it does nothing.

answered Feb 02 '16 at 16:02

Michel de Ruiter

7,131
5
49
74

Ludicrous as it sounds, this was the simplest answer for me (I was browsing as a user, not automating) – Basic Mar 08 '16 at 00:25

Windows file share: why sometimes newly created files aren't visible for some period of time?

5 Answers5

Linked