17

I'm really hoping that the following rings a bell with someone as we're running out of ideas. Answers or suggestions on how to further diagnose would be much appreciated.

We have a Java app that has been running with no problems for 18 months. It is now moving to a new platform running Windows Server 2019 Standard as a VM. On first install everything runs correctly, but periodically the application fails to start and can then only be fixed by re-copying all the jar files. This is temporary as eventually it fails again.

After a lot of monitoring we noticed that there is a Windows process that periodically sets the "L" file attribute on all the files and also creates reparse data. This should not be an issue but once this has happened, the JVM is unable to start the application. (any Windows-whizzes have an idea on what does this?)

A key point is that the app is started by specifying JPMS parameters such as:

java -p MyApp.jar;MyApp_mods -m  mymodule/mypackage.StartGUI

This runs well unit the "L" attribute is set on the jar files and then fails with message:

Error occurred during initialization of boot layer
java.lang.module.FindException: Module format not recognized: MyApp.jar

Renaming MyApp.jar to something else and then copying it back to MyApp.jar fixes the problem as it creates a file without the L attribute and the reparse data (until the process re-applies it)

This behaviour does not just apply to this one operation, but to any where the module system is used such as:

 java --list-modules any-jar-in-the-app.jar

Interestingly(!) if we try a simpler, non-modular app and run as:

java -jar MySimpleApp.jar

then the app runs correctly even with the L attribute set.

Clearly we don't fully understand, but it looks as if running via the module system somehow means that files with the L attribute/reparse data cannot be read (?)

We've tried both the OpenJDK hotspot and OpenJ9 JVMs in various versions but with the same result. Any ideas?

Mark Rotteveel
  • 100,966
  • 191
  • 140
  • 197
Grayman
  • 629
  • 4
  • 14
  • 2
    What is the output of `fsutil reparsePoint query MyApp.jar`? A reparse point is a filesystem object similar to a symbolic link, but more generally supports filesystem extension. Something is clearly changing the attribute, and the first step to debugging is to figure out what kind of RP is being created. – Jim Garrison Apr 14 '21 at 16:50
  • 1
    ...the difference in the scan of module path versus classpath for the files to find out the modules might be the cause for it? [this](https://github.com/openjdk/jdk/blob/05a764f4ffb8030d6b768f2d362c388e5aabd92d/src/java.base/share/classes/jdk/internal/module/ModulePath.java#L248) looks to be the throwing line. – Naman Apr 14 '21 at 18:22
  • fsutil output is: ```Reparse Tag Value : 0x9000601a Tag value: Microsoft Tag value: Directory ...followed by 0x172 bytes of binary data that doesn't include any useful text.``` I'd assumed this was some kind of file system management. My client (a charity) uses a hosting company and I've asked what this process is and whether it can be selectively disabled. – Grayman Apr 15 '21 at 10:01
  • 1
    Naman - thanks for the link. The problem looks to be at line 324 ```if (attrs.isRegularFile()) {```. Once the reparse data has been added, isRegularFile() returns false so there is no match and the exception is thrown. For interest, isSymbolicLink() also returns false and isOther() returns true. Under Windows, Files.readAttributes returns an instance of a class that contains a method isReparsePoint() which can be run via reflection and returns true. I'll do some more experiments and report the behaviour. – Grayman Apr 15 '21 at 10:30

2 Answers2

9

This is only a partial answer but it's worth showing in case someone has the same problem. I'll try to update when there's more info but it could be a while. (update 19/4/21 - bug reported. See JDK-8265439)

As Naman pointed out, the issue is in the JPMS code that looks for module-info in the .jar files. Part of that code gets the file attributes and checks attr.isRegularFile(). If this is false, then the java.lang.module.FindException: Module format not recognized is thrown.

Whenever the Windows file management process runs it creates a reparse point and sets the L attribute on the file. After this, attr.isRegularFile() returns false.

I'm unsure as to exactly what the process is but the Microsoft documentation for reparse tag 0x9000601A is: IO_REPARSE_TAG_CLOUD_6 - Used by the Cloud Files filter, for files managed by a sync engine such as OneDrive. Server-side interpretation only, not meaningful over the wire.

In Java, the class BasicFileAttributes is implemented for Windows by sun.nio.fs.WindowsFileAttributes. Inspection of the source confirms that isRegularFile() will return false if the FILE_ATTRIBUTE_REPARSE_POINT bit is set in the file attributes.

None of this causes a problem if loading .jar files from the classpath so that is the workaround - don't run using JPMS. Having spent a month modularising the app this is not ideal but it works for now.

I think this has to be a bug in JPMS as it makes no sense that .jar files can be loaded via a classpath and not from a module path. I'll gather more info and report it, although reproducing it could be difficult.

Thanks for the responses - they all helped.

Grayman
  • 629
  • 4
  • 14
7

I guess this is a bug.

Reparse Point attribute is used by the NTFS filesystem for implementing symbolic links, implementing mount points and implementing other filesystem techniques like auto moving of less used files to long term storage areas (compressed files/filesystems) / or devices (for example tapes).

"Reparse points and extended attributes are mutually exclusive. The NTFS file system cannot create a reparse point when the file contains extended attributes, and it cannot create extended attributes on a file that contains a reparse point."

So a workaround would be to set an extended attribute at the file to prohibit this behavior / bug.

PS I guess your Windows Server runs some kind of "auto compressing/moving less used files"-service, deactivate this service, when possible.

paladin
  • 765
  • 6
  • 13
  • 1
    "Very definitely" on that last point. Hard-disk drives are *humongous* these days, but apparently that version of Windows didn't get the memo. Also turn off time- and resource-wasters like "virus scans." – Mike Robinson Apr 14 '21 at 17:24
  • Paladin - I think you're right about a bug - see comment above re isRegularFile(). The extended attribute workaround is worth a try, although prior to the L attribute being set the file attributes were "A I" and my belief is that "I" is an extended attribute. I'll try another one. – Grayman Apr 15 '21 at 10:34
  • Paladin again - I set another extended attribute "P" but the process was still able to set a reparse point. Nice idea though. – Grayman Apr 15 '21 at 12:43