I am trying to access HDFS FileSystem from a web application.
The code looks like this:
Configuration conf = new Configuration();
conf.addResource(new Path("/path/core-site.xml"));
conf.addResource(new Path("/path/hdfs-site.xml"));
conf.addResource(new Path("path/mapred-site.xml"));
Path pathToServices = new Path("/file.txt");
try {
FileSystem fs = FileSystem.get(conf);
InputStreamReader reader = new InputStreamReader(fs.open(pathToServices));
...
...
I call this code from within the servlet, but while the application is being deployed or undeployed from tomcat, i see these messages :
14-Sep-2016 12:42:27.850 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [pipelineUIBackend] created a ThreadLocal with key of type [org.apache.hadoop.hdfs.DFSUtil$1] (value [org.apache.hadoop.hdfs.DFSUtil$1@12fb4cd4]) and a value of type [java.util.Random] (value [java.util.Random@3c6bbf2b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
14-Sep-2016 12:42:27.850 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [pipelineUIBackend] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@498268e]) and a value of type [org.apache.hadoop.fs.FileSystem.Statistics.StatisticsData] (value [2439206 bytes read, 2436769 bytes written, 20 read ops, 0 large read ops, 10 write ops]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
14-Sep-2016 12:42:27.850 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [pipelineUIBackend] created a ThreadLocal with key of type [org.apache.hadoop.io.Text$1] (value [org.apache.hadoop.io.Text$1@4c4d4725]) and a value of type [sun.nio.cs.UTF_8.Encoder] (value [sun.nio.cs.UTF_8$Encoder@2dbe2b60]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
14-Sep-2016 12:42:27.850 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [pipelineUIBackend] created a ThreadLocal with key of type [org.apache.hadoop.io.Text$2] (value [org.apache.hadoop.io.Text$2@50a1c86a]) and a value of type [sun.nio.cs.UTF_8.Decoder] (value [sun.nio.cs.UTF_8$Decoder@154828a6]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
I believe this might be due to FileSystem being used from a web app, but i have no idea how to debug this, and if it is really an issue. Do i need to close the FileSystem object somewhere ?
