6

Typical start of working with HDFS using org.apache.hadoop.fs classes on scala is

val conf = new Configuration()
fs = FileSystem.get(conf)

Do we need to call something like IOUtils.closeStream(fs) or fs.close() at the end?

I've found the single answer relating to this topic https://groups.google.com/forum/#!topic/nosql-databases/Mn9KLbtmh7M

In all my experience you let FileSystem instances close themselves.

I am not sure I can rely on it.

Binary Nerd
  • 13,872
  • 4
  • 42
  • 44
MaSEL
  • 505
  • 1
  • 5
  • 20
  • This seems to cover your question: http://stackoverflow.com/questions/20057881/hadoop-filesystem-closed-exception-when-doing-bufferedreader-close – Binary Nerd Aug 09 '16 at 12:19

1 Answers1

0

You can always use try with resources:

try (fs = FileSystem.get(conf)) {
  < your code>
}
    

Since FileSystem is closeable it will be closed at the end of the "try" scope

BDL
  • 21,052
  • 22
  • 49
  • 55