10

In Scala, how does one uncompress the text contained in file.gz so that it can be processed? I would be happy with either having the contents of the file stored in a variable, or saving it as a local file so that it can be read in by the program after.

Specifically, I am using Scalding to process compressed log data, but Scalding does not define a way to read them in FileSource.scala.

EthanP
  • 1,663
  • 3
  • 22
  • 27
  • 2
    You can extract the content from gzip using Java `java.util.zip. GZipInputStream` and then work with its content – 4lex1v Jul 02 '13 at 22:06
  • 2
    with the better-files library, it's `myFile.newInputStream.gzipped.lines`, according to https://github.com/pathikrit/better-files/blob/master/README.md – Seth Tisue Feb 09 '17 at 16:13

1 Answers1

22

Here's my version:

import java.io.BufferedReader
import java.io.InputStreamReader
import java.util.zip.GZIPInputStream
import java.io.FileInputStream

class BufferedReaderIterator(reader: BufferedReader) extends Iterator[String] {
  override def hasNext() = reader.ready
  override def next() = reader.readLine()
}

object GzFileIterator {
  def apply(file: java.io.File, encoding: String) = {
    new BufferedReaderIterator(
      new BufferedReader(
        new InputStreamReader(
          new GZIPInputStream(
            new FileInputStream(file)), encoding)))
  }
}

Then do:

val iterator = GzFileIterator(new java.io.File("test.txt.gz"), "UTF-8")
iterator.foreach(println)
dhg
  • 52,383
  • 8
  • 123
  • 144
  • 1
    Just wanted to say I used this solution, but the BufferedReaderIterator in this solution produced null lines, due to the nature of reader.ready method. Here is a correct [implementation of an iterator on top of a BufferedReader](http://viewfromthefringe.blogspot.de/2007/10/making-bufferedreader-iterable.html) that I found – Yonatan Wilkof Dec 02 '15 at 21:53
  • 1
    Out of curiosity, what aspect of the file caused the null lines? – dhg Dec 03 '15 at 01:56
  • 2
    It is discussed in this [post](http://stackoverflow.com/questions/5244839/does-bufferedreader-ready-method-ensure-that-readline-method-does-not-return): "...The result of calling ready() tells you absolutely nothing about the content you'll get back from a read() call, and so cannot be used to elide a null check." I am using the iterator to read .gz text files from S3 buckets and some lines from the iterator yielded null objects. Then I changed to the implementation of the iterator that I attached and stopped experiencing this behavior. – Yonatan Wilkof Dec 03 '15 at 16:47