Use Scala parser combinator to parse CSV files

Question

I'm trying to write a CSV parser using Scala parser combinators. The grammar is based on RFC4180. I came up with the following code. It almost works, but I cannot get it to correctly separate different records. What did I miss?

object CSV extends RegexParsers {
  def COMMA   = ","
  def DQUOTE  = "\""
  def DQUOTE2 = "\"\"" ^^ { case _ => "\"" }
  def CR      = "\r"
  def LF      = "\n"
  def CRLF    = "\r\n"
  def TXT     = "[^\",\r\n]".r
  
  def file: Parser[List[List[String]]] = ((record~((CRLF~>record)*))<~(CRLF?)) ^^ { 
    case r~rs => r::rs
  }
  def record: Parser[List[String]] = (field~((COMMA~>field)*)) ^^ {
    case f~fs => f::fs
  }
  def field: Parser[String] = escaped|nonescaped
  def escaped: Parser[String] = (DQUOTE~>((TXT|COMMA|CR|LF|DQUOTE2)*)<~DQUOTE) ^^ { case ls => ls.mkString("")}
  def nonescaped: Parser[String] = (TXT*) ^^ { case ls => ls.mkString("") }

  def parse(s: String) = parseAll(file, s) match {
    case Success(res, _) => res
    case _ => List[List[String]]()
  }
}


println(CSV.parse(""" "foo", "bar", 123""" + "\r\n" + 
  "hello, world, 456" + "\r\n" +
  """ spam, 789, egg"""))

// Output: List(List(foo, bar, 123hello, world, 456spam, 789, egg)) 
// Expected: List(List(foo, bar, 123), List(hello, world, 456), List(spam, 789, egg))

Update: problem solved

The default RegexParsers ignore whitespaces including space, tab, carriage return, and line breaks using the regular expression [\s]+. The problem of the parser above unable to separate records is due to this. We need to disable skipWhitespace mode. Replacing whiteSpace definition to just [ \t]} does not solve the problem because it will ignore all spaces within fields (thus "foo bar" in the CSV becomes "foobar"), which is undesired. The updated source of the parser is thus

import scala.util.parsing.combinator._

// A CSV parser based on RFC4180
// https://www.rfc-editor.org/rfc/rfc4180

object CSV extends RegexParsers {
  override val skipWhitespace = false   // meaningful spaces in CSV

  def COMMA   = ","
  def DQUOTE  = "\""
  def DQUOTE2 = "\"\"" ^^ { case _ => "\"" }  // combine 2 dquotes into 1
  def CRLF    = "\r\n" | "\n"
  def TXT     = "[^\",\r\n]".r
  def SPACES  = "[ \t]+".r

  def file: Parser[List[List[String]]] = repsep(record, CRLF) <~ (CRLF?)

  def record: Parser[List[String]] = repsep(field, COMMA)

  def field: Parser[String] = escaped|nonescaped


  def escaped: Parser[String] = {
    ((SPACES?)~>DQUOTE~>((TXT|COMMA|CRLF|DQUOTE2)*)<~DQUOTE<~(SPACES?)) ^^ { 
      case ls => ls.mkString("")
    }
  }

  def nonescaped: Parser[String] = (TXT*) ^^ { case ls => ls.mkString("") }



  def parse(s: String) = parseAll(file, s) match {
    case Success(res, _) => res
    case e => throw new Exception(e.toString)
  }
}

Why are the constants defined with def and not with val? Is there a benefit to it? — Sebastian N., Dec 29 '12 at 20:48
[Check this out](http://stackoverflow.com/questions/8240720/what-are-the-implications-of-using-def-vs-val-for-constant-values). **tl;dr** def uses less memory, val is faster. — rancidfishbreath, Feb 08 '13 at 16:55
For compile-time constants there's really little difference - a "val" will initialize a field with that constant in the constructor then create a method which returns its value, while a "def" will simply return the constant - and for compile-time constants this is effectively free. — Score_Under, May 02 '13 at 10:41
@rancidfishbreath it's an object so there's only 1 instance so if it saves any memory, this will be negligible — herman, Dec 23 '13 at 23:34

score 30 · Accepted Answer · answered Feb 21 '11 at 08:16

30

What you missed is whitespace. I threw in a couple bonus improvements.

import scala.util.parsing.combinator._

object CSV extends RegexParsers {
  override protected val whiteSpace = """[ \t]""".r

  def COMMA   = ","
  def DQUOTE  = "\""
  def DQUOTE2 = "\"\"" ^^ { case _ => "\"" }
  def CR      = "\r"
  def LF      = "\n"
  def CRLF    = "\r\n"
  def TXT     = "[^\",\r\n]".r

  def file: Parser[List[List[String]]] = repsep(record, CRLF) <~ opt(CRLF)
  def record: Parser[List[String]] = rep1sep(field, COMMA)
  def field: Parser[String] = (escaped|nonescaped)
  def escaped: Parser[String] = (DQUOTE~>((TXT|COMMA|CR|LF|DQUOTE2)*)<~DQUOTE) ^^ { case ls => ls.mkString("")}
  def nonescaped: Parser[String] = (TXT*) ^^ { case ls => ls.mkString("") }

  def parse(s: String) = parseAll(file, s) match {
    case Success(res, _) => res
    case _ => List[List[String]]()
  }
}

answered Feb 21 '11 at 08:16

psp

12,138
1
41
51

1

How does that differ from `protected val whiteSpace = """\s+""".r`, which is `RegexParsers`'s default? -- Ah, got it. Newline is space as well, so your override removed it from consideration. – Daniel C. Sobral Feb 21 '11 at 14:50
Thank you very much for pointing out the white space issue! Your solution correctly parses different records. However it also ignores spaces within fields. Please see my updated question to see my solution after adopting your changes. – Rio Feb 21 '11 at 22:51
Change the CRLFs in file to CRLF|LF for both of them if you want to support non-windows line feeds (it's just \n in Linux) – djsumdog Jan 22 '14 at 23:00

Maciej Biłas · Answer 2 · 2015-04-16T19:47:48.423

7

With Scala Parser Combinators library out of the Scala standard library starting from 2.11 there is no good reason not to use the much more performant Parboiled2 library. Here is a version of the CSV parser in Parboiled2's DSL:

/*  based on comments in https://github.com/sirthias/parboiled2/issues/61 */
import org.parboiled2._
case class Parboiled2CsvParser(input: ParserInput, delimeter: String) extends Parser {
  def DQUOTE = '"'
  def DELIMITER_TOKEN = rule(capture(delimeter))
  def DQUOTE2 = rule("\"\"" ~ push("\""))
  def CRLF = rule(capture("\r\n" | "\n"))
  def NON_CAPTURING_CRLF = rule("\r\n" | "\n")

  val delims = s"$delimeter\r\n" + DQUOTE
  def TXT = rule(capture(!anyOf(delims) ~ ANY))
  val WHITESPACE = CharPredicate(" \t")
  def SPACES: Rule0 = rule(oneOrMore(WHITESPACE))

  def escaped = rule(optional(SPACES) ~
    DQUOTE ~ (zeroOrMore(DELIMITER_TOKEN | TXT | CRLF | DQUOTE2) ~ DQUOTE ~
    optional(SPACES)) ~> (_.mkString("")))
  def nonEscaped = rule(zeroOrMore(TXT | capture(DQUOTE)) ~> (_.mkString("")))

  def field = rule(escaped | nonEscaped)
  def row: Rule1[Seq[String]] = rule(oneOrMore(field).separatedBy(delimeter))
  def file = rule(zeroOrMore(row).separatedBy(NON_CAPTURING_CRLF))

  def parsed() : Try[Seq[Seq[String]]] = file.run()
}

edited Apr 16 '15 at 19:47

answered Jul 11 '14 at 16:02

Maciej Biłas

1,966
2
18
20

2

Since you went through the effort of writing such a nice blog about it, we might as well post the link here :-) http://maciejb.me/2014/07/11/a-csv-parser-moving-from-scala-parser-combinators-to-parboiled2/ – harschware Oct 14 '14 at 20:56
1

Shouldn't `CRLF = rule(capture("\n\r" | "\n"))` be `CRLF = rule(capture("\r\n" | "\n"))`? and again for `NON_CAPTURING_CRLF`? – Toby Apr 08 '15 at 14:07
@Toby of course it should! Thank you for pointing that out, I've corrected the answer. – Maciej Biłas Apr 13 '15 at 09:07
Great stuff. Shouldn't it support (double) quoted values out of the box? Looks to me like it should but it doesn't parse it as I'd expect. ie, "a,b", "c" – Toby Apr 15 '15 at 11:04
@Toby it sure should! I've fixed that one as well. :-) – Maciej Biłas Apr 16 '15 at 19:49

score 3 · Answer 3 · answered Feb 21 '11 at 15:02

3

The default whitespace for RegexParsers parsers is \s+, which includes new lines. So CR, LF and CRLF never get a chance to be processed, as it is automatically skipped by the parser.

answered Feb 21 '11 at 15:02

Daniel C. Sobral

295,120
86
501
681

Use Scala parser combinator to parse CSV files

Update: problem solved

3 Answers3

Linked