Erlang: How to split line with commas but avoiding the ones inside quotes?

Question

Split a string by commas but ignore commas within double-quotes using Javascript gives the regexp /(".*?"|[^",\s]+)(?=\s*,|\s*$)/. How to get use it in Erlang re:split()? The regexp doesn't work with Erlang.

1> S = "20140419,\"Blah blah, foo foo\",1,0,0,0,1,2,0,0".
2> re:split(S, "(\".*?\"|[^\",\s]+,)(?=\s*,|\s*$)", [{return,list}]).
["20140421,","\"Blah blah, foo foo\"",",1,0,0,0,1,2,0,0"]

The result I'm looking for is the list

["20140421","\"Blah blah, foo foo\"","1","0","0","0","1","2","0","0"]

Thanks.

score 2 · Accepted Answer · answered Apr 22 '14 at 13:32

Just translate the JavaScript regular expression to Erlang:

Erlang R16B03-1 (erts-5.10.4) [source] [64-bit] [smp:8:8] [async-threads:10] [kernel-poll:false]

Eshell V5.10.4  (abort with ^G)
1> S = "20140419,\"Blah blah, foo foo\",1,0,0,0,1,2,0,0".
"20140419,\"Blah blah, foo foo\",1,0,0,0,1,2,0,0"
2> {ok,R} = re:compile("(\".*?\"|[^\",\\s]+)(?=\\s*,|\\s*$)").
{ok,{re_pattern,1,0,
                <<69,82,67,80,122,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,0,0,0,
                  ...>>}}
3> {match,Matches} = re:run(S, R, [{capture,[1],list},global]).
{match,[["20140419"],
        ["\"Blah blah, foo foo\""],
        ["1"],
        ["0"],
        ["0"],
        ["0"],
        ["1"],
        ["2"],
        ["0"],
        ["0"]]}
4> [M || [M] <- Matches].
["20140419","\"Blah blah, foo foo\"","1","0","0","0","1",
 "2","0","0"]

In shell command 2 note the use of double backslashes in the pattern to specify \s correctly.

score 1 · Answer 2 · 2014-04-21T18:14:41.653

Came up with a parser using pattern matching. Adding it here in case anyone finds it useful.

parse_csv(String) -> parse_csv(String, [], [], [], false).

parse_csv([], S, Acc, [], _) -> lists:reverse(lists:map(fun(X) -> lists:reverse(lists:flatten(X)) end, [Acc|S]));
parse_csv([], S, [], L, _) -> lists:reverse(lists:map(fun(X) -> lists:reverse(lists:flatten(X)) end, [L|S]));
parse_csv(String, S, Acc, L, IsSubStr) ->
    case String of
        [$"|T] when IsSubStr =:= true ->
            % end of substring (ending quote).
            parse_csv(T, S, Acc, [$"|L], false);
        [$"|T] when IsSubStr =:= false  ->
            % beginning of a substring (beginning quote).
            parse_csv(T, S, Acc, [$"], true);
        [$,|T] when IsSubStr =:= true andalso L =/= [] ->
            % comma within a substring
            parse_csv(T, S, Acc, [$,|L], true);
        [$,|T] when IsSubStr =:= false andalso L =/= [] ->
            % comma after a substring.
            parse_csv(T, [[L|Acc]|S], [], [], false);
        [$,|T] when IsSubStr =:= false andalso L =:= [] ->
            % comma after a normal string.
            parse_csv(T, [Acc|S], [], [], false);
        [H|T] when IsSubStr =:= true ->
            % within a substring
            parse_csv(T, S, Acc, [H|L], true);
        [H|T] when IsSubStr =:= false ->
            % a normal string
            parse_csv(T, S, [H|Acc], [], false) end.

Example:

2> ql:parse_csv("foo,\"bar aa\",blah,\"dooo\",phew").                          
["foo","\"bar aa\"","blah","\"dooo\"","phew"]
3> ql:parse_csv("foo,bar,baz").
["foo","bar","baz"]
4> ql:parse_csv("foo,\"foo, bar\",baz").
["foo","\"foo, bar\"","baz"]

fenollp · Answer 3 · 2014-04-21T23:55:50.620

1

How about this:

1> string:tokens(S, ",").
["20140419","\"Blah blah"," foo foo\"","1","0","0","0","1","2","0","0"]

Or even:

2> re:split(S, ",", [{return,list}]).
["20140419","\"Blah blah"," foo foo\"","1","0","0","0","1","2","0","0"]

string:tokens/2's doc.

(@kadaj, that is seriously a lot of code to parse CSVs)

Edit: to properly answer the question, one needs to reassemble "\"…", "…\"" pairs. To do so, a trivial recursive function will do:

finish([[$\"]++Rest=M, Scnd|T], Acc) ->
    finish(T, [M++Scnd|Acc]);
finish([H|T], Acc) ->
    finish(T, [H      |Acc]);
finish([], Acc) ->
    lists:reverse(Acc).

edited Apr 21 '14 at 23:55

answered Apr 21 '14 at 18:06

fenollp

2,406
1
15
14

Nope, the regexp you gave splits a string in between. `"foo,\"foo, bar \",baz"` should give ["foo", "\"foo, bar\"", "baz"]. But simply splitting it on a comma gives `["foo","\"foo"," bar\"","baz"]`. Think that a field which contains a string contains commas inside it. So you need to ignore commas that are within quotes. – Apr 21 '14 at 18:15
1

Well, now you only need to pattern-match and fuse the elements starting with `$"`, like so: `finish([$" ++Rest=M,Scnd|T], Acc) -> finish(T, [M++Scnd|Acc]); finish([H|T], Acc) -> finish(T,[H|Acc]); finish([], Acc) -> lists:reverse(Acc).` – fenollp Apr 21 '14 at 19:11
Right, that's another approach to the problem. Not "parser" per se, but that will do the job. You should update your answer. The code you gave gives an `illegal pattern error` though. – Apr 21 '14 at 23:42

Erlang: How to split line with commas but avoiding the ones inside quotes?

3 Answers3