Really fast Java function to split strings without affecting quoted strings?

Question

I need a string splitting function that is really fast that will break apart a comma delimited string without breaking apart strings encased in double-quotes that contain commas. Is there a function that does this? If it's best handled by a regular expression, please indicate the necessary pattern and if applicable, tell me any speed optimization tips I should know about. For example, if there's a way to invoke the regular expression in such a way that does not require the regular expression pattern to be reevaluated every time, etc. This function will be called thousands of times in a short period of time.

Note, I did see the regular expression posts on SO like this one:

Regular Expression To Split On Comma Except If Quoted

But they were C# and other languages and not Java. Also, if there is a non-regular expression method that is faster I'd like to know about it as I indicated above.

-- roschler

You have over 30 questions without an accepted answer. Perhaps you should ask more questions which can be answered. ;) — Peter Lawrey, Jul 10 '11 at 06:36

score 6 · Accepted Answer · answered Jul 10 '11 at 00:38

6

It sounds like you are trying to parse a CSV formatted strings/files?

If so, maybe you don't have to write the code yourself. Checkout the apache commons library for CSV parsing:

http://commons.apache.org/sandbox/csv/

answered Jul 10 '11 at 00:38

Moe Matar

2,026
13
7

3

+1 Btw, I couldn't help thinking of the Microsoft Paperclip when reading your answer :) – James P. Jul 10 '11 at 00:46
Thanks for the link Moe, I'll have a look. – Robert Oschler Jul 10 '11 at 01:16

score 2 · Answer 2 · answered Jul 10 '11 at 00:40

You can basically rip off the C# code from the linked question, but you need to undo it's iterator stuff, replacing yield return with, say, appending to a list:

public static List<String> SplitCSV(String csvString)
    StringBuilder sb = new StringBuilder();
    boolean quoted = false;

    List<String> list = new ArrayList<String>();

    for(char c : csvString.toCharArray()) {
        if (quoted) {
            if (c == '"')
                quoted = false;
            else
                sb.append(c);
        } else {
            if (c == '"') {
                quoted = true;
            } else if (c == ',') {
                list.add(sb.toString());
                sb = new StringBuilder();
            } else {
                sb.append(c);
            }
        }
    }

    if (quoted)
        throw new IllegalArgumentException("csvString: Unterminated quotation mark.");

    list.add(sb.toString());
    return list;
}

Note that this, of course, won't deal with escaped quotes in quoted strings...

score 1 · Answer 3 · edited Jul 17 '12 at 14:48

1

I think the most popular libraries for Java that do this naturally are supercsv and opencsv. Are you looking for a non-library solution?

edited Jul 17 '12 at 14:48

Eyal

3,412
1
44
60

answered Jul 10 '11 at 00:37

Ray Toal

86,166
18
182
232

score 0 · Answer 4 · answered Jul 18 '12 at 02:04

There's also StrTokenizer in the commons-lang library as well:

StrTokenizer tokenizer = StrTokenizer.getCSVInstance();
tokenizer.reset(input);
String tokens[] = tokenizer.getTokenArray();

There's also a method to get tokens as a list, and it implements The Iterator/ListIterator functions so you can use it in an iterator style while loop.

You can also keep calling the "reset" method to clear the instance, and parse new input data.

One thing to note is that OpenCSV words with Reader instances, and will parse across multiple lines. This class works with strings or char arrays and parses only a single record. It does have some memory overhead in that all the parsing is done upfront when you ask for the first token.

It is however, more configurable than OpenCSV.

DISCLOSURE: I contributed the original version of this class to the library.

Really fast Java function to split strings without affecting quoted strings?

4 Answers4