Extracting data from thousands of lines

Question

I have an .obj file, so far I have been using a messy tokenizer to split the lines. I've found this is really inefficient.

public static String getSpecificToken(String s, int t) {
    Scanner tokens = new Scanner(s);
    String token = "";
    
    for (int i = 0; i < t; i++) {
        if(!tokens.hasNext()) { break; }
        token = tokens.next();
    }
    
    tokens.close();
    return token;
}

The object file formatting looks like this, I struggle to find the most efficient way to split this.

v 1.4870 0.3736 2.2576

v 1.5803 0.3451 2.1859

v 1.6275 0.3111 2.2261

v 1.6343 0.0783 2.4352

v 1.5180 0.0644 2.5398

v 1.4568 0.0720 2.5205

v 1.3953 0.0795 2.5013

Please convince us of the inefficiency with actual benchmark results. I don't see any obvious signs of inefficient code. — Marko Topolnik, Oct 13 '12 at 08:54

Rohit Jain · Answer 1 · 2012-10-13T08:58:25.913

3

Simply use String#split to split them on space, no need to use Tokenizer here. In fact you should avoid using Tokenizer as far as possible: -

Scanner tokens = new Scanner(s);
String[] tokenArr = tokens.split("\\s+");  // Split on 1 or more space

for (String token: tokenArr) {
    System.out.println(token);
}

edited Oct 13 '12 at 08:58

answered Oct 13 '12 at 08:52

Rohit Jain

209,639
45
409
525

But won't that split them by one space only? There can be multiple spaces. – Grandstack Oct 13 '12 at 08:54
@user1650289. You can use Regex as Hristo said. I have edited the code. – Rohit Jain Oct 13 '12 at 08:58
Do you honestly expect this to improve performance? But maybe OP is just using wrong terminology. – Marko Topolnik Oct 13 '12 at 09:02
2

I'd be quite surprised if that increased performance by any noticeable amount - actually I'd assume it'd less efficient... – Voo Oct 13 '12 at 09:05
@Voo on second reading, OP is probably not even talking about performance, or is conflating it with code conciseness. – Marko Topolnik Oct 13 '12 at 09:09
1

@user1650289 Split and scanner both use regex, which aren't necessary for the problem - just writing a trivial handwritten parser should beat both of them easily by a good amount thanks to that. But obviously only benchmarking can answer that question definitely. – Voo Oct 13 '12 at 09:14
@Voo Do you have an example, I don't know where to start. Thanks. – Grandstack Oct 13 '12 at 09:22

score 3 · Answer 2 · edited May 23 '17 at 12:27

3

This example compares java.util.Scanner with java.io.StreamTokenizer, suggesting a slight edge for the latter. You might be able to use a similar approach to profile your use case.

edited May 23 '17 at 12:27

Community

1
1

answered Oct 13 '12 at 09:10

trashgod

203,806
29
246
1,045

Extracting data from thousands of lines

2 Answers2