Dynamic parsing of fixed text files

Question

I want to build a parser for fixed position text files.

What I want to achieve is to make it dynamic so that I could pass an external configuration file containing the format of the file that will be parsed.

Example of configuration file to make the application to load:

Field; Position
Name;0-20
Surname;21-40
Age;40-42
Sex;42-43
...

Example of file to parse:

John William            Hoover23M
Deborah                 Foobar33F
...

I saw googling lot of libraries to parse fixed length file.

Problem is that all of them relies on creating some classes with annotated fields telling the fixed position in the text file.

I want to make a generic parser so this classes should be automatically generated and annotated based on some external configuration file.

Do you know any library or different kind of approach that I could follow?

I'm talking about parsing relatively big files around ~500Mb so also efficiency and speed is important factor.

Thank you all!

score 0 · Accepted Answer · answered Dec 04 '15 at 13:52

0

You dont need to "parse" the big file. You only need to extract at given positions

1 parse the "format" file, with classical regex, and store name, positions in an array. Time doesnt matter there.

2 open your big file, read the lines, and extract at the positions you want. It will be the faster your could do.

answered Dec 04 '15 at 13:52

guillaume girod-vitouchkina

3,061
1
10
26

Really I would need something like this: https://northconcepts.com/docs/examples/read-a-fixed-width-file-fixed-length-record-file/ but this library has to be paid. Does anybody knows something free? Unless I will write myself... – user1821021 Dec 04 '15 at 16:30

score 0 · Answer 2 · answered Jun 24 '16 at 09:52

Try uniVocity-parsers' FixedWidthParser:

    //define field lengths
    FixedWidthFields fields = new FixedWidthFields();
    accountFields.addField("ID", 10); 
    accountFields.addField("Bank", 8);
    accountFields.addField("AccountNumber", 15);
    accountFields.addField("Swift", 12);

    //configure the parser
    FixedWidthParserSettings settings = new FixedWidthParserSettings(fields); //many options here, check the tutorial
    settings.getFormat().setLineSeparator("\n");

    //We can now parse all rows
    FixedWidthParser parser = new FixedWidthParser(settings);
    List<String[]> rows = parser.parseAll(new File("/path/to/file.txt"));

This is just a rough example. There are many other examples here.

Disclosure: I'm the author of this library, it's open-source and free (Apache 2.0 License)

Dynamic parsing of fixed text files

2 Answers2