How to pass multiple comment style to skip the header of a text file?

Asked Jun 17 '21 at 14:27

Active Jun 17 '21 at 14:27

Viewed 147 times

I am trying to read hundreds of .dat file by skipping header lines (I do not know how many of them I need to skip beforehand). Header lines very from 1 to 20 and have at beginning either or "$" oder "!". A sample data (left column - node, right column - microstructure) has always two columns and looks like the following:

!===
!Comment
$Material
    1 1.452E-001
    2 1.446E-001
    3 1.459E-001

I tried the following codeline, assuming I know beforehand that there 3 lines in header:

fid = fopen('Graphite_Node_Test.dat') ;
data = textscan(fid,'%f %f','HeaderLines',3) ;
fclose(fid);

This solution works if the number of header lines is known. How can I change the code so that it can read the .dat file without knowing the number of header lines beginning with either "$" or "!" sign?

asked Jun 17 '21 at 14:27

ofey

With a for loop and `fgetl` – Ander Biguri Jun 17 '21 at 14:36
1

`textscan` has the option to treat lines as comments and skip them. Check the docs: https://uk.mathworks.com/help/matlab/ref/textscan.html#btg0kfb – user12339314 Jun 17 '21 at 14:37
textscan does not allow multiple comment style! – ofey Jun 17 '21 at 14:41
@Ander Biguri, thought about that, but with around 100 files and 100 000 lines in each "for" loop will be time consuming. – ofey Jun 17 '21 at 14:44
Alternatively, `fread` the entire thing, post-process it once loaded. – Ander Biguri Jun 17 '21 at 14:52
you could preprocess outside matlab, e.g. this took ~4.5s for 100 files with 500K rows each: `find . -iname '*.dat' -exec sed -i '/^[!$]/d' {} +` – tdy Jun 17 '21 at 22:59
The for loop (Ander mentioned) should of course have a `break` (i.e. use a while loop). Else I agree with `fread` solution. – Solstad Jun 18 '21 at 10:52

How to pass multiple comment style to skip the header of a text file?

0 Answers0