How to read a file with rows spread on multiple lines?

Question

Rows in my file are spread on multiple lines. In the following chunk of my file, row one starts from 0.0000000000000000E+00 while row two starts from 1.5625000000000000E-02. How do I read numbers from 0.0000000000000000E+00 to one before 1.5625000000000000E-02 as one row?

I am trying numpy's genfromtxt() function and pandas's read_csv() but I haven't yet been able to communicate the system what I indend to do.

#I have put quotation marks here just to indicate the start and end of rows. They
#are not part of the file.

"0.0000000000000000E+00 

00000000     4.9999998882412910E-03       8.7714487508765548E-03  

00000001     5.0000002374872565E-04       5.0877144875087654E-01"


"1.5625000000000000E-02 

00000000     4.9999998882412910E-03       8.4622513106357884E-03 

00000001     5.0000002374872565E-04       5.0864039953085094E-01"

After the correct reading, my input array would look like:

0.0000000000000000E+00   00000000    4.9999998882412910E-03       8.7714487508765548E-03      00000001   5.0000002374872565E-04       5.0877144875087654E-01

1.5625000000000000E-02   00000000    4.9999998882412910E-03       8.4622513106357884E-03      00000001   5.0000002374872565E-04       5.0864039953085094E-01

Have you tried using [this](https://stackoverflow.com/questions/21546739/load-data-from-txt-with-pandas) using `pandas`. The method is to use `pd.read_csv(filename, sep=" ")` — Dillon, Feb 14 '19 at 14:02
@Dillon, it gives "ParserError: Error tokenizing data. C error: Expected 34 fields in line 106, saw 36" — Awais Mirza, Feb 14 '19 at 14:14
@Alderven, it's available at https://www.dropbox.com/s/ewtrbenbpc9gwz4/temp.txt?dl=0. — Awais Mirza, Feb 14 '19 at 14:17
@Christophe, in the data shown here, the number of columns should be 7. If you see the actual [file](https://www.dropbox.com/s/ewtrbenbpc9gwz4/temp.txt?dl=0), the number of columns is 37. — Awais Mirza, Feb 14 '19 at 14:38
@Alderven, the last few lines in my question text above mention the expected output. — Awais Mirza, Feb 14 '19 at 14:39

score 0 · Accepted Answer · answered Feb 14 '19 at 14:12

This should work with the regex package.

text = """

0.0000000000000000E+00 

00000000     4.9999998882412910E-03       8.7714487508765548E-03  

00000001     5.0000002374872565E-04       5.0877144875087654E-01


1.5625000000000000E-02 

00000000     4.9999998882412910E-03       8.4622513106357884E-03 

00000001     5.0000002374872565E-04       5.0864039953085094E-01"""

Code:

import re
xx = re.split(pattern="\n\n\n", string=text)

for xy in xx:
    xy = re.sub(pattern="\s+", repl=" ", string=xy)
    print(xy)
    print("*"*55)

Output:

0.0000000000000000E+00 00000000 4.9999998882412910E-03 8.7714487508765548E-03 00000001 5.0000002374872565E-04 5.0877144875087654E-01
*******************************************************
1.5625000000000000E-02 00000000 4.9999998882412910E-03 8.4622513106357884E-03 00000001 5.0000002374872565E-04 5.0864039953085094E-01
*******************************************************

Alderven · Answer 2 · 2019-02-14T14:44:13.647

Assuming you want to have 7 rows in output data and this is your file. So here is how parse it to pandas dataframe:

import pandas as pd

with open('temp.txt') as f:
    d = f.read().split()

data = {'col1': [], 'col2': [], 'col3': [], 'col4': [], 'col5': [], 'col6': [], 'col7': []}
for i in range(0, len(d), 7):
    for j in range(7):
        data['col{}'.format(j+1)].append(d[j])

df = pd.DataFrame(data)

Output:

Christophe · Answer 3 · 2019-02-14T14:45:00.577

The code below should parse the content of your file correctly:

import re
import pandas

sample = """0.0000000000000000E+00 

00000000     4.9999998882412910E-03       8.7714487508765548E-03  

00000001     5.0000002374872565E-04       5.0877144875087654E-01


1.5625000000000000E-02 

00000000     4.9999998882412910E-03       8.4622513106357884E-03 

00000001     5.0000002374872565E-04       5.0864039953085094E-01
"""


def load_matrix(content):
    lines = (line for line in content.splitlines() if len(line.strip()) > 0)

    rows = list()
    row = list()
    for line in lines:
        fields = line.split()
        is_continuation = re.match(r'^\d{8}$', fields[0])
        if is_continuation:
            row += [float(value) for value in fields[1:]]

        else:
            if (len(row) > 0):
                rows.append(row)

            row = [float(value) for value in fields]

    rows.append(row)
    return pandas.DataFrame(rows)

print(load_matrix(sample))

Displays:

          0      1         2       3         4
0  0.000000  0.005  0.008771  0.0005  0.508771
1  0.015625  0.005  0.008462  0.0005  0.508640

score 0 · Answer 4 · answered Feb 14 '19 at 14:46

Output as two list:

import re

file_object = open("over.txt",'rU')

df1=[]
df2=[]
content = ''

try:
    for line in file_object:
        content = content + line
finally:
     file_object.close()

words = re.split(pattern="\n\n\n", string=content)

num = re.sub(pattern="\s+", repl=",", string=words[0])
for i in num.split(","):
    df1.append(float(i))

num = re.sub(pattern="\s+", repl=",", string=words[1])
for i in num.split(","):
    df2.append(float(i))

print df1
print df2

Output:

[0.0, 0.0, 0.004999999888241291, 0.008771448750876555, 1.0, 0.0005000000237487257, 0.5087714487508765]
[0.015625, 0.0, 0.004999999888241291, 0.008462251310635788, 1.0, 0.0005000000237487257, 0.5086403995308509]

How to read a file with rows spread on multiple lines?

4 Answers4