How to read datas from several files with python

Question

I am new to python and this may be a foolish question but it troubles me for several days. I have about 30 log files, and each of them contains strings and data. They are almost the same except the difference of several data, and their names are arranged regularly, like 'log10.lammps', 'log20.lammps', etc.(The '10''20'represent the temperature of the simulation). I want to write a python script, which loops all these files and read their data in a specific line, (say line3900). Then I want to write these data to another data file, which is arranged like these:

10 XXX 20 XXX 30 XXX . . .

I can read and write from a single file, but I cannot achieve the loop. Could anyone please tell me how to to that. Thanks a lot!

PS. Still another difficulty is that the data in line 3900 is presented like this: "The C11 is 180.1265465616", the one I want to extract is 180.1265465616. How can I extract the number without the strings?

Welcome to Stack Overflow. People here are willing to help. But we want you to show us what you've tried first. How far have you come? Do you have any code? — Niclas Nilsson, Jan 16 '13 at 19:10

score 0 · Answer 1 · answered Jan 16 '13 at 19:15

0

#assuming files is a list of filenames
for filename in files:
    with open(filename) as f:
        <do stuff with file f>

ps. float(line.split(' ')[-1])

answered Jan 16 '13 at 19:15

cmd

5,754
16
30

score 0 · Accepted Answer · edited May 23 '17 at 12:18

This answer covers how to get all the files in a folder in Python. To summarize the top answer, to get all the files in a single folder, you would do this:

import os
import os.path

def get_files(folder_path):
    return [f for f in os.listdir(folder_path) if os.path.isfile(os.path.join(folder_path, f))]

The next step would be to extract the number from the line The C11 is 180.1265465616. I'm assuming that you have a function called get_line that given a filename, will return that exact line.

You can do one of three things. If the length of the number at the end is constant, then you can just grab the last n characters in the string and convert it to a number. Alternatively, you could split the string by the spaces and grab the last item -- the number. Finally, you could use Regex.

I'm just going to go with the second option since it looks the most straightforward for now.

def get_numbers():
    numbers = []
    for file in get_files('folder'):
        line = get_line(file)
        components = line.split(' ')
        number = float(components[-1])
        numbers.append(number)
    return numbers

I wasn't sure how you wanted to write the numbers to the file, but hopefully these should help you get started.

Niclas Nilsson · Answer 3 · 2013-01-16T20:35:58.917

Well, I can give you a hint which path I would have taken (but there might be a better one):

Get all files in a directory to a list with os.listdir
Loop over every one and perform the following:
- Use the re module to extract the temperature from the filename (if not match pattern break) else add it to a list (to_write_out)
- Read the right line with linecache
- Get the value (line.split()[-1])
- Append the value to the list to_write_out.
Join the list to_write_out to a string with join
Write the string to a file.

Help with the regex

The regular expressions can be a bit tricky if you haven't used them before. To extract the temperature from your filenames (first bullet beneath point number 2) you would use something like:

for fname in filenames:
    pattern = 'log(\d+)\.lammps'
    match = re.search(pattern, fname)
    if match:
        temp = match.group(1)

        # Append the temperature to the list.

    else:
        break

    # Continue reading the right line etc.

How to read datas from several files with python

3 Answers3

Help with the regex