Parse log file with python pandas

Question

Problem 1

I am trying to read and parse a log file from a simulation. Ideally I would like to do this with pandas but I am having issues. The file in question is called log and a sample is contained below. Now I try to do

     import pandas as pd
     import numpy as np
     import csv
    
     # This method works:
     results = []
     with open('log') as inputfile:
             for row in csv.reader(inputfile):
                 results.append(row)
    
   
      # This method doesn't work:
      pd.read_csv('log')

The error is

pandas.parser.CParserError: Error tokenizing data. C error: Expected 1 fields in line 77, saw 4

Why is this happening?

Problem 2

What is the most efficient way to parse the log file? Please keep in mind that the file will in reality be very large. Basically I want to parse the residuals for each time step (each time step is designated by Time= x) into a numpy array for plotting. Eg

smoothSolver:  Solving for gamma, Initial residual = 0.813857557031, Final residual = 6.74558898691e-06, No Iterations 2

I would want to append 0.813857557031 into an array res_gamma_init

What is the most efficient way to do this?

/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2.3.x                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 2.3.x-590d57f32fed
Exec   : simpleFoam
Date   : Oct 30 2015
Time   : 17:24:08
Host   : "jack"
PID    : 4034
Case   : /home/jack/OpenFOAM/jack-2.3.x/run/sim
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0

Reading field p

Reading field U

Reading/calculating face flux field phi

Selecting incompressible transport model Newtonian
Selecting RAS turbulence model kOmegaSST
Using gammaReThetat-correlations by Langtry and Menter (2009)
gammaReThetatSSTCoeffs
{
    ca1             2;
    ce1             1;
    ca2             0.06;
    ce2             50;
    cThetat         0.03;
    sigmaf          1;
    sigmaThetat     2;
    s1              2;
    dUds            false;
    alphaK1         0.85034;
    alphaK2         1;
    alphaOmega1     0.5;
    alphaOmega2     0.85616;
    gamma1          0.5532;
    gamma2          0.4403;
    beta1           0.075;
    beta2           0.0828;
    betaStar        0.09;
    a1              0.31;
    c1              10;
    kInf            0;
    omegaInf        0;
}

No finite volume options present


SIMPLE: convergence criteria
    field p  tolerance 1e-06
    field U  tolerance 1e-06
    field "(k|omega|ReThetaTilda|nut|gamma)"     tolerance 1e-06


Starting time loop


forceCoeffs forceCoeffs:
    Not including porosity effects

Time = 1

smoothSolver:  Solving for Ux, Initial residual = 0.999999999388, Final residual = 0.00692443749034, No Iterations 2
smoothSolver:  Solving for Uy, Initial residual = 0.999999994742, Final residual = 0.0027182321684, No Iterations 2
DICPCG:  Solving for p, Initial residual = 1, Final residual = 0.00976261323556, No Iterations 326
time step continuity errors : sum local = 8.88353600919, global = -0.154492041821, cumulative = -0.154492041821
smoothSolver:  Solving for omega, Initial residual = 1, Final residual = 0.0110458877993, No Iterations 2
smoothSolver:  Solving for k, Initial residual = 0.999999999804, Final residual = 0.022014322402, No Iterations 2
bounding k, min: 0 max: 0.415754458684 average: 0.410526729282
smoothSolver:  Solving for ReThetatTilda, Initial residual = 1, Final residual = 0.0259070897862, No Iterations 2
smoothSolver:  Solving for gamma, Initial residual = 1, Final residual = 1.51565684699e-05, No Iterations 2
ExecutionTime = 1.41 s  ClockTime = 1 s

forceCoeffs forceCoeffs output:
    Cm    = 0
    Cd    = -19811.1501384
    Cl    = 4.01861806726e-15
    Cl(f) = 2.00930903363e-15
    Cl(r) = 2.00930903363e-15

Time = 2

smoothSolver:  Solving for Ux, Initial residual = 0.510606164486, Final residual = 0.0151751146151, No Iterations 2
smoothSolver:  Solving for Uy, Initial residual = 0.570639175137, Final residual = 0.0164524779792, No Iterations 2
DICPCG:  Solving for p, Initial residual = 0.160667758287, Final residual = 0.00156013775625, No Iterations 143
time step continuity errors : sum local = 88.8245306208, global = 11.6476210117, cumulative = 11.4931289699
smoothSolver:  Solving for omega, Initial residual = 0.531389189498, Final residual = 0.00899532326598, No Iterations 2
smoothSolver:  Solving for k, Initial residual = 0.562103650426, Final residual = 0.0138022250836, No Iterations 2
bounding k, min: 0 max: 0.425403200237 average: 0.411310758596
smoothSolver:  Solving for ReThetatTilda, Initial residual = 0.301017401945, Final residual = 0.00693946297375, No Iterations 2
smoothSolver:  Solving for gamma, Initial residual = 0.813857557031, Final residual = 6.74558898691e-06, No Iterations 2
ExecutionTime = 2.13 s  ClockTime = 2 s

forceCoeffs forceCoeffs output:
    Cm    = 0
    Cd    = -20283.0256988
    Cl    = 3.99364280461e-15
    Cl(f) = 1.9968214023e-15
    Cl(r) = 1.9968214023e-15

Time = 3

Maybe one these answers solves your first problem: http://stackoverflow.com/questions/29754786/python-3-pandas-error-pandas-parser-cparsererror-error-tokenizing-data-c-erro — BigZ, Oct 30 '15 at 18:18
I guess line 77 isn't clean csv. You could try to add the attribute skiprows=77 to the function call .read_csv() — BigZ, Oct 30 '15 at 18:24
I'll give it a try. I'm still puzzled why the alternative reader I give above works though. Thanks. — Dipole, Oct 30 '15 at 18:26
Thats strange indeed, but i'm not very familiar with pandas. So i hope someone will give an actual answer to your questions. — BigZ, Oct 30 '15 at 18:28

score 1 · Accepted Answer · answered Oct 30 '15 at 18:35

1

To answer your second question. This may help, althought the regular expression could be nicer.

line1 = 'smoothSolver:  Solving for Ux, Initial residual = 0.999999999388, Final residual = 0.00692443749034, No Iterations 2'
line2 = 'smoothSolver:  Solving for Uy, Initial residual = 0.999999994742, Final residual = 0.0027182321684, No Iterations 2'
test = [line1, line2]
import re
for line in test:
    if line.startswith('smoothSolver'):
        print re.findall(r'[residual = ][\d][.][\d]+', line)

The result in the console is:

[' 0.999999999388', ' 0.00692443749034']
[' 0.999999994742', ' 0.0027182321684']

answered Oct 30 '15 at 18:35

BigZ

836
8
19

Great! Ill see if there is any point in using pandas if I can just use the alternative method. – Dipole Oct 30 '15 at 18:36
1

I guess that would be the pythonic way of doing it. Short and simple :) – BigZ Oct 30 '15 at 18:39
Can you post an example how the scientific notation looks like in the file? – BigZ Oct 30 '15 at 19:43
I found a way with RegExp eventually - these things make my head hurt. Thanks for all your help. I'll be happy to accept your answer unless someone knows a way to do this easily with pandas. – Dipole Oct 30 '15 at 19:48
No problem. Glad i could help and i hope someone comes with an actual answer to your question – BigZ Oct 30 '15 at 19:59

Parse log file with python pandas

Problem 1

Problem 2

1 Answers1