Read a text file line by line and store variables on matching specific pattern in Python

Question

We have a large log file containing following two lines:

00 LOG     |   Cycles Run:  120001
00 LOG     ! Virtual: Max> ?????????? bytes (?.???? gb), Current> 640733184 bytes (?.???? gb).

00 LOG     ! Virtual: Max> 1082470400 bytes (?.???? gb), Current> ????????? bytes (?.???? gb).

00 LOG     ! Actual: Max> ????????? bytes (?.???? gb), Current> 472154112 bytes (?.???? gb).

00 LOG     ! Actual: Max> 861736960 bytes (?.???? gb), Current> ????????? bytes (?.???? gb).

As the log file is big in size, we want to read line by line(not to read whole text in a buffer at a time), match specific set of patterns and pick values in separate variables.

e.g.

00 LOG     |   Cycles Run:  120001

We want o pick 120001 and store in a variable say cycle.

On the other hand we parse these lines:

00 LOG     ! Virtual: Max> ?????????? bytes (?.???? gb), Current> 640733184 bytes (?.???? gb).

00 LOG     ! Virtual: Max> 1082470400 bytes (?.???? gb), Current> ????????? bytes (?.???? gb).

00 LOG     ! Actual: Max> ????????? bytes (?.???? gb), Current> 472154112 bytes (?.???? gb).

00 LOG     ! Actual: Max> 861736960 bytes (?.???? gb), Current> ????????? bytes (?.???? gb).

Characters marked with ? can be any digit.

We want to store vairables like followings:

640733184 in var virtual_cur

1082470400 in var virtual_max

472154112 in var actual_cur

861736960 in var actual_max

Written a snippet in Python 3.6 but it's printing empty list:

import re

filename = "test.txt"
with open(filename) as fp:  
   line = fp.readline()
   while line:
       cycle_num = re.findall(r'00 LOG     |   Cycles Run:  (.*?)',line,re.DOTALL)
       line = fp.readline()

print (cycle_num[0])

NOTE: I want to pick each values in seperate variables and use it later on. Need to set 5 patterns one by one, pick value if it matches any specific pattern and put it inrespective variable.

Not sure about the wildcard matching for the second pattern.

Please suggest us a way to do this efficiently.

Answer 1

With the regex

(?:(?:Cycles Run:[ \t]+)|(?:Current>[ \t]+))(\d+)

Demo

You can do something along these lines:

import re
pat=re.compile(r'(?:(?:Cycles Run:[ \t]+)|(?:Current>[ \t]+))(\d+)')
with open('test.txt','r') as f:   
    for line_num, line in enumerate(f):
        m=pat.search(line)
        if m:
            print(line_num, m.group(0))

Answer 2

You may use an alternation here with two lookbehinds:

(?<=Cycles Run:  )\d+|(?<= Current>  )\d+

Regex demo here.

---

Python example:

import re
text = '''
00 LOG     |   Cycles Run:  120001
00 LOG     !   Virtual: Max> 1082470400 bytes (1.0081 gb), Current>  640733184 bytes (0.5967 gb)
'''

pattern = re.compile(r'(?<=Cycles Run:  )\d+|(?<= Current>  )\d+')
matches = re.findall(pattern,text)
num_cycle = matches[0]
current = matches[1]

print(num_cycle,current)

Prints:

120001 640733184

As you are repeating the process in a loop, it is recommended to use re.compile to compile the pattern only once before the loop.

Answer 3

Here we search for some identifier (like cycles and apply a different regex)

import re
with open('test.txt','r') as f:
    for line in f:
        if re.search(r'Cycles',line):
            m=re.findall(r'\d+$',line)
        else:
            m=re.findall(r'Current>  (\d+)',line)
        print(m)