How do I parse extremely large file in python?

Question

I'm Having this log file "internet.log" which is about 10GB. When I parse it in python I get an exception "MemoryError". The log file looks something like this...

Jun 15 16:26:21 dnsmasq[1979]: reply fd-geoycpi-uno.gycpi.b.yahoodns.net is 74.6.160.107
Jun 15 16:26:21 dnsmasq[1979]: reply fd-geoycpi uno.gycpi.b.yahoodns.net is 216.115.100.123
Jun 15 16:26:21 dnsmasq[1979]: reply fd-geoycpi-uno.gycpi.b.yahoodns.net is 216.115.100.124
Jun 15 16:26:21 dnsmasq[1979]: reply fd-geoycpi-uno.gycpi.b.yahoodns.net is 74.6.160.106
Jun 15 16:26:21 dnsmasq[1979]: query[A] fd-geoycpi-uno.gycpi.b.yahoodns.net from 192.168.1.33
Jun 15 16:26:21 dnsmasq[1979]: cached fd-geoycpi-uno.gycpi.b.yahoodns.net is 74.6.160.106
Jun 15 16:26:21 dnsmasq[1979]: cached fd-geoycpi-uno.gycpi.b.yahoodns.net is 216.115.100.124
Jun 15 16:26:21 dnsmasq[1979]: cached fd-geoycpi-uno.gycpi.b.yahoodns.net is 216.115.100.123
Jun 15 16:26:21 dnsmasq[1979]: cached fd-geoycpi-uno.gycpi.b.yahoodns.net is 74.6.160.107
Jun 15 16:26:23 dnsmasq[1979]: query[A] armdl.adobe.com from 192.168.1.24

I'm currently using this method to parse the log file:

def parse():
Date = []
IPAddress = []
DomainsVisited = []
with open("internet.log", "r") as file:
    content = file.readlines()
    for items in content:
        if 'query[A]' in items:
            getDate(Date, items)
            getIPAddress(IPAddress, items)
            getDomainsVisited(DomainsVisited, items)
finalResult = [[i, j, k] for i, j, k in zip(Date, IPAddress, DomainsVisited)]
return display(finalResult)

If I parse a log file of say some 10MB the output is being displayed but when I go to parse the 10GB log file I get the error. How can I Fix this? Thank you.

Well, you're reading the whole file into memory with `file.readlines()`. Saying `for items in file:` will read it a line at a time. — Peter Wood, Jun 22 '18 at 08:41
The rest of your code doesn't look right though. E.g. for each `item` you are clobbering `Date` rather than appending to the list. — Peter Wood, Jun 22 '18 at 08:42
@PeterWood for items in file also didn't work. I got this message in python console "Process finished with exit code 247" — Manoj Jahgirdar, Jun 22 '18 at 08:55

score 0 · Answer 1 · answered Jun 22 '18 at 08:40

0

You should not use file.readlines(). Doing so immediately reads the whole file into memory, which is likely to fill it up immediately. Instead, iterate over the file:

with open("internet.log", "r") as file:
    for items in file:

(Of course, depending on what you're doing with the data this could still break as you go through the file.)

answered Jun 22 '18 at 08:40

Daniel Roseman

588,541
66
880
895

Didn't work :( I got this message in python console "Process finished with exit code 247" – Manoj Jahgirdar Jun 22 '18 at 09:02

score 0 · Answer 2 · answered Jun 22 '18 at 09:06

You're reading the whole file into memory with readlines.

You can read a line at a time by saying for items in file.

Cleaning up your code a little, using better variable names, and a list comprehension to build the result:

with open("internet.log") as log:
    finalResults = [[getDate(line), getIPAddress(line), getDomainsVisited(line)]
                    for line in log
                    if 'query[A]' in line]

I would extract the result to a function:

def parse_log_line(line):
    return [getDate(line),
            getIPAddress(line),
            getDomainsVisited(line)]

Then your code would be:

with open("internet.log") as log:
    finalResults = [parse_log_line(line)
                    for line in log
                    if 'query[A]' in line]

How do I parse extremely large file in python?

2 Answers2