Reset iterator error

Question

I have been trying to create a csv file that will change all the subnets and IP address ranges so none overlap. The original question is here Editing csv file to edit subnets so there is no overlap in IP ranges (ipaddress module in Python 3.3) but I have fixed it so the if statements stop the ValueErrors and the second issue actually is not a problem. I believe I can make this work if I can reset the iterator. It needs to compare every line in file1 with every line in file2.

When I try this :

import ipaddress
import csv
from csv import DictReader, DictWriter

with open(r'file1.csv', newline='') as fin3,\
     open(r'file2.csv', newline='') as fin4,\
     open(r'file3.csv', 'w', newline='') as fout3:


    read3 = DictReader(fin3) # fin 3 and 4 are copies of the same file
    read4 = DictReader(fin4)

    writenum3 = DictWriter(fout3, fieldnames=read3.fieldnames) 
    writenum3.writeheader()

    for line3 in read3:
        line3['Range']=ipaddress.ip_network(line3['Range']) # Convert IP ranges to ip network object
        for line4 in read4:
            line4['Range']=ipaddress.ip_network(line4['Range']) # Convert IP ranges to ip network object
            if line3['Range'].netmask < line4['Range'].netmask: # To avoid "Not contained in" errors
                if line3['Range'].overlaps(line4['Range']): # Tests for IP overlap
                    lst=list(line3['Range'].address_exclude(line4['Range'])) # List of subnets excluding line4 subnet
                    print (lst) # Temporary to view contents of lst
                    for val in lst:
                        line3['Range']=val # New range
                        line3[' IPStart']=val.network_address #New net address
                        line3[' IPStop']=val.broadcast_address #New broadcast address
                        writenum3.writerow(line3) # Write lines

It compares only the first line (not header) of file1 with the second line (not header) of file2.

When I have this (using file.seek(0)):

import ipaddress
import csv
from csv import DictReader, DictWriter

with open(r'file1.csv', newline='') as fin3,\
     open(r'file2.csv', newline='') as fin4,\
     open(r'file3.csv', 'w', newline='') as fout3:


    read3 = DictReader(fin3) # fin 3 and 4 are copies of the same file
    read4 = DictReader(fin4)

    writenum3 = DictWriter(fout3, fieldnames=read3.fieldnames) 
    writenum3.writeheader()

    for line3 in read3:
        line3['Range']=ipaddress.ip_network(line3['Range']) # Convert IP ranges to ip network object
        fin4.seek(0)
        for line4 in read4:
            line4['Range']=ipaddress.ip_network(line4['Range']) # Convert IP ranges to ip network object
            if line3['Range'].netmask < line4['Range'].netmask: # To avoid "Not contained in" errors
                if line3['Range'].overlaps(line4['Range']): # Tests for IP overlap
                    lst=list(line3['Range'].address_exclude(line4['Range'])) # List of subnets excluding line4 subnet
                    print (lst) # Temporary to view contents of lst
                    for val in lst:
                        line3['Range']=val # New range
                        line3[' IPStart']=val.network_address #New net address
                        line3[' IPStop']=val.broadcast_address #New broadcast address
                        writenum3.writerow(line3) # Write lines

It gives an error ValueError: 'Range' does not appear to be an IPv4 or IPv6 network.

If I try the list() method (explained here: Can iterators be reset in Python?):

import ipaddress
import csv
from csv import DictReader, DictWriter

with open(r'file1.csv', newline='') as fin3,\
     open(r'file2.csv', newline='') as fin4,\
     open(r'file3.csv', 'w', newline='') as fout3:


    read3 = list(DictReader(fin3)) # fin 3 and 4 are copies of the same file
    read4 = list(DictReader(fin4))

    writenum3 = DictWriter(fout3, fieldnames=read3.fieldnames) 
    writenum3.writeheader()

    for line3 in read3:
        line3['Range']=ipaddress.ip_network(line3['Range']) # Convert IP ranges to ip network object
        fin4.seek(0) # Reset iterator
        for line4 in read4:
            line4['Range']=ipaddress.ip_network(line4['Range']) # Convert IP ranges to ip network object
            if line3['Range'].netmask < line4['Range'].netmask: # To avoid "Not contained in" errors
                if line3['Range'].overlaps(line4['Range']): # Tests for IP overlap
                    lst=list(line3['Range'].address_exclude(line4['Range'])) # List of subnets excluding line4 subnet
                    print (lst) # Temporary to view contents of lst
                    for val in lst:
                        line3['Range']=val # New range
                        line3[' IPStart']=val.network_address #New net address
                        line3[' IPStop']=val.broadcast_address #New broadcast address
                        writenum3.writerow(line3) # Write lines

It gives the AttributeError: 'list' object has no attribute 'fieldnames' error.

I'm wondering if I should use the http://docs.python.org/3.4/library/itertools.html tee() for this but I'm not sure how to use it here or if it would even work.

This is my file1 and file2:

Zone Name, IPStart, IPStop,Range,Source
Group A,10.0.0.0,10.127.255.255,10.0.0.0/9,New List
Group A Sales,10.16.0.0,10.31.255.255,10.16.0.0/12,New List
Group A Marketing,10.62.0.0,10.62.255.255,10.62.0.0/16,New List
Group A Research,10.62.0.0,10.63.255.255,10.62.0.0/15,Old List
Group A Sales Primary routers,10.23.1.0,10.23.1.15,10.24.1.0/28,New List
Group A Sales Web Servers - Primary,10.18.0.0,10.18.0.255,10.18.0.0/24,New List
Group A Sales Web Servers,10.16.0.0,10.19.255.255,10.16.0.0/14,New List
Group B,10.128.0.0,10.255.255.255,10.128.0.0/9,Old List

And this is what I ULTIMATELY want (although all of the subnets for each compare is okay for now):

Zone Name, IPStart, IPStop,Range,Source
Group A,10.0.0.0,10.15.255.255,10.0.0.0/12,New List
Group A,10.32.0.0,10.47.255.255,10.32.0.0/12,New List
Group A,10.48.0.0,10.55.255.255,10.48.0.0/13,New List
Group A,10.56.0.0,10.59.255.255,10.56.0.0/14,New List
Group A,10.60.0.0,10.61.255.255,10.60.0.0/15,New List
Group A,10.64.0.0,10.127.255.255,10.64.0.0/10,New List
Group A Marketing,10.62.0.0,10.62.255.255,10.62.0.0/16,New List
Group A Research,10.63.0.0,10.63.255.255,10.63.0.0/16,Old List
Group A Sales,10.20.0.0,10.21.255.255,10.20.0.0/15,New List
Group A Sales,10.22.0.0,10.22.255.255,10.22.0.0/16,New List
Group A Sales,10.23.0.0,10.23.0.255,10.23.0.0/24,New List
Group A Sales,10.23.1.128,10.23.1.255,10.23.1.128/25,New List
Group A Sales,10.23.1.16,10.23.1.31,10.23.1.16/28,New List
Group A Sales,10.23.1.32,10.23.1.63,10.23.1.32/27,New List
Group A Sales,10.23.1.64,10.23.1.127,10.23.1.64/26,New List
Group A Sales,10.23.128.0,10.23.255.255,10.23.128.0/17,New List
Group A Sales,10.23.16.0,10.23.31.255,10.23.16.0/20,New List
Group A Sales,10.23.2.0,10.23.3.255,10.23.2.0/23,New List
Group A Sales,10.23.32.0,10.23.63.255,10.23.32.0/19,New List
Group A Sales,10.23.4.0,10.23.7.255,10.23.4.0/22,New List
Group A Sales,10.23.64.0,10.23.127.255,10.23.64.0/18,New List
Group A Sales,10.23.8.0,10.23.15.255,10.23.8.0/21,New List
Group A Sales,10.24.0.0,10.31.255.255,10.24.0.0/13,New List
Group A Sales,10.24.32.0,10.24.63.255,10.24.32.0/19,New List
Group A Sales Primary routers,10.23.1.0,10.23.1.15,10.24.1.0/28,New List
Group A Sales Web Servers,10.16.0.0,10.17.255.255,10.16.0.0/15,New List
Group A Sales Web Servers,10.19.0.0,10.19.255.255,10.19.0.0/16,New List
Group A Sales Web Servers,10.18.128.0,10.18.255.255,10.18.128.0/17,New List
Group A Sales Web Servers,10.18.64.0,10.18.127.255,10.18.64.0/18,New List
Group A Sales Web Servers,10.18.32.0,10.18.63.255,10.18.32.0/19,New List
Group A Sales Web Servers,10.18.16.0,10.18.31.255,10.18.16.0/20,New List
Group A Sales Web Servers,10.18.8.0,10.18.15.255,10.18.8.0/21,New List
Group A Sales Web Servers,10.18.4.0,10.18.7.255,10.18.4.0/22,New List
Group A Sales Web Servers,10.18.2.0,10.18.3.255,10.18.2.0/23,New List
Group A Sales Web Servers,10.18.1.0,10.18.1.255,10.18.1.0/24,New List
Group A Sales Web Servers - Primary,10.18.0.0,10.18.0.255,10.18.0.0/24,New List
Group B,10.128.0.0,10.255.255.255,10.128.0.0/9,Old List

Any help is appreciated getting this to work how it is supposed to.

Thanks, B0T

A comment in your code says `# fin 3 and 4 are copies of the same file`, yet you supply different contenst for `file1.csv` and `file1.csv`. What is the meaning of this? — martineau, Jul 23 '13 at 01:10
@martineau I used copies of the same file because I couldn't wrap my mind around how to compare the same file to itself and couldn't find any good examples of it for csv dictreader/writer. — CircuitB0T, Jul 29 '13 at 17:20

score 1 · Accepted Answer · answered Jul 23 '13 at 01:16

1

I think you just need to insert a read4 = DictReader(fin4) after the fin4.seek(0) # Reset iterator in your second attempt so the DictReader instance starts at the beginning of the file again, too.

answered Jul 23 '13 at 01:16

martineau

119,623
25
170
301

Wow, this seems to have fixed my problem. I just need to check my expected output against this to make sure. Thanks! – CircuitB0T Jul 29 '13 at 17:23
Go to hear. It's important to realize that after you hand a file over to `DictReader()` you have to be fairly careful about what you do to it directly since the latter is not also accessing it. In general it would be hard to keep them in-sync, hence the need for a hard reset like this. – martineau Jul 29 '13 at 19:16

score 0 · Answer 2 · answered Sep 19 '13 at 14:24

Just in case you want to subclass DictReader / Writer then you will need another method to reset the class.

For DictReader:

f = open(filename, "rb")
d = csv.DictReader(f, delimiter=",")

f.seek(0)
d.__init__(f, delimiter=",")

For DictWriter:

f = open(filename, "rb+")
d = csv.DictWriter(f, fieldnames=fields, delimiter=",")

f.seek(0)
f.truncate(0)
d.__init__(f, fieldnames=fields, delimiter=",")
d.writeheader()
f.flush()

Reset iterator error

2 Answers2