How to use collections.defaultdict, to ensure a straightforward comparison of values in Python

Question

Basically this is a code modification of a previous question i saw on stack-overflow & I found it interesting to pursue which deals with port scanning . In this code I am trying to compare 2 pickle files which hold the scan result of 2 scans performed one after another. I am interested in finding

3 operations on the set of ports

& (intersection): to see which ports have remained constant across scans (same ports)

old - new: to see which ports were in the old scan but no longer in the new (deleted ports)

new - old: to see which ports are in the new scan but were not in the old (added ports)

def comp_ports(self,filename):
  try:
        f = open(filename)
        self.prev_report = pickle.load(f) # NmapReport

        for s in self.prev_report.hosts:
            self.old_port_dict[s.address] = collections.defaultdict(set())
            for x in s.get_open_ports():
                self.old_port_dict[s.address].add(x)

        for s in self.report.hosts:
            self.new_port_dict[s.address] = collections.defaultdict(set())
            for x in s.get_open_ports():
               self.new_port_dict[s.address].add(x)

        hosts = sorted(set(self.prev_report.hosts.keys() + self.report.hosts.keys()))

        for host in hosts:
                scan_same[host] = self.prev_report.hosts[host] & self.report.hosts[host]
                scan_new[host]  = self.report.hosts[host] - self.prev_report.hosts[host]
                scan_del[host]  = self.prev_report.hosts[host] - self.report.hosts[host]

        print()
        print('-' * 10, 'Same')
        for host, ports in scan_same.items():
            print(host, ':')
            for port in ports:
                 print(':::', port[0], '/', port[1])

        print()
        print('*' * 10, 'Added')
        for host, ports in scan_new.items():
            print(host, ':')
            for port in ports:
                  print(':::', port[0], '/', port[1])

        print()
        print('=' * 10, 'Deleted')
        for host, ports in scan_del.items():
            print(host, ':')
            for port in ports:
                   print(':::', port[0], '/', port[1])

  except Exception as l:
         print l

But the code throws :

first argument must be callable

Help me to use collection efficiently.

P.S : Trying to improve this way How to compare dictionaries and see what changed?

falsetru · Accepted Answer · 2016-01-19T05:55:14.367

0

You need to pass a callable (function / class / method) which will return a default value, not a default value itself.

Follow expression:

collections.defaultdict(set())

should be replaced with:

collections.defaultdict(set)

UPDATE

You need to create defaultdicts outside of the for loops:

self.old_port_dict = collections.defaultdict(set)
for s in self.prev_report.hosts:
    for x in s.get_open_ports():
        self.old_port_dict[s.address].add(x)

self.new_port_dict = collections.defaultdict(set)
for s in self.report.hosts:
    for x in s.get_open_ports():
        self.new_port_dict[s.address].add(x)

edited Jan 19 '16 at 05:55

answered Jan 19 '16 at 05:29

falsetru

357,413
63
732
636

okay got it.. but then it says : 'collections.defaultdict' object has no attribute 'add' , and basically it means i am trying to add() a list of ports inside it.. basically add() is for a set /dict operation – Jan 19 '16 at 05:33
@PythonFreak, Make a `defaultdict` outside of `for` loop: `self.old_port_dict = collections.defaultdict(set)`. updated the answer accordingly. – falsetru Jan 19 '16 at 05:54
It says : 'list' object has no attribute 'keys' , i believe due to `hosts = sorted(set(self.prev_report.hosts.keys() + self.report.hosts.keys()))` – Jan 19 '16 at 06:09
@PythonFreak, Try `hosts = sorted(set(self.old_port_dict) | set(self.new_port_dict))` – falsetru Jan 19 '16 at 06:22
list indices must be integers, not str – Jan 19 '16 at 06:33
list indices must be integers, not str Traceback (most recent call last): File "portwatch.py", line 308, in report.comp_ports(config.get('system','scan_directory') + '/nmap-report-old.pkl') File "portwatch.py", line 145, in comp_ports scan_same[host] = self.prev_report.hosts[host] & self.report.hosts[host] TypeError: list indices must be integers, not str – Jan 19 '16 at 07:14

score 0 · Answer 2 · answered Jan 19 '16 at 05:30

0

To answer the question in your title, you need to do

collections.defaultdict(set)

You need to pass in something that can be instantiated. Passing in collections.defaultdict(set()) is passing in something that already is instantiated.

answered Jan 19 '16 at 05:30

Martin Konecny

57,827
19
139
159

It throws : after 'collections.defaultdict' object has no attribute 'add' .. how do i add the ports then ? – Jan 19 '16 at 05:32
You need to treat it like a dictionary - you would do something like `x = collections.defaultdict(set)` and then `x['key'] = value` – Martin Konecny Jan 19 '16 at 05:32
i am not able to put it in code .. can u edit the loop for s in self.prev_report.hosts: self.old_port_dict[s.address] = collections.defaultdict(set) for x in s.get_open_ports(): self.old_port_dict[s.address].add(x) – Jan 19 '16 at 05:37
I think you want `old_port_dict` to be a defaultdict? In that case find where you do `self.old_port_dict = {}` and make it `self.old_port_dict = collections.defaultdict(set)` – Martin Konecny Jan 19 '16 at 05:39
Please look into http://stackoverflow.com/questions/34851595/how-to-compare-dictionaries-and-see-what-changed/34852463?noredirect=1#comment57446991_34852463 , if that explains better ..i am unable to do so – Jan 19 '16 at 05:48
If you want `self.old_port_dict[s.address].add(x)` to behave you're adding `x` to a set in a `defaultdict`, then simply take my suggestion from above. All you are doing is changing a `dict` to a `defaultdict` with that suggestion. – Martin Konecny Jan 19 '16 at 05:53
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/101041/discussion-between-pythonfreak-and-martin-konecny). – Jan 19 '16 at 06:57

How to use collections.defaultdict, to ensure a straightforward comparison of values in Python

2 Answers2