-1

I am only beginning to code in Python and I still try to figure out how everything works, so if the logic behind it is not right, please correct me.

I am writing a script that queries one device and compares the returned data to another device. The goal is to keep both devices configuration in sync.

Here's the script:

import requests
import json
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

net_objects = ['aaa','availability_group','dns_group','dns_host','group','host','multicast','network','range']

for obj in net_objects:
    src_URL = 'https://172.16.2.100:4444/api/objects/network/' + obj + '/'
    src_headers = {
        'Accept': 'application/json',
        'Authorization': 'Basic dG9rZW46U2tCaENIZUlRZnlVeEpBU3dqYUh2c0VHRmZjdWtDTFg='
        }
    src_net_obj = requests.get(src_URL, headers=src_headers, timeout=15, verify=False)

    dst_URL = 'https://172.16.2.101:4444/api/objects/network/' + obj + '/'
    dst_headers = {
        'Accept': 'application/json',
        'X-Restd-Err-Ack': 'all',
        'Authorization': 'Basic dG9rZW46dlZ3WnVZZGxpd01IRkxNVXpKVXZtZXhiZGZHSExobnI='
        }
    dst_net_obj = requests.get(dst_URL, headers=dst_headers, timeout=15, verify=False)

    for src_dict_item in src_net_obj.json():
        if src_dict_item["name"] in dst_net_obj.json():
            update =  requests.patch(url, headers=dst_headers, timeout=15, verify=False)
        else:
            update =  requests.put(url, headers=dst_headers, timeout=15, verify=False)

I am not sure about the synthax. Here's what I expect it to do:

  1. Query the Source device and return JSON formated data
  2. Query the Destination device and return JSON formated data
  3. Compare both queries
  4. If an object in the Source device also exists in the Destination device, update it using a PATCH request
  5. If an object in the Source device doesn't exist in the Destination device, create it using a PUT request

I am filtering on the name value returned by the query. If the name exists in both the Source and Destination device, I want it to update the data in the Destination object using the Source object. I would eventually like to compare the actual data in the ocjects in just skip the current object if the data is the same in both Source and Destination, but the will come later.

The returned data from a GET request on the Source device looks like this:

[{'_locked': '', '_ref': 'REF_DefaultL2TPPool', '_type': 'network/network', 'address': '10.242.3.0', 'address6': 'fd32:5a88:8e98:3::', 'comment': 'Default L2TP VPN Remote Access IP Pool', 'interface': '', 'name': 'VPN Pool (L2TP)', 'netmask': 24, 'netmask6': 64, 'resolved': True, 'resolved6': True}, {'_locked': '', '_ref': 'REF_DefaultPPTPPool', '_type': 'network/network', 'address': '10.242.1.0', 'address6': 'fd32:5a88:8e98:1::', 'comment': 'Default PPTP Remote Access IP Pool', 'interface': '', 'name': 'VPN Pool (PPTP)', 'netmask': 24, 'netmask6': 64, 'resolved': True, 'resolved6': True}, {'_locked': '', '_ref': 'REF_NetNetNetVlan66Publi', '_type': 'network/network', 'address': '192.168.66.0', 'address6': '', 'comment': '', 'interface': '', 'name': 'VLAN 66 Public', 'netmask': 24, 'netmask6': 128, 'resolved': True, 'resolved6': False}, {'_locked': '', '_ref': 'REF_DefaultRWPool', '_type': 'network/network', 'address': '10.242.4.0', 'address6': 'fd32:5a88:8e98:4::', 'comment': 'Default IPsec VPN Remote Access IP Pool', 'interface': '', 'name': 'VPN Pool (IPsec)', 'netmask': 24, 'netmask6': 64, 'resolved': True, 'resolved6': True}, {'_locked': 'user', '_ref': 'REF_NetworkInternet', '_type': 'network/network', 'address': '0.0.0.0', 'address6': '', 'comment': '"Any" network, bound to interfaces with default IPv4 gateway', 'interface': 'REF_IntEthExternaWan', 'name': 'Internet IPv4', 'netmask': 0, 'netmask6': 0, 'resolved': True, 'resolved6': False}, {'_locked': '', '_ref': 'REF_DefaultSSLPool', '_type': 'network/network', 'address': '10.242.2.0', 'address6': 'fd32:5a88:8e98:2::', 'comment': 'Default SSL VPN IP Pool', 'interface': '', 'name': 'VPN Pool (SSL)', 'netmask': 24, 'netmask6': 64, 'resolved': True, 'resolved6': True}, {'_locked': 'user', '_ref': 'REF_NetworkInternet6', '_type': 'network/network', 'address': '0.0.0.0', 'address6': '::', 'comment': '"Any" network, bound to interfaces with default IPv6 gateway', 'interface': '', 'name': 'Internet IPv6', 'netmask': 0, 'netmask6': 0, 'resolved': False, 'resolved6': False}, {'_locked': '', '_ref': 'REF_DefaultCiscoRWPool', '_type': 'network/network', 'address': '10.242.5.0', 'address6': 'fd32:5a88:8e98:5::', 'comment': 'Default IPsec VPN Remote Access IP Pool for Cisco clients', 'interface': '', 'name': 'VPN Pool (Cisco)', 'netmask': 24, 'netmask6': 64, 'resolved': True, 'resolved6': True}, {'_locked': '', '_ref': 'REF_NetNetNetVlan1Site', '_type': 'network/network', 'address': '192.168.0.0', 'address6': '', 'comment': '', 'interface': '', 'name': 'NET - VLAN 1 Site 1', 'netmask': 24, 'netmask6': 128, 'resolved': True, 'resolved6': False}]

Here's the data returned from the GET request on the Destination device:

[{'_locked': '', '_ref': 'REF_DefaultL2TPPool', '_type': 'network/network', 'address': '10.242.3.0', 'address6': 'fd32:5a88:8e98:3::', 'comment': 'Default L2TP VPN Remote Access IP Pool', 'interface': '', 'name': 'VPN Pool (L2TP)', 'netmask': 24, 'netmask6': 64, 'resolved': True, 'resolved6': True}, {'_locked': '', '_ref': 'REF_DefaultPPTPPool', '_type': 'network/network', 'address': '10.242.1.0', 'address6': 'fd32:5a88:8e98:1::', 'comment': 'Default PPTP Remote Access IP Pool', 'interface': '', 'name': 'VPN Pool (PPTP)', 'netmask': 24, 'netmask6': 64, 'resolved': True, 'resolved6': True}, {'_locked': '', '_ref': 'REF_DefaultRWPool', '_type': 'network/network', 'address': '10.242.4.0', 'address6': 'fd32:5a88:8e98:4::', 'comment': 'Default IPsec VPN Remote Access IP Pool', 'interface': '', 'name': 'VPN Pool (IPsec)', 'netmask': 24, 'netmask6': 64, 'resolved': True, 'resolved6': True}, {'_locked': 'user', '_ref': 'REF_NetworkInternet', '_type': 'network/network', 'address': '0.0.0.0', 'address6': '', 'comment': '"Any" network, bound to interfaces with default IPv4 gateway', 'interface': 'REF_IntEthExternaWan', 'name': 'Internet IPv4', 'netmask': 0, 'netmask6': 0, 'resolved': True, 'resolved6': False}, {'_locked': '', '_ref': 'REF_DefaultSSLPool', '_type': 'network/network', 'address': '10.242.2.0', 'address6': 'fd32:5a88:8e98:2::', 'comment': 'Default SSL VPN IP Pool', 'interface': '', 'name': 'VPN Pool (SSL)', 'netmask': 24, 'netmask6': 64, 'resolved': True, 'resolved6': True}, {'_locked': 'user', '_ref': 'REF_NetworkInternet6', '_type': 'network/network', 'address': '0.0.0.0', 'address6': '::', 'comment': '"Any" network, bound to interfaces with default IPv6 gateway', 'interface': '', 'name': 'Internet IPv6', 'netmask': 0, 'netmask6': 0, 'resolved': False, 'resolved6': False}, {'_locked': '', '_ref': 'REF_DefaultCiscoRWPool', '_type': 'network/network', 'address': '10.242.5.0', 'address6': 'fd32:5a88:8e98:5::', 'comment': 'Default IPsec VPN Remote Access IP Pool for Cisco clients', 'interface': '', 'name': 'VPN Pool (Cisco)', 'netmask': 24, 'netmask6': 64, 'resolved': True, 'resolved6': True}]

I am trying to compare that returned data from the Source to the data returned from the Destination.

This obviously doesn't work as I expect it to work. I have tried multiple combinations but I can't seem to find the right way to do it.

Any help would be appreciated.

Noct03
  • 37
  • 1
  • 7
  • Which results were you expecting? – jeph Oct 27 '19 at 20:49
  • Hey Jeph, sorry I may not have expressed myself correctly. The returned data is from the Source device. What I am trying to do is compare it to the returned data on the Destination device. I will correct it in the original post. Thanks – Noct03 Oct 27 '19 at 21:05
  • What's the return data from the destination device? – Jack Fleeting Oct 27 '19 at 22:02
  • It's also a JSON formated output, only with less information since the device is new. Some "objects" are the same (eg. VPN Pools). I want to compare both a them and update the repeating entries and create the missing ones. Thanks – Noct03 Oct 27 '19 at 23:37
  • Can you post a couple of examples of return data from the destination device? – Jack Fleeting Oct 28 '19 at 10:54
  • Sure, I updated the initial question with the returned data from the destination device. It's basically the same thing but with less data. Thanks! – Noct03 Oct 28 '19 at 11:39
  • do you get any error when you run the code you've shared above? like KeyError? – jeph Oct 28 '19 at 11:49
  • 1. Even if the We're not yet clear what you want to achieve(the patch and put requests are not updating anything since there is no data associated with the requests, it's only header information) 2. the if condition in the for-loop is looking for a key but it was supposed to compare data, not keys. fd – jeph Oct 28 '19 at 12:08
  • Ok, now I understand the issue, but not the desired result: you say in step 4, "update it using a `PATCH` request"; what is the "it" there - source or destination? Are you basically trying to make destination exactly equal to source? – Jack Fleeting Oct 28 '19 at 13:23
  • Exactly yes. Basically, these are Sophos firewalls and they are using the Swagger API (if that matters). I want to be able to replicate changes done on the Source device to the Destination device using a python script and the REST API. – Noct03 Oct 28 '19 at 18:51
  • @jeph No I don't get an error. As for you second comment, what should I do to compare the actual data instead of the keys? Thanks! – Noct03 Oct 28 '19 at 23:12

1 Answers1

0

Since I'm not really sure what you want to do, here's something that may at least part of the way there: it compares the Source with the Destination and gives you a report on the differences. And you can take it from there. The included function is lifted from here.

src = [the Source data you posted above]
dest = [the Destination data you posted above]

Import itertools

def compare_dict(dict1, dict2):
    for x1 in set(dict1.keys()).union(dict2.keys()):
        z = dict1.get(x1) == dict2.get(x1)
        if not z:
            print('key', x1)
            print('value A', dict1.get(x1), '\nvalue B', dict2.get(x1))
            print('-----\n')

for s,d in itertools.zip_longest(src,dest):
    if d is not None and s is not None:
        compare_dict(s,d)
    else:
        print('Not in Destination:',list(s.items()))

Output:

key netmask6
value A 128 
value B 64

.....
Not in Destination: [('_locked', ''), ('_ref', 'REF_DefaultCiscoRWPool')...]

etc. Hope it helps.

Jack Fleeting
  • 24,385
  • 6
  • 23
  • 45
  • Hi Jack, really appreciate the help. I will try to better explain what I want to achieve. We have 2 Sophos firewalls, one being in production and the other one is a "spare" if the main one fails. I want to query the main device and return all of the objects, whch are represented as a dictionnary in the JSON output (which kind of looks like a nested dictionnary) and update (`PATCH`) or create (`PUT`) each objects on the spare device. I need a way to compare each objects (dict) from the main device and see if it exists on the spare device. Let me know if that makes sense. Thanks! – Noct03 Oct 29 '19 at 00:41
  • By the way I looked at your suggestion but it doesn't compare objects (dict) based on their name so they will always differ. Thanks again! – Noct03 Oct 29 '19 at 00:45