Script to scrape web gui for string, errors on Python 2.7 runs on Python 3

Question

As part of my job I may need to access the GUI of a Cisco phone (Cisco 8851, 7960, etc.) to determine what MAC address is assigned to the device. The MAC address is prefixed with SEP (e.g. SEPAABBCCDDEEFF) as the syslogs provided by the call processing server, on specific events, only includes the IP address, not the MAC address. MAC address is needed to confirm if a given set of configurations exist for that device on the server. Manually pulling up a chunk of 50-100 phones to check MAC addresses via http is awful. I tried to automate this, and sort of did, but I missed the mark in Python3 and crapped out entirely on Python2, won't run past user input collection.

My questions:

1) General question regarding awk and/or similar tools -- to check the data returned from the website for the necessary SEP* string I use awk to print out multiple instances of SEP* that exist on the same line, but it provides two outputs rather than just the first. I've tried using "grep -o "SEP*" but this provides only SEP as the response. Thoughts on how I can have this return the first instance of SEP* (e.g. SEPAABBCCDDEEFF) only, instead of an entire line of html code?

Issue #1 - As you can see the awk attempt does provide the first instance cleanly but the second instance it provides a lengthy bit of garbage on both ends. My intention was to only provide a single SEP* value per web link it parses.

@ubuntu:~/Scripts/CiscoScripts$ python transientPhones.py 
How many phones?: 1
What is the phone IP address?: <ip-addr>
SEPAABBCCDDEEFF
width=20></TD><TD><B>SEPAABBCCDDEEFF</B></TD></TR><TR><TD><B>
kenneth@ubuntu:~/Scripts/CiscoScripts$

2) Running the script in a Python 2.7 environment, the script fails on SyntaxError: invalid syntax after collection of the user input. I am failing to understand why (beyond I'm doing it wrong or in an incompatible way). My home environment is python 3.x (latest) and I did not take that into consideration when working up scripts to use in a python 2.7 environment, and as I am new to python and coding I have really only been learning python3 syntax and conventions. Any thoughts here?

Issue #2 -- This one has me confused. I'm sure there's a simple answer/solution here... I'm not experienced enough to see it.

$:python transientPhones.py
How many phones?: 1
What is the phone IP address?: 192.168.1.1
Traceback (most recent call last):
  File "transientPhones.py", line 13, in <module>
    ipAddress.append(input('What is the phone IP address?: '))
  File "<string>", line 1
    192.168.1.1
            ^
SyntaxError: invalid syntax

Code:

#!/usr/bin/python
#import required modules
import subprocess

#Define Variables
x = input('How many phones?: ')
x = int(x)
ipAddress = []

#Loop to grab IP addresses
for i in range(x) : #Loop X amount of times based on input from user
        ipAddress.append(input('What is the phone IP address?: '))

#Grab XML Data and awk it for SEP*.
for n in ipAddress :
        subprocess.call ("curl --max-time 5 -s http://" + n + "/CGI/Java/Serviceability?adapter=device.statistics.device | awk '/SEP*/{for(i=1;i<=NF;++i)if($i~/SEP*/)print $i}'", shell=True)

Your problem is that in Python 2, input = eval in Python 3, and raw_input in Python 2 = input in Python 3. So you need to use raw_input instead of input if you want the app to run on Python 2. I'll add a more detailed explanation in an answer now. — Demian Wolf, Apr 24 '20 at 12:51

score 0 · Accepted Answer · answered Apr 24 '20 at 13:07

You get this error because input in Python 2 and input in Python 3 are absolutely different functions.

In Python 2, input gets user input and executes it, while in Python 3 you can achieve the same thing using eval(input(...)). So, you call 192.168.1.1 as Python code. Of course, this is not a valid Python code, hence you get the SyntaxError.

In Python 3, input just gets user input, while in Python 2 raw_input does the same thing.

This means that in your case you need to use raw_input for Python 2 and input for Python 3. Of course, you may just replace input with raw_input, but if you try to run the code on Python 3 it wouldn't work.

There are some good solutions to this problem. You may redefine the input function with raw_input on Python 2 and leave everything as it is on Python 3.

#!/usr/bin/python
#import required modules
import subprocess

try:
    input = raw_input
except NameError:
    pass

#Define Variables
x = input('How many phones?: ')
x = int(x)
ipAddress = []

#Loop to grab IP addresses
for i in range(x) : #Loop X amount of times based on input from user
        ipAddress.append(input('What is the phone IP address?: '))

#Grab XML Data and awk it for SEP*.
for n in ipAddress:
        subprocess.call("curl --max-time 5 -s http://" + n + "/CGI/Java/Serviceability?adapter=device.statistics.device | awk '/SEP*/{for(i=1;i<=NF;++i)if($i~/SEP*/)print $i}'", shell=True)

Look at this answer for more information: Use of input/raw_input in python 2 and 3

Thank you for the response. I was looking more at the code this morning and the input function (where it seems to break) and you're entirely right there. Not only, when learning, did I not restrict myself to the same Python version the environment I would run the code in is using, but I also didn't learn syntax from python2 as a result. I'll make the suggested edit after reviewing the provided documentation! — Kenneth Perry, Apr 24 '20 at 15:54

score 0 · Answer 2 · answered Apr 30 '20 at 21:14

I want to post this as an answer because it accomplishes what I was trying to do much more eloquently, in terms of actually using python and modules as opposed to just spawning a subprocess to run regular commands. I'll also link to where I store my script.

https://github.com/Unhall0w3d/mind-enigma/blob/master/transientPhones_v2.py

Caveats: No error handling. If the script times out attempting to hit the http page, huge traceback. http needs to be accessible. Target URL structure has to be the same (and is in Cisco IP Communicator software, as well as most of their 7XXX, 8XXX and 9XXX series phones).

Script:

#!/usr/var/python

import re
import requests
from bs4 import BeautifulSoup

#Define how many phones we need to hit
x = input('How many phones?: ')
x = int(x)

#Collect IP addresses
ipAddress = []

#Here we loop to grab the list of IP Addresses to access.
for i in range(x):
        ipAddress.append(input('What is the phone IP address?: '))

#Here we loop to access each IP address provided (equivalent of Network Configuration page) to collect Device Type + MAC + Registered state.
for n in ipAddress:
        URL = 'http://' + n + '/CGI/Java/Serviceability?adapter=device.statistics.configuration' #URL is dynamically created based on IPs collected
        page = requests.get(URL, timeout=6)
        soup = BeautifulSoup(page.content, 'html.parser')
#looking for instance of SEP* or CIPC*, such as CIPCKPERRY or SEPAABBCCDDEEFF. Returned as variable 'results'
        results = soup.find(text=re.compile('SEP*|CIPC*'))
#looking for instance of "Active" on the webpage indicating device is registered to a given CCM. Returned as variable 'results2'
        results2 = soup.find_all(text=re.compile('Active'))
#conditional statement that dictates if "Active" is not found, report only the device model and name. Otherwise report the device it is registered to. (e.g. cucmpub.ipt.local Active)
        if results2 is None:
                print(results)
        else:
                print(results, results2)

Expected Return:

kenneth@ubuntu:~/Scripts/CiscoScripts$ python transientPhones_v2.py
How many phones?: 2
What is the phone IP address?: 1.1.1.1
What is the phone IP address?: 2.2.2.2
Cisco Unified IP Phone Cisco Communicator ( CIPCKPERRY ) ['SERVER-FQDN   Active']
Cisco IP Phone CP-8851 ( SEPAABBCCDDEEFF ) ['SERVER-FQDN  Active']

Use Case: Transient Connection Events on Cisco Unified Communications Manager generate syslogs that report back a phone/endpoint IP address that attempted to register against the server but failed mid-process. This can be due to rehoming to a higher priority server, lack of server side configuration for the endpoint, loss of network on the side of the endpoint. Looking through the web pages manually takes significantly more time to identify the MAC address associated with the endpoint. As the IP can typically vary for the endpoints their configurations are saved against the phone model and MAC address. This dramatically speeds up the collection of those models, mac addresses, and IF the phone is actively registered against a server (e.g. if Active is found in the html response -- results2), report back the server FQDN/IP and Active (as they are held within the same html tag).

Again, I'm a novice, but in about an hour this morning I was able to get this rolling and it's functional for what it is.

Script to scrape web gui for string, errors on Python 2.7 runs on Python 3

2 Answers2