I am trying to develop a PowerShell tool to gather "Discovery" information about devices on our network. We already have a commercial discovery tool but in quite a lot of cases, it is not giving us very much information.
The idea is to probe a subnet for devices (typically appliances) that have Web-based management interfaces. Our theory is that in many cases the home page content will allow us to detect what sort of device it is (containing manufacturer name, device model name etc). Obviously, such info will need to be extracted by parsing the body of the page. So, the script I have written first uses Test-NetConnection to do a port 80 test and a port 443 test. If the device is listening on port 80 or 443 the script will then use Invoke-WebRequest to grab the contents of the page.
I have used some code from here Ignore SSL warning with powershell downloadstring to disable certificate warnings as a lot of these devices will have self-signed untrusted certificates. That all works OK. The problem that I am having is that some of the devices I am testing on will display a page in a browser but using PowerShell's Invoke-WebRequest raises an error. After some investigation, this is because the Webserver of the device returns a non 200 status code. An example of this is setting up Apache on a Linux Box and enabling https with a self-signed certificate. Accessing the page using MS Edge displays the "Testing 123" page with a not secure warning on the address bar. However, accessing the same page via Invoke-WebRequest throws an exception. In this particular case it is because Apache returns a 403 Forbidden error. This is by design for Apache straight out of the box with the "Require all denied" setting in the httpd.conf file. Of course, the exception can be caught (which I have done) but the web page content is not available in this case within PowerShell even though it is displayed in a browser.
My next thoughts were that the Web server is behaving differently because it knows the PowerShell script is not one of the common browsers. So, I tried to use the -UserAgent parameter to Invoke-WebRequest to fooling the server into behaving the same way as it does with the browser and returning the content. However, this does not achieve what I am looking for.
The 403 return is just one example. It seems Appliances with a home page that require credentials (most/all? of them) returns a 401 error and again the page content is not available within PowerShell.
Does anyone have any pointers as to how I can make this work?
