urllib2 returns HTTP 404 from web service that is up

Question

I have a Spring 3 web service that serves up some data. It works perfectly from Firefox, but when I tried to access it via a simple Python script using urllib2, I consistently get back HTTP 404.

This happens whether or not I am running my web service via Tomcat under Eclipse, or Tomcat running as a Windows service.

The only thing I can think of (which still doesn't seem likely) is that something about the web service (under the hood) is unhappy about the urllib2 user agents string...

Can anyone give me some ideas about what to try next?

Thanks,

Mitch

Here is a simplified version of the code, followed by the screen output:

import sys
import urllib2
import urllib
import datetime
import time
import httplib
from datetime import timedelta

url = 'http://localhost:8086/OamDataWebService/oamdatawebservice/oamdata5    /SYRC01TAMP20/1334127600000/1334131199000'

handler=urllib2.HTTPHandler(debuglevel=1)
opener = urllib2.build_opener(handler)
urllib2.install_opener(opener)

req = urllib2.Request(url=url)
req.add_header('Content-Type', 'text/xml')

try:
        resp = urllib2.urlopen(req)
except urllib2.HTTPError, e:
        print "ERROR: caught HTTPError exception"
        print "HTTP error code:", e.code
        print e.read()
        sys.exit(1)

content = resp.read()

print content

$ python test.py send: 'GET /OamDataWebService/oamdatawebservice/oamdata5/SYRC01TAMP20/1334127600000/1334131199000 HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: tbdivb2400 2.corp.local:8086\r\nContent-Type: text/xml\r\nConnection: close\r\nUser-Agent: Python-urllib/2.7\r\n\r\n' reply: 'HTTP/1.1 404 Not Found\r\n' header: Server: Apache-Coyote/1.1 header: Content-Type: text/html;charset=utf-8 header: Content-Length: 952 header: Date: Fri, 13 Apr 2012 13:56:28 GMT header: Connection: close ERROR: caught HTTPError exception HTTP error code: 404 Apache Tomcat/6.0.35 - Error report

HTTP Status 404 -

type Status report

message
description The requested resource () is not available.
Apache Tomcat/6.0 .35

I think the problem is in my Spring Controller configuration, but I still don't understand why it works so consistently one way and not from Python. I am running now from Chrome,and I believe I am sending the same thing.

My Tomcat logs show the following when I send the request via urllib:

2012-04-13 14:31:26,782 WARN org.springframework.web.servlet.PageNotFound.handleNoSuchRequestHandlingMethod:142 - No matching handler method found for servlet request: path '/oamdata5/SYRC01TAMP20/1334127600000/1334131199000', method 'GET', parameters map[[empty]]

My Spring web.xml servlet mapping is:

<servlet-mapping>
    <servlet-name>spring</servlet-name>
    <url-pattern>/</url-pattern>
</servlet-mapping>

And my RequestMapping entry is:

@RequestMapping(value = "/{interval}/{serviceId}/{startTime}/{endTime}", 
    method = RequestMethod.GET, 
    headers="Accept=application/xml, application/json")

A lot of servers are unhappy about the Python user-agent for a good reason. If this is the case, my code sample should help. — Fenikso, Apr 13 '12 at 13:15
Have you tried running some "hello world" type app on your server and getting the HTML by Python? — Fenikso, Apr 13 '12 at 15:00

score 1 · Answer 1 · answered Apr 13 '12 at 14:15

1

Your demo code contains a few whitespaces in the URL after oamdata5. Is that correct?

answered Apr 13 '12 at 14:15

I don't know how that got in there, the real code does not have this... I can copy the URL directly from Firefox, and it is okay there. – mitchmcc Apr 13 '12 at 14:33
Still the same problem... I have been looking at this all day! I did pull up the Tomcat logs, and I see what is probably the big clue: – mitchmcc Apr 13 '12 at 18:34

score 0 · Answer 2 · answered Apr 13 '12 at 13:12

0

Just set the user-agent to Firefox.

headers = {"User-agent": "Mozilla/5.0"}
request = urllib2.Request(url, None, headers)
result = urllib2.urlopen(request)
html = result.read()

answered Apr 13 '12 at 13:12

Fenikso

9,251
5
44
72

score 0 · Answer 3 · answered Apr 13 '12 at 13:14

0

Use Wireshark to capture the request that is being sent by urllib2. I guess that this way, you should be able to spot what's wrong.

answered Apr 13 '12 at 13:14

LubosD

781
6
18

score 0 · Answer 4 · edited May 23 '17 at 11:56

What produced the 404? What is the actual content of the 404 page (they frequently explain what the problem is)? Are you sure it was the destination website? Sometimes filtering firewalls will reject outgoing HTTP connections which don't go via an internal authenticating proxy. Does the code successfully fetch HTTP content from other external websites?

OK, since the server is on the same machine and we get an error in the Tomcat log, it's not a firewall problem. My next suggestion is to set up a local HTTP proxy (see for example http://code.google.com/p/python-proxy/ or seriously simple python HTTP proxy?). Then change your web browser config to use it. Turn on as much logging in the proxy as you can. Fetch the page from your browser, save the log data, and then try to fetch the data from your python code (having updated it to use the proxy). You should be able to figure out the critical difference by comparing the data logged by the proxy.

I am running this from a Windows cmd.exe terminal to Tomcat running as a Windows service on my own box. I have verified that the URL and Tomcat port are correct (they work in Firefox)... — mitchmcc, Apr 13 '12 at 14:09
James, yes, the same code gets data okay from weather.yahoo.com (for example). — mitchmcc, Apr 13 '12 at 14:41

urllib2 returns HTTP 404 from web service that is up

HTTP Status 404 -

Apache Tomcat/6.0 .35

4 Answers4