2

I am trying to automate a process. For that I need to fetch XML by hitting a URL, multiple times in 1 run, and then parse it. For 1 run of the program, the URL could be hit anywhere between 4 to 25 times. This all seems fine until a 403 error response is returned.

Interestingly, the 403 always comes up for every 5th or 6th time the URL is hit.

I am using JDOM to parse the XML response.

I have tried the codes:

Document doc = builder.build(new InputSource(url.openStream()));

and 

HttpURLConnection conn = (HttpURLConnection)url.openConnection();
conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB;     rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 (.NET CLR 3.5.30729)"); 
Document doc = builder.build(conn.getInputStream());

With the second one I get the Exception:

org.jdom.input.JDOMParseException: Error on line 1: White spaces are required between publicId and systemId.

Could someone please help me in getting rid of the 403. Please note that I do not have any control over the source if a change is required to be made as talked about here

Also, I am not sure if this link is helpful.

Thank you.

[UPDATE 1]: This is somehow working, without having to sleep: 
try{
            doc = builder.build(conn.getInputStream());
        }catch(IOException ioEx){
            doc = builder.build(new InputSource(url.openStream()));
}
Community
  • 1
  • 1
hkansal
  • 221
  • 5
  • 16
  • 2
    Did you try to 'slow down' the processing, e.g. sleep after each call? – home Aug 09 '11 at 14:02
  • I did think about it but have not tried it yet. Do you think it would be better to first hit normally and then `sleep` momentarily if a 403 is received? I want to avoid having to `sleep` after each hit. – hkansal Aug 09 '11 at 14:06
  • Why do you have to hit the page so many times? Does the XML change each time? If not can't you just temporarily store all of it and parse out what you need? – Collecter Aug 09 '11 at 14:13
  • It's a different XML response each time as the query is different. I really wish I could get 1 XML response for all my queries. But unfortunately that's not the case. – hkansal Aug 09 '11 at 14:17
  • 1
    Alright. Try putting it to sleep then like home suggested. Chances are they are forbidding you from accessing the page because of the multiple queries in a very short period of time. – Collecter Aug 09 '11 at 14:19
  • Maybe the web server is refusing your successive requests. – Evan Mulawski Aug 09 '11 at 14:21
  • Thank you for trying to help. I understand that the server could be refusing requests because of the bulk. But here's something more interesting that is mentioned as [UPDATE 1]. I will watch it for about 2 days and get back to you people. Thanks for the magic support :) – hkansal Aug 09 '11 at 14:52
  • But you should look whether they have a req/sec and work from there. – ssedano Aug 09 '11 at 14:54
  • It looks all like they have some limit on petitions per seconds. The time it takes to create a new InputSource is the key. You should ask the website (do they have documented that resource?) if they have such limitation. – ssedano Aug 10 '11 at 07:11
  • As all of you said, the problem I think was with the bulk and a small pause was required. The exception once thrown above, gives the required pause, perhaps. Now, it's working fine. Thank you everyone! – hkansal Aug 18 '11 at 11:43

1 Answers1

3

403 means that the request is understood but the server refuses to process it. Look the headers you send. And when fails run a TRACE http method to retrieve the exact petition you are performing.

When you stablish an http connection you send along with the request the method you want to perform.

One of this methods is TRACE.

By performing a TRACE method you can see in the body response the petition you just performed. So you can see if it is still valid.

Maybe you are exceeding the max number of petitions if they had any mechanism.

ssedano
  • 8,322
  • 9
  • 60
  • 98
  • Kindly pardon my ignorance but I'm not really sure what we mean by running a `debug` http method. Is it as done [here](http://www.discursive.com/books/cjcook/reference/http-webdav-sect-debug-http)? – hkansal Aug 09 '11 at 14:41