6

I try to use the code as below to get the website htm source and it works. However, I cannot get the result when I visit the website http://reserve.apple.com/WebObjects/ProductReservation.woa/wa/reserveProduct by using code as below. But, I can access this page by using browser properly. Would you give me some hints or tips to fix this problem? Thank you.

#!/usr/bin/perl

use strict;
use warnings;

# create a new browser
use WWW::Mechanize;
my $browser = WWW::Mechanize->new();

# tell it to get the main page

my $sURL = 'http://www.apple.com';

#my $sURL = 'http://reserve.apple.com/WebObjects/ProductReservation.woa/wa/reserveProduct';

$browser->get($sURL);

print $browser->content;

exit(0);
Tommy Liu
  • 239
  • 4
  • 13

1 Answers1

6

It's a strange behavior, but site at url you want to retrieve requires following headers to be defined: Accept, Accept-Encoding, Accept-Language, Accept-Charset, Cookie.

Otherwise server does not respond at all.

You can easy do this just inserting following code before your "get" request:

$browser->add_header(
    "Accept"          => "",
    "Accept-Encoding" => "",
    "Accept-Language" => "",
    "Accept-Charset"  => "",
    "Cookie"          => ""
);

Instead of empty fields you can insert some real values, but this works too.

yko
  • 2,710
  • 13
  • 15
  • yko - Awesome! Actually, I also think of that. But, would you share with me why you know that? Thank you. – Tommy Liu Nov 13 '11 at 13:23
  • @TommyLiu it's just an experience. I copied headers from chrome request and put them into test mechanize request. And once it worked I started to remove headers one by one. Oops - it stopped to work if I remove any header. Then I removed headers content chunk by chunk in attempt to find significant data. I actually was surprised when I found it doesn't need any data. Just headers names – yko Nov 13 '11 at 13:28
  • I was able to strip the headers down to `'Accept' => 'text/html', 'Accept-Language' => 'en', 'Cookie' => ''`. – Alan Haggai Alavi Nov 13 '11 at 13:28
  • Nice catch. I looked into this a bit earlier today, but didn't get anywhere with the usual suspects. – flesk Nov 13 '11 at 18:46
  • @flesk thanks, I just had already similar situation somewhere. – yko Nov 13 '11 at 19:02
  • yko - Do you have any idea on this question http://stackoverflow.com/questions/8147006/website-button-click-perl-wwwmechanize? Thank you. – Tommy Liu Nov 16 '11 at 05:50