20

I'm trying to write a simple script that fetches text from a webpage and processes that string. But, that website requires me to be logged in. I was successful in logging in to that website. This is how I logged in:

var payload = {"name1":"val1","name2":val2"};

var opt ={"payload":payload,"method":"post"};

var respose = UrlFetchApp.fetch("http://website.com/login",opt);

After logging in, the website places me in http://website.com/home. I checked response.getContentText() and I can confirm that I have been logged in successfully as it contains the text from http://website.com/home. Now I need to get the contents of http://website.com/page and process it. I first assumed the script can handle cookies by itself and proceeded with 

var pagedata = UrlFetchApp.fetch("http://website.com/page);//Did not work

That obviously didnt work and pagedata.getContentText() says me to login first, which indicates cookies were not successfully passed..

I then tried to extract cookies which the server responded during login and to send it along with this request.

var cookie = response.getAllHeaders()['Set-Cookie'];     

// variable cookie now contains  a legitimate cookie.

// It contains 'JSESSIONID=blabla;Path=/' and 
// it is the ONLY cookie that server responds.

I tried to send that cookie in my page request.

var header = {'Cookie':cookie};

var opt2 = {"header":header};

var pagedata = UrlFetchApp.fetch("http://website.com/page",opt2);

I think even now cookies were not properly sent, as the content again says me to login.

Am I passing cookies correctly? I need help regarding the correct method of sending cookies in a request.

Rubén
  • 34,714
  • 9
  • 70
  • 166
akarthik10
  • 322
  • 1
  • 4
  • 11

2 Answers2

12

Here you can find cookies specification: http://www.w3.org/Protocols/rfc2109/rfc2109

You have a potential issue in your code: response.getAllHeaders()['Set-Cookie'] can return either a string or a table of string if multiple 'set-cookie' attributes are sent back from the server.

Eric is right, you cannot return the cookie without digesting it.

Second error in your code:

var opt2 = {"header":header};

should be

var opt2 = {"headers":header};

Be aware also that GAS uses Google IPs. It can happen that two consecutive fetch use different IPs. The server your are connecting to may be session-IP dependant.

Are you sure the server only send you back one cookie after an authentification ?

Thierry Chevillard
  • 504
  • 3
  • 9
  • Thanks for pointing out it!. Yep, i'm sure that server sends only one auth cookie. I'll try and report back with `"headers"` – akarthik10 Jun 06 '12 at 12:04
  • 1
    EDIT : I tried it with `"headers"` and it worked. I can confirm it is working. I made a PHP file, that records all the headers with which it was requested. I requested it with UrlFetchApp and I could see in my log, all the custom headers I sent from UrlFetchApp. So where I made mistake was `"header"` and it must be `"headers"`. Thank you! – akarthik10 Jun 06 '12 at 12:57
3

It looks like you are setting the headers correctly in UrlFetchApp.fetch().

I believe that the data in the Set-Cookie header is in a different format than the data that is expected in Cookie header. For example, Set-Cookie contains information about expiration, etc.

Eric Koleda
  • 12,420
  • 1
  • 33
  • 51
  • I've verified that it is `Set-Cookie: JSESSIONID=A04~1E321E3265CB36814498F30B8D134667.w804; Path=/` . I have tried passing the entire cookie (including `Path=/`) and sending only the `JSESS..` (using split with ; as delimiter) cookie. Both have failed. – akarthik10 Jun 04 '12 at 17:45