Ruby's open-uri and cookies

Question

I would like to store the cookies from one open-uri call and pass them to the next one. I can't seem to find the right docs for doing this. I'd appreciate it if you could tell me the right way to do this.
NOTES: w3.org is not the actual url, but it's shorter; pretend cookies matter here.

h1 = open("http://www.w3.org/")
h2 = open("http://www.w3.org/People/Berners-Lee/", "Cookie" => h1.FixThisSpot)

Update after 2 nays: While this wasn't intended as rhetorical question I guarantee that it's possible. Update after tumbleweeds: See (the answer), it's possible. Took me a good while, but it works.

For what you're trying to do I'd recommend using [Mechanize](http://mechanize.rubyforge.org/mechanize/). It's designed for this sort of thing. From its description: "The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, can follow links, and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history." — the Tin Man, Jan 08 '11 at 04:44
That mechanize link is dead, here's the new one http://mechanize.rubyforge.org/ — MCB, Feb 28 '14 at 15:14
Mechanize is now on github: https://github.com/sparklemotion/mechanize — JESii, Apr 07 '14 at 15:35

dlamblin · Accepted Answer · 2009-09-02T14:28:09.280

I thought someone would just know, but I guess it's not commonly done with open-uri. Here's the ugly version that neither checks for privacy, expiration, the correct domain, nor the correct path:

h1 = open("http://www.w3.org/")
h2 = open("http://www.w3.org/People/Berners-Lee/",
          "Cookie" => h1.meta['set-cookie'].split('; ',2)[0])

Yes, it works. No it's not pretty, nor fully compliant with recommendations, nor does it handle multiple cookies (as is).

Clearly, HTTP is a very straight-forward protocol, and open-uri lets you at most of it. I guess what I really needed to know was how to get the cookie from the h1 request so that it could be passed to the h2 request (that part I already knew and showed). The surprising thing here is how many people basically felt like answering by telling me not to use open-uri, and only one of those showed how to get a cookie set in one request passed to the next request.

score 13 · Answer 2 · answered Sep 01 '09 at 06:31

13

You need to add a "Cookie" header.

I'm not sure if open-uri can do this or not, but it can be done using Net::HTTP.

# Create a new connection object.
conn = Net::HTTP.new(site, port)

# Get the response when we login, to set the cookie.
# body is the encoded arguments to log in.
resp, data = conn.post(login_path, body, {})
cookie = resp.response['set-cookie']

# Headers need to be in a hash.
headers = { "Cookie" => cookie }

# On a get, we don't need a body.
resp, data = conn.get(path, headers)

answered Sep 01 '09 at 06:31

Matthew Schinckel

35,041
6
86
121

2

Yes, open-uri can send additional headers: open(url, "Cookie" => cookie) # http://www.ruby-doc.org/stdlib/libdoc/open-uri/rdoc/classes/OpenURI.html – glenn jackman Sep 01 '09 at 12:49
Does this Net::HTTP version take into account the cookie expiration, path, or domain? – dlamblin Sep 02 '09 at 00:53
@dlamblin No, deliberately not. First, learn to crawl, then walk, then run. – Matthew Schinckel Nov 13 '12 at 04:50

score 4 · Answer 3 · edited Sep 26 '11 at 17:42

Thanks Matthew Schinckel your answer was really useful. Using Net::HTTP I was successful

        # Create a new connection object.
          site = "google.com"
          port = 80
          conn = Net::HTTP.new(site, port)

        # Get the response when we login, to set the cookie.
        # body is the encoded arguments to log in.
          resp, data = conn.post(login_path, body, {})
          cookie = resp.response['set-cookie']

        # Headers need to be in a hash.
          headers = { "Cookie" => cookie }

        # On a get, we don't need a body.
          resp, data = conn.get(path, headers)

          puts resp.body

score 2 · Answer 4 · edited Aug 10 '20 at 15:51

2

Depending on what you are trying to accomplish, check out webrat. I know it is usually used for testing, but it can also hit live sites, and it does a lot of the stuff that your web browser would do for you, like store cookies between requests and follow redirects.

edited Aug 10 '20 at 15:51

builder-7000

7,131
3
19
43

answered Sep 02 '09 at 04:41

John F. Miller

26,961
10
71
121

2

I'd recommend Mechanize instead. It will hit live sites, handle cookies and follow redirects too, and, it was actually designed to do all that. – the Tin Man Jan 08 '11 at 04:43

ADAM · Answer 5 · 2011-01-08T01:12:35.353

1

you would have to roll your own cookie support by parsing the meta headers when reading and adding a cookie header when submitting a request if you are using open-uri. Consider using httpclient http://raa.ruby-lang.org/project/httpclient/ or something like mechanize instead http://mechanize.rubyforge.org/ as they have cookie support built in.

edited Jan 08 '11 at 01:12

answered Sep 01 '09 at 06:28

ADAM

3,903
4
29
45

I'm afraid "no support for cookies" is too strong of a choice of words. I do appreciate the links though. The latter's documentation seems sparse. http://mechanize.rubyforge.org/mechanize/WWW/Mechanize/CookieJar.html – dlamblin Sep 02 '09 at 00:57
Ruby's Mechanize is closely based on Perl's WWW::Mechanize, which has some nice docs. The description of cookies in the Perl docs should help figure out how the Ruby version works. It's been a while since I used it, but I think it'll supply a cookie jar and will handle them automatically. You can define your own jar if you want to switch them in or out or store it on disk for later reuse. – the Tin Man Jan 08 '11 at 04:52

score 0 · Answer 6 · answered Oct 13 '11 at 12:21

0

There is a RFC 2109 and RFC 2965 cookie jar implementation to be found here for does that want standard compliant cookie handling.

https://github.com/dwaite/cookiejar

answered Oct 13 '11 at 12:21

Darwin

4,686
2
30
22

Ruby's open-uri and cookies

6 Answers6