How to get the page source with Mechanize/Nokogiri

Question

I'm logged into a webpage/servlet using Mechanize.

I have a page object:

jobShortListPg = agent.get(addressOfPage)

When I use:

puts jobShortListPg

I get the "mechanized" version of the page which I don't want:

#<Mechanize::Page::Link "Home" "blahICScriptProgramName=WEBLIB_MENU.ISCRIPT3.FieldFormula.IScript_DrillDown&target=main0&Level=0&RL=&navc=3171">

How do I get the HTML source of the page instead?

In Ruby we use snake_case, not camelCase for variables or method names. ItIsAReadabilityThing. — the Tin Man, Feb 14 '20 at 03:48

score 35 · Accepted Answer · edited Feb 14 '20 at 03:56

35

Use .body:

puts jobShortListPg.body

edited Feb 14 '20 at 03:56

the Tin Man

158,662
42
215
303

answered Jun 26 '11 at 22:34

Dogbert

212,659
41
396
397

I still get the mechanized version of the page :(. – Waley Chen Jun 26 '11 at 22:38
Are you sure you did `.body` on the page that was returned by `.get`? I get a pure string back. `Mechanize.new.get("http://www.google.com").body.class` => `String` – Dogbert Jun 26 '11 at 22:50
Silly me, I was working on the wrong file. Did .body on the right file and it worked! Thank you Dogbert! – Waley Chen Jun 26 '11 at 23:45
Mechanized version is better than source. they have many helpers in them – carbonr Dec 25 '12 at 09:01
@carbonr unless calling an API and the source is the JSON you actually need – Martin Capodici Oct 11 '13 at 20:41

score 2 · Answer 2 · edited Feb 14 '20 at 03:49

2

Use the content method of the page object.

jobShortListPg.content

edited Feb 14 '20 at 03:49

the Tin Man

158,662
42
215
303

answered Jan 25 '13 at 16:32

ramasamyz

43
3

score 0 · Answer 3 · answered Feb 14 '20 at 03:54

In Nokogiri use to_s or to_html on the main document Node:

require 'nokogiri'

doc = Nokogiri::HTML(<<EOT)
<html>
  <head></head>
  <body>foo</body>
</html>
EOT

doc.to_html
# => "<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.0 Transitional//EN\" \"http://www.w3.org/TR/REC-html40/loose.dtd\">\n" +
#    "<html>\n" +
#    "  <head><meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"></head>\n" +
#    "  <body>foo</body>\n" +
#    "</html>\n"

or:

doc.to_s
# => "<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.0 Transitional//EN\" \"http://www.w3.org/TR/REC-html40/loose.dtd\">\n" +
#    "<html>\n" +
#    "  <head><meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"></head>\n" +
#    "  <body>foo</body>\n" +
#    "</html>\n"

If it distracts you to see the embedded new-lines, this might help:

puts doc.to_s

# >> <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
# >> <html>
# >>   <head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head>
# >>   <body>foo</body>
# >> </html>

How to get the page source with Mechanize/Nokogiri

3 Answers3

Linked