Why does Mechanize HTML content print ?

Question

I want to login to my bank account with Mechanize (2.7.3) and print out the current balance.

But after successful login something's not quite right because Mechanize is not showing the correct information that's supposed to be on the page.

The output is interesting. Especially the  part.

Here's the code I'm using:

require 'mechanize'
require 'logger'

# Instantiate a new Mechanize object
a = Mechanize.new do |agent|
    agent.user_agent_alias = 'Mac Safari'
    agent.follow_meta_refresh = true
    agent.log = Logger.new "mech.log"
end

# Fetch URL with Mechanize
a.get('https://mijn.ing.nl/internetbankieren/SesamLoginServlet') do |page|
    login_form = page.forms.first

    # Store the randomly generated input names into variables
    username_input_name = login_form.fields[0].name
    password_input_name = login_form.fields[1].name

    # Fill in the username and password form
    login_form.field_with(:name => username_input_name).value = 'username'
    login_form.field_with(:name => password_input_name).value = 'password'

    # Login
    dashboard_page = login_form.submit(nil, {'Cookie' => HTTP::Cookie.cookie_value(a.cookie_jar.cookies)})

    puts dashboard_page.content

    # Check if the login was successfull
    puts    "=================="
    puts check_1 = dashboard_page.title == 'Mijn ING Overzicht  - Mijn ING' ?  "CHECK 1 LOGIN SUCCESS" : "CHECK 1 LOGIN FAIL"
    puts    "=================="
end

Here's the console output with a.follow_meta_refresh = true:

<html>
<body>
<!-- Session expired -->
<script>
<!-- Hide script from old browsers
function urlencode(str) {
return escape(str).replace('%3A', ':').replace('+', '%2B').replace('%20', '+').replace('*', '%2A').replace('/', '%2F').replace('@', '%40').replace('%2F', '/');
}
var url_encoded_referrer = urlencode(document.location);
var url = 'https://' + document.location.hostname + '/ssm/sso/login?Target=' + url_encoded_referrer;
window.location = url;
//-- Stop hiding script -->
</script>
</body>
</html>
==================
CHECK 1 LOGIN FAIL
==================

This is the logfile:

# Logfile created on 2013-12-27 09:16:17 +0100 by logger.rb/41954
I, [2013-12-27T09:16:17.112862 #978]  INFO -- : Net::HTTP::Get: /internetbankieren/SesamLoginServlet
D, [2013-12-27T09:16:17.112910 #978] DEBUG -- : request-header: accept-encoding => gzip,deflate,identity
D, [2013-12-27T09:16:17.112935 #978] DEBUG -- : request-header: accept => */*
D, [2013-12-27T09:16:17.112957 #978] DEBUG -- : request-header: user-agent => Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/534.51.22 (KHTML, like Gecko) Version/5.1.1 Safari/534.51.22
D, [2013-12-27T09:16:17.112978 #978] DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
D, [2013-12-27T09:16:17.112999 #978] DEBUG -- : request-header: accept-language => en-us,en;q=0.5
D, [2013-12-27T09:16:17.113019 #978] DEBUG -- : request-header: host => mijn.ing.nl
I, [2013-12-27T09:16:17.678728 #978]  INFO -- : status: Net::HTTPOK 1.1 200 OK
D, [2013-12-27T09:16:17.678787 #978] DEBUG -- : response-header: date => Fri, 27 Dec 2013 08:16:17 GMT
D, [2013-12-27T09:16:17.678813 #978] DEBUG -- : response-header: cache-control => no-cache, no-store
D, [2013-12-27T09:16:17.678834 #978] DEBUG -- : response-header: pragma => no-cache
D, [2013-12-27T09:16:17.678855 #978] DEBUG -- : response-header: expires => Thu, 01 Jan 1970 00:00:00 GMT
D, [2013-12-27T09:16:17.678877 #978] DEBUG -- : response-header: set-cookie => sessiontype=mpb; Secure, aac=332016DB0A85C1245596211F7D403A78; Expires=Sat, 27 Dec 2014 08:16:17 GMT; Domain=.ing.nl; Secure, internetbankierenmi=1575004352.20480.0000; path=/, TS765584=312694932a75764058a09928f7990ca23a092c849a2d804452bd3751d4efbe89959deba9debace3fc06c3994e27e0b6b0f2fcdc7; Path=/
D, [2013-12-27T09:16:17.678900 #978] DEBUG -- : response-header: vary => Accept-Encoding,User-Agent
D, [2013-12-27T09:16:17.678921 #978] DEBUG -- : response-header: content-encoding => gzip
D, [2013-12-27T09:16:17.678942 #978] DEBUG -- : response-header: keep-alive => timeout=90, max=1000
D, [2013-12-27T09:16:17.678962 #978] DEBUG -- : response-header: connection => Keep-Alive
D, [2013-12-27T09:16:17.678983 #978] DEBUG -- : response-header: content-type => text/html;charset=ISO-8859-1
D, [2013-12-27T09:16:17.679003 #978] DEBUG -- : response-header: content-language => en
D, [2013-12-27T09:16:17.679023 #978] DEBUG -- : response-header: transfer-encoding => chunked
D, [2013-12-27T09:16:17.679118 #978] DEBUG -- : Read 10 bytes (10 total)
D, [2013-12-27T09:16:17.679974 #978] DEBUG -- : Read 1443 bytes (1453 total)
D, [2013-12-27T09:16:17.680403 #978] DEBUG -- : Read 1448 bytes (2901 total)
D, [2013-12-27T09:16:17.680469 #978] DEBUG -- : Read 1112 bytes (4013 total)
D, [2013-12-27T09:16:17.680566 #978] DEBUG -- : gzip response
D, [2013-12-27T09:16:17.683042 #978] DEBUG -- : saved cookie: sessiontype=mpb
D, [2013-12-27T09:16:17.683568 #978] DEBUG -- : saved cookie: aac=332016DB0A85C1245596211F7D403A78
D, [2013-12-27T09:16:17.683684 #978] DEBUG -- : saved cookie: internetbankierenmi=1575004352.20480.0000
D, [2013-12-27T09:16:17.683778 #978] DEBUG -- : saved cookie: TS765584=312694932a75764058a09928f7990ca23a092c849a2d804452bd3751d4efbe89959deba9debace3fc06c3994e27e0b6b0f2fcdc7
I, [2013-12-27T09:16:17.685622 #978]  INFO -- : form encoding: ISO-8859-1
D, [2013-12-27T09:16:17.685989 #978] DEBUG -- : query: "a8yzXvCb7f5ZKsFaT=n.aramjan&a9jSD_u8G4nO23oXI=EYoss2303"
I, [2013-12-27T09:16:17.686528 #978]  INFO -- : Net::HTTP::Post: /internetbankieren/SesamLoginServlet
D, [2013-12-27T09:16:17.686562 #978] DEBUG -- : request-header: accept-encoding => gzip,deflate,identity
D, [2013-12-27T09:16:17.686587 #978] DEBUG -- : request-header: accept => */*
D, [2013-12-27T09:16:17.686609 #978] DEBUG -- : request-header: user-agent => Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/534.51.22 (KHTML, like Gecko) Version/5.1.1 Safari/534.51.22
D, [2013-12-27T09:16:17.686631 #978] DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
D, [2013-12-27T09:16:17.686653 #978] DEBUG -- : request-header: accept-language => en-us,en;q=0.5
D, [2013-12-27T09:16:17.686677 #978] DEBUG -- : request-header: cookie => TS765584=312694932a75764058a09928f7990ca23a092c849a2d804452bd3751d4efbe89959deba9debace3fc06c3994e27e0b6b0f2fcdc7; aac=332016DB0A85C1245596211F7D403A78; internetbankierenmi=1575004352.20480.0000; sessiontype=mpb
D, [2013-12-27T09:16:17.686711 #978] DEBUG -- : request-header: host => mijn.ing.nl
D, [2013-12-27T09:16:17.686739 #978] DEBUG -- : request-header: referer => https://mijn.ing.nl/internetbankieren/SesamLoginServlet
D, [2013-12-27T09:16:17.686760 #978] DEBUG -- : request-header: content-type => application/x-www-form-urlencoded
D, [2013-12-27T09:16:17.686781 #978] DEBUG -- : request-header: content-length => 55
I, [2013-12-27T09:16:18.225851 #978]  INFO -- : status: Net::HTTPOK 1.1 200 OK
D, [2013-12-27T09:16:18.225913 #978] DEBUG -- : response-header: date => Fri, 27 Dec 2013 08:16:17 GMT
D, [2013-12-27T09:16:18.225938 #978] DEBUG -- : response-header: x-ci => r=HPHBPNBMP;c=;a=VRRCSL;u=JXWMBKM
D, [2013-12-27T09:16:18.225962 #978] DEBUG -- : response-header: cache-control => no-cache, no-store, no-cache, no-store
D, [2013-12-27T09:16:18.225983 #978] DEBUG -- : response-header: pragma => no-cache, no-cache
D, [2013-12-27T09:16:18.226005 #978] DEBUG -- : response-header: expires => Thu, 01 Jan 1970 00:00:00 GMT
D, [2013-12-27T09:16:18.226030 #978] DEBUG -- : response-header: set-cookie => SESSESSIONID=0000e3Sar84Bz7XlOymbQkge4gX:17s9ivth6; Path=/; Domain=.ing.nl; Secure, iid=VQGIG0Xq3%2BzTjLSPo5YDiw%3D%3D%3Bmss1; Expires=Wed, 26 Feb 2014 08:16:17 GMT; Path=/; Domain=.ing.nl, gsc=m=s; HttpOnly; Path=/; Domain=.mijn.ing.nl; Secure, Session201=HKVCKUYMLURFZQIENENOANWSSNRUERNF051e7e43; HttpOnly; Path=/; Domain=.mijn.ing.nl; Secure, cookiepref=3; Expires=Sat, 27 Dec 2014 08:16:17 GMT; Path=/; Domain=.ing.nl, TS765584=9e3c30b00a5f1162344f71403a543e573a092c849a2d804452bd3751d4efbe89959deba9debace3fc06c3994e27e0b6b0f2fcdc7eccc06cabcea90646ac25a59abcf00af2532f3ca7d510a274552b96c4b433c8b5f6a631185c8cdc8; Path=/
D, [2013-12-27T09:16:18.226053 #978] DEBUG -- : response-header: vary => Accept-Encoding,User-Agent
D, [2013-12-27T09:16:18.226073 #978] DEBUG -- : response-header: content-encoding => gzip
D, [2013-12-27T09:16:18.226094 #978] DEBUG -- : response-header: keep-alive => timeout=90, max=1000
D, [2013-12-27T09:16:18.226115 #978] DEBUG -- : response-header: connection => Keep-Alive
D, [2013-12-27T09:16:18.226139 #978] DEBUG -- : response-header: content-type => text/html;charset=ISO-8859-1
D, [2013-12-27T09:16:18.226159 #978] DEBUG -- : response-header: content-language => en
D, [2013-12-27T09:16:18.226180 #978] DEBUG -- : response-header: transfer-encoding => chunked
D, [2013-12-27T09:16:18.226252 #978] DEBUG -- : Read 326 bytes (326 total)
D, [2013-12-27T09:16:18.226381 #978] DEBUG -- : gzip response
D, [2013-12-27T09:16:18.226730 #978] DEBUG -- : saved cookie: SESSESSIONID=0000e3Sar84Bz7XlOymbQkge4gX:17s9ivth6
D, [2013-12-27T09:16:18.226933 #978] DEBUG -- : saved cookie: iid=VQGIG0Xq3%2BzTjLSPo5YDiw%3D%3D%3Bmss1
D, [2013-12-27T09:16:18.227051 #978] DEBUG -- : saved cookie: gsc=m=s
D, [2013-12-27T09:16:18.227152 #978] DEBUG -- : saved cookie: Session201=HKVCKUYMLURFZQIENENOANWSSNRUERNF051e7e43
D, [2013-12-27T09:16:18.227311 #978] DEBUG -- : saved cookie: cookiepref=3
D, [2013-12-27T09:16:18.227423 #978] DEBUG -- : saved cookie: TS765584=9e3c30b00a5f1162344f71403a543e573a092c849a2d804452bd3751d4efbe89959deba9debace3fc06c3994e27e0b6b0f2fcdc7eccc06cabcea90646ac25a59abcf00af2532f3ca7d510a274552b96c4b433c8b5f6a631185c8cdc8
I, [2013-12-27T09:16:18.230318 #978]  INFO -- : Net::HTTP::Get: /particulier/betalen/index
D, [2013-12-27T09:16:18.230355 #978] DEBUG -- : request-header: accept-encoding => gzip,deflate,identity
D, [2013-12-27T09:16:18.230379 #978] DEBUG -- : request-header: accept => */*
D, [2013-12-27T09:16:18.230401 #978] DEBUG -- : request-header: user-agent => Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/534.51.22 (KHTML, like Gecko) Version/5.1.1 Safari/534.51.22
D, [2013-12-27T09:16:18.230421 #978] DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
D, [2013-12-27T09:16:18.230456 #978] DEBUG -- : request-header: accept-language => en-us,en;q=0.5
D, [2013-12-27T09:16:18.230478 #978] DEBUG -- : request-header: cookie => SESSESSIONID=0000e3Sar84Bz7XlOymbQkge4gX:17s9ivth6; Session201=HKVCKUYMLURFZQIENENOANWSSNRUERNF051e7e43; cookiepref=3; gsc=m=s; iid=VQGIG0Xq3%2BzTjLSPo5YDiw%3D%3D%3Bmss1
D, [2013-12-27T09:16:18.230501 #978] DEBUG -- : request-header: host => bankieren.mijn.ing.nl
I, [2013-12-27T09:16:18.714637 #978]  INFO -- : status: Net::HTTPOK 1.1 200 OK
D, [2013-12-27T09:16:18.714692 #978] DEBUG -- : response-header: content-length => 506
D, [2013-12-27T09:16:18.714716 #978] DEBUG -- : response-header: content-type => text/html
D, [2013-12-27T09:16:18.714738 #978] DEBUG -- : response-header: date => Fri, 27 Dec 2013 08:16:18 GMT
D, [2013-12-27T09:16:18.714760 #978] DEBUG -- : response-header: p3p => CP="NON CUR OTPi OUR NOR UNI"
D, [2013-12-27T09:16:18.714781 #978] DEBUG -- : response-header: cache-control => no-cache
D, [2013-12-27T09:16:18.714801 #978] DEBUG -- : response-header: pragma => no-cache
D, [2013-12-27T09:16:18.714825 #978] DEBUG -- : response-header: set-cookie => S-SESSION-ID=2_0_nG1XBMZWH2kJdOBjCloZA651nXttTm5wiYowRBYDQY7gxH1L; Path=/; Secure; HttpOnly, lb-4-25a-1=208176394.20480.0000; path=/, TS31e294=06caec21f242b201e4b52847e45a8a475bd58411f71b544852bd3752022ecc54fb717d9b1b4d09a87ea0bb21; Path=/
D, [2013-12-27T09:16:18.714889 #978] DEBUG -- : Read 506 bytes (506 total)
D, [2013-12-27T09:16:18.715236 #978] DEBUG -- : saved cookie: S-SESSION-ID=2_0_nG1XBMZWH2kJdOBjCloZA651nXttTm5wiYowRBYDQY7gxH1L
D, [2013-12-27T09:16:18.715340 #978] DEBUG -- : saved cookie: lb-4-25a-1=208176394.20480.0000
D, [2013-12-27T09:16:18.715437 #978] DEBUG -- : saved cookie: TS31e294=06caec21f242b201e4b52847e45a8a475bd58411f71b544852bd3752022ecc54fb717d9b1b4d09a87ea0bb21

Any ideas? It would be great to solve this on Christmas day!

Maybe you need to send the cookies alongs with the submit request, like this: `login_form.submit(nil, {'Cookie' => HTTP::Cookie.cookie_value(a.cookie_jar.cookies)})` — Ahmad Sherif, Dec 25 '13 at 14:05
You are probably being blocked, try setting agents like Mozila or Sfari, look that in Google. — PepperoniPizza, Dec 27 '13 at 00:00
@PepperoniPizza the result is still the same. I've used `logger` this time to log an attempt. Check out the contents of the logfile above, might be useful? Thanks — narzero, Dec 27 '13 at 08:20

score 1 · Accepted Answer · edited May 23 '17 at 12:19

1

Double check if cookies are persisted across clicks, and try selenium - it's possible some funny JS stuff is going on that page and selenium would handle it for you (http://rubygems.org/gems/selenium-webdriver)

UPDATE

See this answer for more info on cookie handling in Mechanize: Maintaining cookies between Mechanize requests

Also, try disabling JS in the browser and then log into your bank, make sure everything still works without JS (if it works then great, if not then you need to use a JS-capable driver)

CONCLUSION

Cause in this case was the bank's website, it depends on client-side JavaScript code to function properly so a JS-incapable driver like mechanize won't do. You will need to use one of the JavaScript-capable drivers - most popular of which are selenium, webkit and poltergeist - all decent, but neither really satisfying.

Considering all three require some host-side dependencies (poltergeist least of all though) the "what installs best" might be the prevailing factor in choosing between them.

For a Heroku deployment....

poltergeist does seem to have some issues (websocket related, seems heroku is blocking them) but it's the best bet for heroku deployment https://github.com/jonleighton/poltergeist/issues/194
and for selenium and webkit, quick google shows a number of issues - they seem to require X and/or xvfb, see Is it possible to run capybara-webkit (i.e. forked webkit_server) on Heroku Cedar? and Is there a working nodejs/phantomjs Heroku buildpack? and Running selenium browser on server (Flask/Python/Heroku) for further reading

edited May 23 '17 at 12:19

Community

1
1

answered Jan 06 '14 at 19:34

bbozo

7,075
3
30
56

How do I double check if cookies are persisted across clicks? I don't like Selenium. I've got this working successfully with a Selenium alternative called Watir (http://watir.com/). But I want to solve the Mechanize problem and get that working. – narzero Jan 07 '14 at 09:35
@narzero I updated the answer with an SO link about cookie handling with Mechanize – bbozo Jan 07 '14 at 09:41
Thank you for the updated answer. I disabled JS and logged into my bank account. It did not work, I get to see a blank page and when I take a look at the source of the page it's exactly the same as the console output of my Mechanize code. It's safe to say that I need a JS-capable driver. Do you happen to know a good one? – narzero Jan 07 '14 at 13:28
@narzero, nah, they're all bad :) (see my rant on http://stackoverflow.com/a/20888890/1983072) I've had decent-enough experiences with webkit, selenium and poltergeist. I mostly use poltergeist for headless execution but it has its quirks. More problems arise when you try and deploy those tests on a remote host (dependencies) so take *"what installs best on my deployment host"* into consideration too. But... If you've got watir working, why not go watir? – bbozo Jan 07 '14 at 13:43
I'm not going with Watir because I can't deploy it to Heroku. Yes the answer covers my initial question. Please update your original answer with the conclusion/cause and I'll accept it. Cheers – narzero Jan 07 '14 at 22:14

Why does Mechanize HTML content print ?

1 Answers1