Using ruby to retrieve a document from a website

Question

I have written a script in ruby that navigates through a website and gets to a form page. Once the form page is filled out the script hits the submit button and then a dialogbox opens asking you where to save it too. I am having trouble trying to get this file. I have searched the web and cant find anything. How would i go about retrieveing the file name of the document?

I would really appreciate if someone could help me

My code is below:

browser = Mechanize.new

## CONSTANTS
LOGIN_URL = 'https://business.airtricity.com/ews/welcome.jsp'
HOME_PAGE_URL = 'https://business.airtricity.com/ews/welcome.jsp'
CONSUMPTION_REPORT_URL = 'https://business.airtricity.com/ews/touConsChart.jsp?custid=209495'
LOGIN = ""
PASS = ""
MPRN_GPRN_LCIS = "10000001534"
CONSUMPTION_DATE = "20/01/2013"
END_DATE = "27/01/2013"
DOWNLOAD = "DL"



### Login page
begin
    login_page = browser.get(LOGIN_URL)
rescue Mechanize::ResponseCodeError => exception
    login_page = exception.page
end

puts "+++++++++"
puts login_page.links
puts "+++++++++"

login_form = login_page.forms.first
login_form['userid'] = LOGIN
login_form['password'] = PASS
login_form['_login_form_'] = "yes"
login_form['ipAddress'] = "137.43.154.176"
login_form.submit



## home page
begin
    home_page = browser.get(HOME_PAGE_URL)
rescue Mechanize::ResponseCodeError => exception
    home_page = exception.page
end

puts "----------"
puts home_page.links
puts "----------"


# Consumption Report
begin
    Report_Page = browser.get(CONSUMPTION_REPORT_URL)
rescue Mechanize::ResponseCodeError => exception
    Report_Page = exception.page
end

puts "**********"
puts Report_Page.links
pp Report_Page
puts "**********"

Report_Form = Report_Page.forms.first
Report_Form['entity1'] = MPRN_GPRN_LCIS
Report_Form['start'] = CONSUMPTION_DATE
Report_Form['end'] = END_DATE
Report_Form['charttype'] = DOWNLOAD
Report_Form.submit

## Download Report

begin
    browser.pluggable_parser.csv = Mechanize::Download
    Download_Page = browser.get('https://business.airtricity.com/ews/touConsChart.jsp?custid=209495/meter_read_download_2013-1-20_2013-1-27.csv').save('Hello')
rescue Mechanize::ResponseCodeError => exception
    Download_Page = exception.page
end

There is no error. I just cant find any code on how to take the file and save it somewhere. Like i need to download the file from the website with a script and then extract all the information from the file. I just dont know how to retrieve the file from the dialogbox that pops up when you do it manually from the website — thedarkknight228, Jan 29 '13 at 15:22
This question/answers provide some ways to retrieve pages/documents that are available at a URL, as well as some ruby libraries for curl-like functionality. http://stackoverflow.com/questions/929652/equivalent-of-curl-for-ruby — jefflunt, Jan 29 '13 at 15:30
Would i use Nokogiri to get the url of the document and then use curl to retrieve the file? — thedarkknight228, Jan 29 '13 at 15:33
You'd use Nokogiri, which is already loaded by Mechanize and passed to you. You could use Curb, or OpenURI or many other gems to retrieve the document. It's not a file until you save it. In transit it's just data on the wire. — the Tin Man, Jan 29 '13 at 17:09

score 1 · Answer 1 · answered Jan 29 '13 at 15:40

http://mechanize.rubyforge.org/Mechanize.html#method-i-get_file
File downloading from url it's pretty straightforward with mechanize:

browser = Mechanize.new
file_url = 'https://raw.github.com/ragsagar/ragsagar.github.com/c5caa502f8dec9d5e3738feb83d86e9f7561bd5e/.html'
downloaded_file = browser.get_file file_url
File.open('new_file.txt', 'w') { |file| file.write downloaded_file }

score 0 · Answer 2 · answered Jan 29 '13 at 16:28

0

I've seen automation fail because of the browser agent. Perhaps you could try

browser.user_agent_alias = "Windows Mozilla"

answered Jan 29 '13 at 16:28

Gregory Ostermayr

1,123
10
17

Using ruby to retrieve a document from a website

2 Answers2