Ruby, Feedzirra, typhoeus

Question

I'm really new to Ruby (first day!) and I'm struggling with this here. I'm trying to build a rss-parser in combination with Typhoeus since I'm parsing over 100 feeds and also because I want to get it working.

So that is my code here:

require 'typhoeus'
require 'feedzirra'

feed_urls = ["feed1", "feed2"]

hydra = Typhoeus::Hydra.new
feeds = {}
entry = {}
feed_urls.each do |feed|
  r = Typhoeus::Request.new(feed)
  r.on_complete do |response|
    feeds[r.url] = response.body
    feeds[r.url] = Feedzirra::Feed.parse(response.body)
    entry = feeds.entries.each do |entry|
      puts entry.title
    end
    hydra.queue r
  end
end

hydra.run

I'm sure it's some syntax problem. I still have some hard times. For example I always keep closing lines with ; although I forget it all the time when writing PHP. So, maybe someone could help? Getting feed-results without typhoeus wasn't too hard.

edit:

>> puts feeds.entries.inspect
[["http://feedurl", #<Feedzirra::Parser::AtomFeedBurner:0x1023b53f0 @title="Paul Dix Explains Nothing", @entries=[#<Feedzirra::Parser::AtomFeedBurnerEntry:0x1023b05d0 @published=Thu Jan 13 16:59:00 UTC 2011, @author="Paul Dix", @summary="Earlier this week I had the opportunity to sit with six other people from the NYC technology scene and talk to NYC Council Speaker Christine Quinn and a few members of her staff. Charlie O'Donnell organized the event to help...", @updated=Thu Jan 13 17:55:31 UTC 2011, @title="Water water everywhere and not a drop to drink: The Myth and Truth of the NYC engineer shortage", @entry_id="tag:typepad.com,2003:post-6a00d8341f4a0d53ef0148c793f692970c", @content="....

So, I at least get something.

In the future, try to remember to put a `ruby` tag on your question so that those who help with Ruby see it. — Phrogz, Jun 10 '11 at 23:25
I edited your question for indentation. What you're doing now isn't going to work at all though since the code is pushing the Typhoeus request into the Hydra queue inside the complete callback. Can you verify that your code doesn't do this and add a bit more information about what problem you're having into the question? — coreyward, Jun 11 '11 at 00:55

Thiago Silveira · Accepted Answer · 2011-06-11T00:45:26.483

2

It seems that you are queing inside the on_complete block. Shouldn't you queue in the feed_urls.each block? Or maybe you should go through all the entries after all requests have been completed? Like this:

hydra = Typhoeus::Hydra.new
feeds = {}
entry = {}
feed_urls = ["feed1", "feed2"]

feed_urls.each do |feed|
  r = Typhoeus::Request.new(feed)
  r.on_complete do |response|
      feeds[r.url] = response.body
      feeds[r.url] = Feedzirra::Feed.parse(response.body)
  end

  hydra.queue r
end

hydra.run

feeds.entries.each do |feed|
  puts "-- " + feed[1].title

  feed[1].entries.each do |entry|
    puts entry.title
  end
end

edited Jun 11 '11 at 00:45

answered Jun 10 '11 at 23:54

Thiago Silveira

5,033
4
26
29

i tried that but now i'm not getting anything, but it might also be that i'm just struggling with the syntax. – brainbug Jun 11 '11 at 00:11
Hmm, maybe it's the feeds.entries.each block? Why are you doing it inside the on_complete? Shouldn't you do it after everything has been done? I will edit the question with code. Done. I left the hydra.queue the way I thought it was correct, if you aren't getting anything, I think you should do that the way you did before. – Thiago Silveira Jun 11 '11 at 00:14
That one isn't working, though a little bit success: if I "inspect" feeds.entries, I get a big result, see above. – brainbug Jun 11 '11 at 00:33
Well, I think I just fell in love with you. And it even makes sense, I get behind the code. Wow. :) Now, I'm actually having one other question which isn't so big so I'm just posting it here: Is there a way to get the last url from urls like tinyurl, bit.ly and co? I'm not that good in curl-stuff, so Is there a already-written function/class which might solve this? – brainbug Jun 11 '11 at 01:01
That's great! About your question, I think this question (and answers!) should give you a good start: http://stackoverflow.com/questions/902192/how-to-get-long-url-from-short-url – Thiago Silveira Jun 11 '11 at 01:09
And, I hate to say that, especially into my answers, but as you seem new to stackoverflow... Don't forget to mark the answer which helped you the most as accepted. More information about it: http://meta.stackexchange.com/questions/5234/how-does-accepting-an-answer-work – Thiago Silveira Jun 14 '11 at 03:54

score 0 · Answer 2 · answered Jun 10 '11 at 23:24

0

You're missing an end at the end of your block. If you indent your code consistently, you will see the hole:

require 'typhoeus'
require 'feedzirra'
feed_urls = ["feed1", "feed2"]    
hydra = Typhoeus::Hydra.new
feeds = {}
entry = {}
feed_urls.each do |feed|
  r = Typhoeus::Request.new(feed)
  r.on_complete do |response|
    feeds[r.url] = response.body
    feeds[r.url] = Feedzirra::Feed.parse(response.body)
    entry = feeds.entries.each do |entry|
      puts entry.title
    end
    hydra.queue r
  end

### You need an 'end' on this line to close the `each` ###

hydra.run

Welcome to Ruby and Stack Overflow! :)

answered Jun 10 '11 at 23:24

Phrogz

296,393
112
651
745

Hello, thanks :), unfortunatelly it doesn't solve the problem. (Actually I think, the end just isn't there, because I copied the code in two parts and forgot that single line) – brainbug Jun 10 '11 at 23:38
@brainbug You haven't actually stated what the error is, and your sample code cannot run on its own given the 'urls' you have supplied. Perhaps you would like to edit your question and provide more details on the exact problem (in behavior or error message) that you are having. – Phrogz Jun 10 '11 at 23:40
Oh, sorry, you're right. So, what happens: instead of showing every entry.title of every feed, it just shows all the titles of the first feed. I think now it's more clear on what I actually planned to do. thanks for the quick response. – brainbug Jun 10 '11 at 23:45
@brainbug Please edit your question with relevant details, don't provide them hidden in the comments to one particular (in this case wrong) answer. – Phrogz Jun 11 '11 at 01:18

Ruby, Feedzirra, typhoeus

2 Answers2