0

I'm looking to dynamically generate new scheduler tasks because the number of tasks I have depends on the size of the database. The larger the database gets, the more tasks I will need, and I'm wondering if there is any way for me to write my Rails app to do this automatically rather than having to manually go in and add more tasks each time. Any ideas?

My rufus scheduler .rb file has blocks like 

scheduler.every("7d") do 
    size = Rails.cache.fetch("size")
    get_data_portion(0, size)
end

scheduler.every("7d") do 
    size = Rails.cache.fetch("size")
    get_data_portion(1, size)
end

where size determines how big of a section of my database I am querying, and the first argument (0 and 1) determines exactly what section of the database it is querying. Currently, the size is determined after seeing how large the database is currently, breaking it into even chunks, and then having that many scheduler events to correspond with each chunk of data.

The next time the scheduler is run, however, the database may have increased so much that the get_data_portion method would be querying too much data if I stayed with the same amount of scheduler events. Because of this, I was hoping to be able to create more scheduler events, but without having to manually do it each time. Does this make my inquiry clearer?

ccj
  • 11
  • 1
  • 1
    I flagged this as off topic, because you are, essentially, asking for others to recommend tools, libraries, or other outside resources. Answers would be link-only, and not very fruitful. – onebree Aug 10 '15 at 18:46
  • With all due respect, I don't see how this question differs that much from a question such as this one. http://stackoverflow.com/questions/2777802/how-to-write-to-file-in-ruby I'm pretty new to Rails and generally just don't know if this would be possible, even after reading through the rufus-scheduler documentation, which is why I'm asking. – ccj Aug 10 '15 at 21:25
  • 1
    I have not read every post on SO. I try to help out by reviewing (and possibly answering) questions newly posted for ruby/rails. I do not know if the off-site resource flag was available back in 2010. You do have the power to edit your post, possibly asking for more concrete information. Do you have any code to share to help us out? – onebree Aug 10 '15 at 21:29
  • I've updated the post. Hopefully this is asking for more concrete information? – ccj Aug 10 '15 at 21:43

1 Answers1

0

It's not "more scheduler events" but "less interval between scheduler events". It takes some inspiration from https://github.com/jmettraux/rufus-scheduler#every-jobs-and-changing-the-next_time-in-flight :

BIG_SIZE = 1000

(1..2).each do |i|

  scheduler.every '7d' do |job|

    size = Rails.cache.fetch('size')
    get_data_portion(i, size)

    job.next_time = Time.now + 2 * 24 * 60 * 60 if size >= BIG_SIZE
  end
end
jmettraux
  • 3,511
  • 3
  • 31
  • 30
  • With this solution, the size of data for each get_data_portion method call would stay the same though right? I guess my question was not looking for more scheduler events but for more jobs so that each method call will handle small data sizes in the event that the database gets too large. I've been having memory problems on heroku when running the methods continuously in a loop, which is why having separate job processes is optimal. – ccj Aug 11 '15 at 19:18
  • If Rails.catch.fetch('size') always returns the same value then, yes, the size of data would stay the same. That put aside, what prevents you from calling with a higher frequency? Roughly, calling every day rather than every week would lead to data portion that are a seven times smaller each session. "job" == "Heroku job process" for you? That would have been a nice thing to mention in your question. If you already know what is optimal, then go on, implement it. – jmettraux Aug 11 '15 at 23:05