1
  1. Does loading multiple Models in sidekiq worker can cause memory leak? Does it get garbage collected?

For example:

class Worker
  include Sidekiq::Worker

  def perform
    Model.find_each do |item|

    end
  end
end
  1. Does using ActiveRecord::Base.connection inside worker can cause problems? Or this connection automatically closes?
Zilvinas
  • 11
  • 3
  • Worth a read (from author of Sidekiq)~: https://www.mikeperham.com/2015/10/14/optimizing-sidekiq/ .. also, possible duplicate of http://stackoverflow.com/questions/18978396/sidekiq-not-deallocating-memory-after-workers-have-finished – Damien Roche May 12 '16 at 12:59
  • check here https://stackoverflow.com/a/34462164/3011280 – Oshan Wisumperuma Mar 13 '19 at 02:42

1 Answers1

0

I think you are running into a problem that I also had with a "worker" - the actual problem was the code, not Sidekiq in any way, shape or form.

In my problematic code, I thoughtlessly just loaded up a boatload of models with a big, fat, greedy query (hundreds of thousands of instances).

I fixed my worker/code quite simply. For my instance, I transitioned my DB call from all to use find_in_batches with a lower number of objects pulled for the batch.

Model.find_in_batches(100) do |record| 
# ... I like find_in_batches better than find_each because you can use lower numbers for the batch size
# ... other programming stuff

As soon as I did this, a job that would bring down Sidekiq after a while (running out of memory on the box) has run with find_in_batches for 5 months without me even having to restart Sidekiq ... Ok, I may have restarted Sidekiq some in the last 5 months when I've deployed or done maintenance :), but not because of the worker!

craig.kaminsky
  • 5,588
  • 28
  • 31
  • 2
    The OP already uses `find_each` what does basically the same than `find_in_batches(1000) { |batch| batch.each ...` – spickermann May 12 '16 at 13:25
  • Sorry, yes. I will edit my post ... I like find_in_batches better because you can control the batch size (versus find_each, which grabs a default 1000). – craig.kaminsky May 12 '16 at 13:30