Rails 4: How can I prevent loading all database records before a call to Model.all

Question

I've encountered a performance issue where Processing.all gets called prior to a filter, which in turn runs a second query on the Processing model. Processing has a good number of records, and loading them all into memory, only to run a second query on them is causing a spike in RAM which I'd like to fix.

The line in the controller is:

@filter = ProcessingFilter.new(base_collection: Processing.all, options: params[:filter])

As you can see, Processing.all is being passed in here as the base_collection param.

The ProcessingFilter then runs:

class ProcessingFilter
  def initialize(base_collection:, options: {})
    @base_collection = base_collection
    @options = options
  end

  def collection
    @collection ||= begin
      scope = @base_collection
      if condition1
        scope = scope.joins(:another_entry).where(another_entry: {customer_id: customer_id})
      end
      if condition2
        scope = scope.where('created_at >= ?', created_at_start)
      end
      if condition3
        scope = scope.where('created_at <= ?', created_at_end)
      end
      if condition4
        scope = scope.where(number: processing_number)
      end
      scope
    end
  end
end

This filter chains together the various if conditions creating a single ActiveRecord query, which is fine.

The problem is that I cannot do away with this filter as it sets some instance variables that are being used elsewhere. I'm trying to think of a clever way to not have the Processing.all query run the first time, but instead have it chain together the other options despite being in a separate class. Is this possible?

Thanks in advance!

Jay-Ar Polidario · Answer 1 · 2018-01-10T09:43:44.987

DISCLAIMER: This is not an ANSWER yet per se, but because it's too long and needs to type a code then:

Processing.all doesn't yet load the records to memory as the records are "lazily loaded" as it only returns an ActiveRecord_Relation object. Only after you use an Array method on it such as each, first, last, map, or [], does it only start to actually fetch the records from the database into memory.

To demonstrate:

processings = Processing.all

puts processings.class
# => Processing::ActiveRecord_Relation

puts processings.first.class
#   Processing Load (2.9ms)  SELECT  "processings".* FROM "processings"  ORDER BY "processings"."id" ASC LIMIT 1
# => Processing

Now that we know that .all does not eager load the records immediately into the memory, then we need to find out why your code is still loading the records immediately into the memory when you call @filter = ProcessingFilter.new(base_collection: Processing.all, options: params[:filter])

Limited to the code that you show, I cannot see anything inside your ProcessingFilter class that would trigger the loading of records into memory (no Array methods called that would load them into memory); they're all just ActiveRecord_Relation objects. Therefore, my current guess is that somewhere between two filters, you are calling an Array method:

@filter = ProcessingFilter.new(base_collection: Processing.all, options: params[:filter])

# An Array method is called:
@filter.collection.first

If you are doing this in rails console, then you'd need to append ; nil instead to prevent "processing" the value per-line called as it would eager load the records immediately:

@filter = ProcessingFilter.new(base_collection: Processing.all, options: params[:filter]); nil

@filter.collection.first; nil

loybert · Answer 2 · 2018-01-03T23:27:40.397

1

in case you aren't working with filtering default_scopes (or want to ignore the default filtering anyway) Processing.unscoped would do the trick.

Which version of rails are you currently using by the way?

Additional link about the deprecated .scoped method: With Rails 4, Model.scoped is deprecated but Model.all can't replace it

edited Jan 03 '18 at 23:27

answered Jan 03 '18 at 23:08

loybert

416
1
3
12

Thanks for this. I'm using Rails `4.2.6` for this project. I'm not trying to ignore the default filtering, just trying to make a SQL query with several if statements in another model, and I'd like to prevent loading all the DB entries prior to that logic. – DaniG2k Jan 04 '18 at 09:08
then `where(nil)` is the way to go. Works also for 5.1.4 – loybert Jan 04 '18 at 09:38

score 0 · Answer 3 · answered Jan 05 '18 at 01:27

I would consider extracting each of these conditionals into their own scope methods and then deprecating this ProcessingFilter class. It seem like decorator that is not well designed.

You can use the base_collection value to determine the model being called, and then proxy the call to ProcessingFilter to the appropriate scope in the initialization. Should save yourself some trouble and just call base_collection scope. I suspect you're getting duplicate query calls because of that usage.

Rails 4: How can I prevent loading all database records before a call to Model.all

3 Answers3

Linked