Multithreading in Rails: Circular dependency detected while autoloading constant

Question

I have a Rails app in which I have a Rake task that uses multithreading functions supplied by the concurrent-ruby gem.

From time to time I encounter Circular dependency detected while autoloading constant errors.

After Googling for a bit I found this to be related to using threading in combination with loading Rails constants.

I stumbled upon the following GitHub issues: https://github.com/ruby-concurrency/concurrent-ruby/issues/585 and https://github.com/rails/rails/issues/26847

As explained here you need to wrap any code that is called from a new thread in a Rails.application.reloader.wrap do or Rails.application.executor.wrap do block, which is what I did. However, this leads to deadlock.

The recommendation is then to use ActiveSupport::Dependencies.interlock.permit_concurrent_loads to wrap another blocking call on the main thread. However, I am unsure which code I should wrap with this.

Here's what I tried, however this still leads to a deadlock:

@beanstalk = Beaneater.new("#{ENV.fetch("HOST", "host")}:#{ENV.fetch("BEANSTALK_PORT", "11300")}")
tube_name = ENV.fetch("BEANSTALK_QUEUE_NAME", "queue")

pool = Concurrent::FixedThreadPool.new(Concurrent.processor_count * 2)

# Process jobs from tube, the body of this block gets executed on each message received
@beanstalk.jobs.register(tube_name) do |job|
    ActiveSupport::Dependencies.interlock.permit_concurrent_loads do
      @logger.info "Received job: #{job.id}"
      Concurrent::Future.execute(executor: pool) do
        Rails.application.reloader.wrap do
          # Stuff that references Rails constants etc
          process_beanstalk_message(job.body)
        end
      end
    end
end

@beanstalk.jobs.process!(reserve_timeout: 10)

Can anyone shed a light as to how I should solve this? The odd thing is I encounter this in production while other information on this topic seems to imply it should normally only occur in development.

In production I use the following settings:

config.eager_load = true

config.cache_classes = true.

Autoload paths for all environments are Rails default plus two specific folders ("models/validators" & "jobs/concerns").

eager_load_paths is not modified or set in any of my configs so must be equal to the Rails default.

I am using Rails 5 so enable_dependency_loading should equal to false in production.

Answer 1

You likely need to change your eager_load_paths to include the path to the classes or modules that are raising the errors. eager_load_paths is documented in the Rails Guides.

The problem you're running into is that Rails is not loading these constants when the app starts; it automatically loads them when they are called by some other piece of code. In a multithreaded Rails app, two threads may have a race condition when they try to load these constants.

Telling Rails to eagerly load these constants means they will be loaded once when the Rails app is started. It's not enough to say eager_load = true; you have to specify the paths to the class or module definitions as well. In the Rails application configuration, this is an Array under eager_load_paths. For example, to eager load ActiveJob classes:

config.eager_load_paths += ["#{config.root}/app/jobs"]

Or to load a custom module from lib/:

config.eager_load_paths += ["#{config.root}/lib/custom_module"]

Changing your eager load settings will affect the behavior of Rails. For example, in the Rails development environment, you're probably used to running rails server once, and every time you reload one of the endpoints it will reflect any changes to code you've made. That will not work with config.eager_load = true, because the classes are loaded once, at startup. Therefore, you will typically only change your eager_load settings for production.

Update

You can check your existing eager_load_paths from the rails console. For example, these are the default values for a new Rails 5 app. As you can see, it does not load app/**/*.rb; it loads the specific paths that Rails is expected to know about.

Rails.application.config.eager_load_paths
=> ["/app/assets",
 "/app/channels",
 "/app/controllers",
 "/app/controllers/concerns",
 "/app/helpers",
 "/app/jobs",
 "/app/mailers",
 "/app/models",
 "/app/models/concerns"]

Answer 2

I had this issue while trying out two gems that handles parallel processing;

1. pmap gem
2. parallel gem

For pmap I kept getting an error related to Celluloid::TaskTerminated and for parallel I was getting a Circular dependency detected while autoloading constant for when I ran it with more than 1 thread. I knew this issue was related to how my classes and modules were eager loading and race to be placed on a thread. I try enabling both of the configs to true config.cache_classes = true and config.eager_load = true in the development env and that did the trick for me.

Answer 3

In my gems (i.e., in plezi and iodine) I solve this with if statements, mostly.

You'll find code such as:

require 'uri' unless defined?(::URI)

or

begin
  require 'rack/handler' unless defined?(Rack::Handler)
  Rack::Handler::WEBrick = ::Iodine::Rack # Rack::Handler.get(:iodine)
rescue Exception

end

I used these snippets because of Circular dependency detected warnings and errors.

I don't know if this helps, but I thought you might want to try it.