2

We're writing a Rails app that we want to be able to talk to any of several external data repositories via a uniform adapter interface, and we'd like to be able to add more later just by writing new implementations of that adapter interface (cf. ActiveRecord adapters).

Any one installation of the app will only need one adapter running, and we don't want to have to rev the code or even the Gemfile whenever we introduce a new adapter. Assuming we write each adapter as a standalone gem, what's the proper way to incorporate just one adapter gem at runtime, based on configuration?

David Moles
  • 48,006
  • 27
  • 136
  • 235
  • I had a similar question that come-up. My scenario is different though and goes like this. We have an older version of a library (in a gem) that we want to upgrade. However, this gem participates in a critical part of our externally consumed API and we cannot update it at once and want to migrate clients to the new gem one by one. We don't want to stand up an entirely separate version of the API for this either (overkill). Are there practical strategies for this? @david-moles – Shyam Habarakada May 23 '18 at 01:36
  • @ShyamHabarakada Personally, I would stand up two versions of the API (with something in front of them to direct clients to one or the other). I absolutely would not trust that I could load two versions of one gem into the same Ruby application and safely switch between them for any given request. But this seems like a question for [softwareengineering.stackexchange.com](https://softwareengineering.stackexchange.com). – David Moles May 23 '18 at 21:44
  • Yes @david-moles you are right that we cannot trust this. Thanks – Shyam Habarakada Jul 31 '18 at 15:27

3 Answers3

2

Normally in Rails, you can only access gems that are listed in the Gemfile; Bundler enforces this. However that doesn't stop you from loading gems manually.

If your adapter gems are installed in a well-known location, then loading an adapter gem could be as simple as this:

$LOAD_PATH << "/path/to/adapters/my_adapter/lib"
require "my_adapter"

Where "my_adapter" is something you discover at runtime via configuration.

Matt Brictson
  • 10,904
  • 1
  • 38
  • 43
2

Put your configuration in the environment, as you're probably already doing.

For example, I use the dotenv gem.

Set an environment variable to your adapter gem name.

The result is an ENV var that Rails can access, such as:

ENV["adapter"] #=> "my_custom_gem_name"

Use the environment to pick the adapter, e.g. in your Rails config:

require ENV['adapter']

This presumes your gem name is the same as the require name. If your gem name is different, use the require name.

When you deploy a new adapter, you can put it anywhere on the Rails load path. Or you can adjust the Rails load path as you like e.g. adding a path to $LOAD_PATH.

An example of how to adjust the load path:

# /config/application.rb
module MyApp
  class Application < Rails::Application
  $LOAD_PATH << "/path/to/custom/gem" 
  require("my_custom_gem_name")
  …

For example, put the gem code in ./lib, or install the gem into ./vendor or systemwide, using any tool you like such as scp or rsync or ansible, thus bypassing the typical bundle command.

You won't need to update the Gemfile.

joelparkerhenderson
  • 34,808
  • 19
  • 98
  • 119
  • Bundler, which Rails uses, restricts the load path to _not_ include system-installed gems, unless they are mentioned in the Gemfile. If you put it into local app `./lib/` or `./vendor`, then you're modifying source code (I think?) and might as well just add it to the Gemfile as normal, what are you gaining? I guess you could put the gem somewhere custom system-wide (not normal `gem install`), and custom add it to your apps load path? Or wait, you're going to manually scp it into your app's dir, but not commit it to your vcs? I guess you _can_.... – jrochkind Mar 05 '15 at 12:39
  • In my experience bundler sets a load path, but does not retrict it thereafter; i.e. I can edit the load path as I like within the app. What he's gaining is not needing to rev the Gemfile. He could use a system `gem install` (or not). He could choose to commit the code to a VCS (or not). My team does something similar using torrents to distribute gems, rather than bundler, because torrents are so much faster. – joelparkerhenderson Mar 05 '15 at 14:34
  • True, it doesn't restrict it after, it just takes the usual system gem location off the load path. I guess you could put it back... or maybe less confusingly install your gems somewhere other than the system location and point to them there (no idea if this is less confusing or not actually, heh) – jrochkind Mar 05 '15 at 15:03
1

I don't think there's a good way to do quite exactly what you ask. Part of the bundler system (that's what uses Gemfiles) is that only gems mentioned in your gemfile (and their dependencies) are available to your app, it is isolated to only them. If this sounds like a bad thing, we could have a long conversation about the pre-bundler dependency management problems that this system has successfully solved.

Option 1 -- what are you worried about anyway? There is probably no downside to including all the possible gems in your Gemfile, even if a given installation will only use one. If using Rails, you will want to make sure these gems are not require'd on launch. (If not using Rails, this might not be neccesary, as other environments don't necessarily ask bundler to require all gems on boot (which is wise), but won't hurt).

So all the gems will get installed in every installation, yes, but only the one you want to use will be loaded, when you issue the require at runtime -- you'll still want to do this on application load, perhaps in response to an ENV variable, to avoid any concurrency or load time weirdness. Anyway, the unused gems will just be sitting there on disk un-loaded. How big a downside is having them installed but not used? How much are you willing to increase the confusingness of your setup to get rid of this downside?

Option 1 is what I'd do.

Option 2 -- separate Gemfiles. Another option is preparing a separate Gemfile for each type of setup. Gemfiles are just ruby code, so you could have one 'base' gemfile including common gems, and then a separate Gemfile for each type of setup, which uses ruby to load/include the base gemfile and then adds the setup-specific gems. You'd give each Gemfile a separate name, Gemfile_adapter1 or whatever.

You're going to have to commit something to your source when you add support for a new adapter type, aren't you? Where does this adapter come from? I don't understand how you could do it without revising any code. Anyway, adding a new Gemfile for that adapter type when you add a new adapter doesn't seem like a huge barrier, but I dunno.

You can launch Rails specifying which of these gemfiles to use with the BUNDLE_GEMFILE env variable: BUNDLE_GEMFILE=./Gemfile_one rails server. And in every other command you do that will use the Gemfile. BUNDLE_GEMFILE=./SOME_GEMFILE bundle install. BUNDLE_GEMFILE=./SOME_GEMFILE RAILS_ENV=production bundle exec rake assets:precompile. If you use capistrano that's doing some of these things for you, figure out how to make sure cap uses the right BUNDLE_GEMFILE when it executes it. Figure out how to make your app server do that when it launches your app. Etc.

This will work out fine -- but is going to end up being a pain to keep track of and make sure it's working right in your entire devops stack.

I suppose you could even generate the Gemfiles at install time, instead of having them in your source repo, for even more confusing situation and another thing that can go wrong and be confusing to debug! (I wouldn't).

I'd consider this Option 2, but would prefer option 1 unless there were really good reasons it wasn't going to work.

Option 3 -- Don't use Bundler. You could abandon bundler and gemfiles entirely. Maybe, if you can convince Rails to do this somehow. It might be hard to convince Rails to do this.

If you could, you are in a situation where you have to install all your gems on your deploy system by hand, making sure they are the right versions and figuring out what versions are compatible with what other versions, and that no more recent versions you don't want are installed.

Then, at runtime, you just need to require all the gems you want, and the app will get the most recent version of that gem installed on the system.

I would never, ever, ever, do this. I don't know if you can get Rails to do it, but even if you could, I remember the dependency hell from before Bundler existed, and would never want to go back.

Community
  • 1
  • 1
jrochkind
  • 22,799
  • 12
  • 59
  • 74
  • _You're going to have to commit something to your source when you add support for a new adapter type, aren't you?_ -- I shouldn't have to, because I'm not "adding support for a new adapter type" to the application, I'm just creating (in a separate gem) another class that responds to the methods the application expects for an adapter. – David Moles Mar 05 '15 at 03:40
  • How does your app know to use the gem? The name of it, or where to load it from, which adapter to choose? Okay, figure out some way to do that using just ENV variables to tell it the name of the gem it should load, and conventions for what the class name inside that gem is, and all constructed dynamically? Me, I would not want to go back to the pre-Bundler days where the dependency tree was not built, checked for consistency, and frozen in a Gemfile.lock. If you can figure out a way to, maybe based on Matt's answer... I predict you'll independently re-discover why bundler is a good idea. – jrochkind Mar 05 '15 at 12:29
  • The class of problem that arose in pre-Bundler days, for example, is when your new gem uses dependency Widgetizer but only version 1.x and won't work with Wigetizer 2.x, but your app (or some existing dependency of it) needs Wigetizer 2.x.... and this is revealed by a mysterious exception that takes you an hour to figure out that's what's going on. And you have trouble controlling which version of Widgetizer actually gets loaded anyway, it depends on what gems are installed on the system-wide level. But if you want to try that path, Matt Brictson's solution seems like a direction. – jrochkind Mar 05 '15 at 12:36
  • So how does ActiveRecord do it? You don't need to edit a gemfile and re-bundle your application (or somebody else's application that you're just running) to switch databases, you just edit your `database.yml`. If you're using something other than Postgres/MySQL/SQLite, say Oracle or DB2 for some reason, you install some extra gems. It's not rocket surgery. Or am I missing something? – David Moles Mar 05 '15 at 19:55
  • In a Rails app, you _do_ add the gems for the particular database connectors you need to your Gemfile and rebundle. If you switch from mysql to postgres, you usually remove the mysql gem from your gemfile and add the postgres gem. The actual ActiveRecord adapters are _all_ [included with ActiveRecord](https://github.com/rails/rails/tree/master/activerecord/lib/active_record/connection_adapters), you get all of them AR supports out of the box included. If you want something else, you might install an extra gem--AND add it to your Gemfile and rebundle. It's not going outside the Gemfile at all. – jrochkind Mar 05 '15 at 22:26
  • If you want to add gem dependencies without editing the Gemfile, that's not what Rails/ActiveRecord does. You actually DO need to edit a gemfile and re-bundle (bundle install) your app to add or change dependencies, if you need a gem that wasn't already in your Gemfile (or a dependency in the gemspec of a gem in your Gemfile). (I do really like 'rocket surgery' as a phrase though, heh) – jrochkind Mar 06 '15 at 01:42