rails migration runs on local Postgresql but fails on prod

Question

In short, dropping, creating, and running migrate on the local Postgres instance will work any number of times for creating a working database for my app, but the same technique on Heroku's prod always produces:

heroku run rake db:migrate
Running `rake db:migrate` attached to terminal... up, run.9674
PG::UndefinedTable: ERROR:  relation "mytable" does not exist
LINE 5:                WHERE a.attrelid = '"mytable"'::regclass
                                          ^
:               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
                     pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod
                FROM pg_attribute a LEFT JOIN pg_attrdef d
                  ON a.attrelid = d.adrelid AND a.attnum = d.adnum
               WHERE a.attrelid = '"mytable"'::regclass
                 AND a.attnum > 0 AND NOT a.attisdropped
               ORDER BY a.attnum

rake aborted!
PG::UndefinedTable: ERROR:  relation "mytable" does not exist
LINE 5:                WHERE a.attrelid = '"mytable"'::regclass
                                          ^
:               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
                     pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod
                FROM pg_attribute a LEFT JOIN pg_attrdef d
                  ON a.attrelid = d.adrelid AND a.attnum = d.adnum
               WHERE a.attrelid = '"mytable"'::regclass
                 AND a.attnum > 0 AND NOT a.attisdropped
               ORDER BY a.attnum

EDIT: see bottom of question for clue on controller getting accessed during migrate.

This is what works locally:

rake db:drop
rake db:create
rake db:migrate

I have

confirmed that Heroku's db is blank before migration by doing heroku pg:psql and then \dt to verify 0 tables exist.
tried heroku pg:reset DATABASE before the migrate
confirmed both local and prod Postgres is version 9.2.4
tried renaming the "mytable" migration file in db/migrate to have the earliest timestamp so as to get run first

This is a pretty simple app so it's very frustrating that something basic like creating a db from scratch keeps failing. Any ideas?

"mytable" migration:

class CreateMytable < ActiveRecord::Migration
  def change
    create_table :mytable do |t|
      t.string :codes
      t.string :name

      t.timestamps
    end
  end
end

likely referecing migration:

class CreateTable2 < ActiveRecord::Migration
  def change
    create_table :table2 do |t|
      t.references :a, index: true
      t.references :b, index: true
      t.string :c
      t.string :d

      t.timestamps
    end
  end
end

EDIT: noticed a clue when running db:migrate with --trace. The top of the stack trace below error shows a controller error?? Why is a controller getting involved with a migration?

/app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.0/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `exec'
/app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.0/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `exec_no_cache'
/app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.0/lib/active_record/connection_adapters/postgresql/database_statements.rb:138:in `block in exec_query'
/app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.0/lib/active_record/connection_adapters/abstract_adapter.rb:425:in `block in log'
/app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.0/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.0/lib/active_record/connection_adapters/abstract_adapter.rb:420:in `log'
/app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.0/lib/active_record/connection_adapters/postgresql/database_statements.rb:137:in `exec_query'
/app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.0/lib/active_record/connection_adapters/postgresql_adapter.rb:915:in `column_definitions'
/app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.0/lib/active_record/connection_adapters/postgresql/schema_statements.rb:174:in `columns'
/app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.0/lib/active_record/connection_adapters/schema_cache.rb:114:in `block in prepare_default_proc'
/app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.0/lib/active_record/connection_adapters/schema_cache.rb:56:in `yield'
/app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.0/lib/active_record/connection_adapters/schema_cache.rb:56:in `columns'
/app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.0/lib/active_record/model_schema.rb:208:in `columns'
/app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.0/lib/active_record/model_schema.rb:242:in `column_defaults'
/app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.0/lib/active_record/locking/optimistic.rb:169:in `column_defaults'
/app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.0/lib/active_record/core.rb:181:in `initialize'
/app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.0/lib/active_record/inheritance.rb:27:in `new'
/app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.0/lib/active_record/inheritance.rb:27:in `new'
/app/app/controllers/home_controller.rb:7:in `<class:HomeController>'

The line 7 in question contains a call to Mytable.new(... . How is it controller code is coming into scope during a db:migrate?

Can you post this migration and the one that includes "mytable"? — steakchaser, Sep 07 '13 at 23:22
Added "mytable" migration, but I don't know how to determine what other migration is running when the error occurs. — Josh Diehl, Sep 07 '13 at 23:31
It's whatever migration references mytable through t.references, add_reference or remove_reference. Do a quick search across your migration files. — steakchaser, Sep 07 '13 at 23:38
I'm actually not sure... I would try some of the approaches listed in here (including removing your app entirely). http://stackoverflow.com/questions/5450930/heroku-postgres-error-pgerror-error-relation-organizations-does-not-exist — steakchaser, Sep 07 '13 at 23:51
Can you run the migrate with --trace? This should give you the migration that's running when it fails. — Jason N, Sep 08 '13 at 00:07
@steakchaser I have tried deleting the app from heroku twice, second time with an app rename. — Josh Diehl, Sep 08 '13 at 00:09
@abcdef123 trace yielded a weird clue, that somehow rails controller code is getting called or referenced. See question edit. — Josh Diehl, Sep 08 '13 at 00:17
Josh, were these migrations generated with rails generate? It seems strange to me that mytable is singular, table names are by convention plural in rails. It's possible there's some assumptions being made related to this causing the failure. — Jason N, Sep 08 '13 at 00:26
You can safely ignore the singular/plural references, I've changed the actual class/tables names to the generic "mytable" etc. since the app is not open source. Hopefully shouldn't impede reading through them. — Josh Diehl, Sep 08 '13 at 00:28
that make sense now thanks :) I just came across this relating to the controller code being loaded during a migration. You may be running into the same bug, https://github.com/activescaffold/active_scaffold/issues/131. A quick test would be to add config.cache_classes = false to production.rb. — Jason N, Sep 08 '13 at 00:32
OK, I've identified the offending controller code and modified it to work. I'll post as an answer, I'd be interested to hear if this is something I was doing wrong, or a bug in rake, rails, or Heroku's config. — Josh Diehl, Sep 08 '13 at 01:47

score 1 · Answer 1 · answered Sep 08 '13 at 01:59

This was somehow being caused by setting a class variable in a Rails controller that referenced the "mytable" class. So basically this:

 class HomeController < ApplicationController
  @@data = {MyTable.new(...

Moving this inside of a controller method and making it a non-class variable fixed the problem( I didn't isolate whether it was the class var aspect or location right below class def that was the cause).

I still don't understand why controller code was a factor in a db migrate. I'd be interested to hear if this is something I was doing wrong, or a bug in rake, rails, or something else. Only happened in Heroku production, Rails 4.0.0

Josh, running a migration loads all the app code and because you had this this in a class variable it was loaded when the app code was initially loaded (because the class itself was run through the interpreter), causing an error when the db table didn't exist. You could probably comment this line out to run the migrations and then uncomment it if you wanted to keep it. Either way, interesting failure! — Jason N, Sep 08 '13 at 03:33

rails migration runs on local Postgresql but fails on prod

1 Answers1