2

I’m going to preface this by saying I’m a relative beginner, and while I may not understand a lot yet, I am more than willing to learn. I hope you can help with this one, as I have been trying to figure this out for a long time and am still stuck. I know this is a long one - I wouldn't throw it out here unless I had already put in many days of trying to figure it out myself first.

I am using Mechanize to run a script in my Rails app to fill out several webforms. It worked perfectly locally. However, I need to use delayed_job in order to run the time-intensive script in the background. For now, I am developing this locally (it was previously on Heroku) until the problem is fixed.

So I installed the delayed_job_active_record gem, and have been trying to finagle this into working. I DO know that although my form successfully takes in all of parameters successfully, nothing is getting inserted into the SQL table. It’s not picking up on the object’s ID. It's not recognizing the object's class (it sees it as NilClass), so it considers even regular Ruby methods (such as .strip) as unknown. (The console's output showing all of this is toward the bottom).

I’m clearly doing something wrong. It could very well be something painfully obvious - but I don't have the skills yet to figure that out. I’d be forever indebted to you if you can help me figure out what I’m doing wrong here.

First, here is my code:

MODEL:

require 'digest'
class Jumper < ActiveRecord::Base

after_save do |runscript|
  runscript.delay.scrape
end

validates :myuserid, :presence => true
validates :mypass, :presence => true
validates :mydate, :presence => true, :numericality => true
validates :mymonth, :presence => true, :numericality => true
validates :myyear, :presence => true, :numericality => true
validates :mylist, :presence => true

attr_accessor :myuserid, :mypass, :mydate, :mymonth, :myyear, :mylist

def scrape

 agent = Mechanize.new 

 page = agent.get('http://mywebsite.org/')

 myform = page.form_with(:name => 'signinForm')

 myuserid_field = myform.field_with(:name => "email")
 myuserid_field.value = myuserid  
 mypass_field = myform.field_with(:name => "password")
 mypass_field.value = mypass 

 page = agent.submit(myform, myform.buttons.first)

 mylistarray = mylist.strip.split(/[\s]+/)

 mylistfinal = mylistarray.map{|l| l[0..-5].sub(/(.*)\./,'\1').gsub('.','/')}.uniq

 mylistfinal.each do |doi|
  url ='http://mywebsite=' + doi + '&role=some_info#some_dates' 
  page = agent.get("http://mywebsite.org/submit")
  page = agent.get("#{url}") 

  entryform = page.form_with(:name => 'submit') 

  entryform.field_with(:name => 'month').options[("#{mymonth}").to_i].select
  entryform.field_with(:name => 'day').options[("#{mydate}").to_i].select
  entryform.field_with(:name => 'year').options[("#{myyear}").to_i].select

  page = agent.submit(entryform, entryform.button_with(:name => 'continue'))

    end
  end
end`

CONTROLLER:

def index
  @doilists = Doilist.all

respond_to do |format|
  format.html # index.html.erb
  format.json { render json: @doilists }
 end
end

def show
 @doilist = Doilist.find(params[:id])

 respond_to do |format|
  format.html # show.html.erb
  format.json { render json: @doilist }
 end
end

def new
  @doilist = Doilist.new

 respond_to do |format|
   format.html # new.html.erb
   format.json { render json: @doilist }
 end
end

def create
@jumper = Jumper.new(params[:jumper])

  if @jumper.save
    flash[:notice] = "Sucessfully submitted your information."
    redirect_to @jumper
  else
    render :action => 'new'
  end
 end`

GEMFILE:

source 'https://rubygems.org'

gem 'rails'

group :development do
  gem 'sqlite3', '1.3.5'
end

group :assets do
  gem 'sass-rails',   '~> 3.2.5'
  gem 'coffee-rails', '~> 3.2.2'
  gem 'uglifier', '>= 1.2.3'
end

gem 'jquery-rails', '2.0.2'
gem 'mechanize'

group :production do
  gem 'pg', '0.12.2'
end

gem 'delayed_job_active_record'
gem 'daemons'
gem 'thin'`

PROCFILE:

web: bundle exec rails server thin -p $PORT -e $RACK_ENV

worker:  bundle exec rake jobs:work`

script/delayed_job:

require File.expand_path(File.join(File.dirname(__FILE__), '..', 'config', 'environment'))
require 'delayed/command'
Delayed::Command.new(ARGV).daemonize

RAILS_ENV=production script/delayed_job start --exit-on-complete`

Here is the output when I run Foreman in my terminal:

11:33:30 web.1    | Started POST "/MyJumper" for 127.0.0.1 at 2013-05-02 11:33:29 -0400
11:33:30 web.1    |   Delayed::Backend::ActiveRecord::Job Load (1.2ms)  SELECT "delayed_jobs".* FROM "delayed_jobs" WHERE ((run_at <= '2013-05-02 15:33:30.437882' AND (locked_at IS NULL OR locked_at < '2013-05-02 11:33:30.438065') OR locked_by = 'delayed_job host:myroot pid:7081') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 5
1:33:30 web.1    | Processing by JumpersController#create as HTML
11:33:30 web.1    |   Parameters: {"utf8"=>"✓", "jumper"=>{"myuserid"=>"email@gmail.com", "mypass"=>"mypassword123”, "mylist"=>"listitem", "mymonth"=>"4", "mydate"=>"30", "myyear"=>"1"}, "commit"=>"Submit Your Entry"}
11:33:30 web.1    |    (0.1ms)  begin transaction
11:33:30 web.1    |   SQL (21.4ms)  INSERT INTO "jumpers" ("created_at", "mydate", "mylist", "mymonth", "mypass", "myuserid", "myyear", "updated_at") VALUES (?, ?, ?, ?, ?, ?, ?, ?)  [["created_at", Thu, 02 May 2013 15:33:30 UTC +00:00], ["mydate", nil], ["mylist", nil], ["mymonth", nil], ["mypass", nil], ["myuserid", nil], ["myyear", nil], ["updated_at", Thu, 02 May 2013 15:33:30 UTC +00:00]]
11:33:30 web.1    |   SQL (0.5ms)  INSERT INTO "delayed_jobs" ("attempts", "created_at", "failed_at", "handler", "last_error", "locked_at", "locked_by", "priority", "queue", "run_at", "updated_at") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)  [["attempts", 0], ["created_at", Thu, 02 May 2013 15:33:30 UTC +00:00], ["failed_at", nil], ["handler", "--- !ruby/object:Delayed::PerformableMethod\nobject: !ruby/ActiveRecord:Jumper\n  attributes:\n    id: 68\n    mylist: \n    created_at: 2013-05-02 15:33:30.920857000 Z\n    updated_at: 2013-05-02 
15:33:30.920857000 Z\n    myuserid: \n    mypass: \n    mymonth: \n    mydate: \n    myyear: \nmethod_name: :scrape\nargs:\n- :jumper\n"], ["last_error", nil], ["locked_at", nil], ["locked_by", nil], ["priority", 0], ["queue", nil], ["run_at", Thu, 02 May 2013 15:33:30 UTC +00:00], ["updated_at", Thu, 02 May 2013 15:33:30 UTC +00:00]]
11:33:30 web.1    |    (10.7ms)  commit transaction
11:33:30 web.1    | Redirected to `http://mylocalhost:5000/MyJumper/68`
11:33:30 web.1    | Completed 302 Found in 74ms (ActiveRecord: 33.1ms)
11:33:31 web.1    | Started GET "/show/68" for 127.0.0.1 at 2013-05-02 11:33:30 -0400
11:33:31 web.1    | Processing by JumpersController#show as HTML
11:33:31 web.1    |   Parameters: {"id"=>"68"}
11:33:31 web.1    |   Jumper Load (0.3ms)  SELECT "jumpers".* FROM "jumpers" WHERE "jumpers"."id" = ? LIMIT 1  [["id", "68"]]
11:33:34 worker.1 | [Worker(host:myroot pid:10470)] Jumper#scrape failed with NoMethodError: undefined method `strip' for nil:NilClass - 0 failed attempts
11:33:34 worker.1 | [Worker(host:myroot pid:10470)] 1 jobs processed at 0.8520 j/s, 1 failed ...
CodeBiker
  • 2,985
  • 2
  • 33
  • 36
  • It looks like your Jumper object gets inserted into the database with all nil values. I'm not sure how this isn't triggering validation errors since you have presence validators set on the object. – Ryan May 04 '13 at 00:02

2 Answers2

0

It looks like you've got some undefined's. For example, where's mymonth?

I want to give you a tip for simplifying your code while I'm here:

instead of:

myuserid_field = myform.field_with(:name => "MSTRServlet.email")
myuserid_field.value = myuserid

Try just:

myform['MSTRServlet.email'] = myuserid

That's not likely to solve your problem, but maybe it will get you closer to something simple enough to solve.

And finally, I know that Heroku is a current media darling, but you should consider that it might not be the best fit for your particular project.

pguardiario
  • 53,827
  • 19
  • 119
  • 159
  • Thank you for your response. The format you suggested for `myform` may work, but it's not in the Mechanize documentation. The script in the `scrape` method works perfectly, although considering my level of expertise, there is likely room for improvement. As for Heroku, I should have clarified that I plan to deploy this on Heroku, but I am developing it locally until this problem is fixed (I'll make a note of this above). The problem, from what I can tell, is that the background workers are not picking up on `@jumper`'s ID, and so the object's attributes are seen as `nil`. – CodeBiker May 03 '13 at 14:03
  • Also to clarify: `mymonth`is an attribute, not a method. I just added the full model and controller code above. – CodeBiker May 03 '13 at 14:13
0

Thanks for your suggestions. I ended up figuring out how to solve this on my own. The issue apparently was that Delayed_Job has significant problems with attr_accessor in Rails 3.

There is more information on this here: delayed_job: attribute accessor values are not stored in delayed_jobs table in rails 3 after upgrade from rails 2

And here: YAML, delayed_job : Psych vs Syck. How to make pysch read attr_accessors for a ruby object

And a workaround for this problem is explained here: https://gist.github.com/jrafanie/3011499

After simply removing the attr_accessor part of my model, everything works perfectly. What a relief!

Also: I removed the after_save code from my model and simply added @jumper.delay.scrape in my controller, right after my controller saved my object. This may not be necessary, but it's more consistent with the documentation:

if @jumper.save
    @jumper.delay.scrape 
    flash[:notice] = "Currently sending your info"
    redirect_to start_index_path

As I move towards production, I will take a further examination of the workaround and see the best way to my project so I can have attr_accessor in my model. But for now, just glad to have this working!

Community
  • 1
  • 1
CodeBiker
  • 2,985
  • 2
  • 33
  • 36