1

I have this error while loading the data from files in to database. Please help me out in getting this right.

here is the method :

clusters.each do |cluster|
        cluster_path = cluster.path
        root = current_root + cluster.name+'/'
        Log.info "<br/> --------- Starting data popuation for #{cluster_path} ---------------"

         population_time[:bp_data] = populate_bp_data(root+fileName[:bp_data], cluster_path, @dataset.id)
         population_time[:m_lint] = populate_m_lint(root+fileName[:m_lint], cluster_path, @dataset.id)
         population_time[:single_use] = populate_single_use(root+fileName[:single_use], cluster_path, @dataset.id)
         population_time[:test_points] = populate_test_points(root+fileName[:test_points], cluster_path, @dataset.id)
         population_time[:users] = populate_user(root+fileName[:users], cluster_path, @dataset.id)
         population_time[:test_times] = populate_test_times(root+fileName[:test_times], cluster_path, @dataset.id)
         population_time[:qeinbat] = populate_qeinbat_data(root+fileName[:qeinbat], cluster_path, @dataset.id)
         population_time[:clearall] = populate_clearall_data(root+fileName[:clearall], cluster_path, @dataset.id)
         population_time[:closeall] = populate_closeall_data(root+fileName[:closeall], cluster_path, @dataset.id)
         population_time[:clearclass] = populate_clearclass_data(root+fileName[:clearclass], cluster_path, @dataset.id)
         population_time[:clearmax] = populate_clearmax_data(root+fileName[:clearmax], cluster_path, @dataset.id)
         population_time[:pause] = populate_pause_data(root+fileName[:pause], cluster_path, @dataset.id)
         population_time[:num_of_test] = populate_num_of_test_data(root+fileName[:num_of_test], cluster_path, @dataset.id)
         population_time[:test_description] = populate_test_description(root+fileName[:test_description], cluster_path, @dataset.id)
         population_time[:component] = populate_components(root+fileName[:component], cluster.id, cluster_path, @dataset.id)


         population_time.each do |key, value|
         total_time += value
        end
     end

each populate method called here fetches the file and loops through its contents and feeds data in to the database using the following stepps :

CSV.foreach(fileName) do |line|
    completePath = line[0]                                                
    num_of_bps = line[1]

    completePath = cluster_path+ '/' + completePath
    inode = FileOrFolder.find_by_fullpath(completePath, :select=>"id") 

    metric_instance = MetricInstance.find(:first, :conditions=>["file_or_folder_id = ? AND dataset_id = ?", inode.id, dataset_id])
    add_entry(metric_instance.id, num_of_bps, num_of_bp_tests) 
end



def self.add_entry(metaid, num_of_bps, num_of_bp_tests)
    entry = Bp.new
    entry.metric_instance_id = metaid
    entry.num_of_bps = num_of_bps
    entry.num_of_bp_tests = num_of_bp_tests
    entry.save
    return entry
end
Vinay
  • 237
  • 2
  • 8
  • 17
  • What is the exact error you get? And on which line? – Ermin Dedovic Jun 17 '13 at 19:05
  • exact error : "!! Unexpected error while processing request: failed to allocate memory" ................. I have 3 clusters and each cluster has a number of files which am loading in to database by populate method. for loop goes fine for the first 2 clusters and exactly fails while loading the third cluster. when I restart by loading with the third cluster alone after this error . am able to load it fine. – Vinay Jun 17 '13 at 20:11
  • Have you tried with a smaller cluster? Have you tried my answer from http://stackoverflow.com/questions/17109031/unexpected-error-while-processing-request-failed-to-allocate-memory ? – Ermin Dedovic Jun 17 '13 at 22:03
  • yeah it runs fine with the small clusters... i have triggered the script last night using the method you suggested .... so far its good .... its still running .... guess its very slow. lets c . But do we have any other option apart from this ? – Vinay Jun 18 '13 at 13:12
  • Same error again with the method you suggested :( /home/vinay/.rvm/gems/ruby-1.9.3-p392/gems/railties-3.2.13/lib/rails/commands/server.rb:70:in `start' /home/vinay/.rvm/gems/ruby-1.9.3-p392/gems/railties-3.2.13/lib/rails/commands.rb:55:in `block in ' /home/vsadhu/.rvm/gems/ruby-1.9.3-p392/gems/railties-3.2.13/lib/rails/commands.rb:50:in `tap' /home/vsadhu/.rvm/gems/ruby-1.9.3-p392/gems/railties-3.2.13/lib/rails/commands.rb:50:in `' script/rails:6:in `require' script/rails:6:in `
    ' !! Unexpected error while processing request: failed to allocate memory
     – Vinay Jun 18 '13 at 13:30
  • Can you try increasing your swap? Do you have 32-bit or 64-bit Ruby? – Ermin Dedovic Jun 18 '13 at 14:22
  • I have 64-bit Ruby .... how do we increase swap ... sorry i dint get you .... am new to ruby. – Vinay Jun 18 '13 at 14:55
  • Not Ruby, your Linux swap partition size... – Ermin Dedovic Jun 18 '13 at 16:02
  • its 4gb now ..... and script should work on the production too ...so I am not sure if I can alter the memory usage there ....so that this scripts wrks same there. ...... is there any way to clear the GC ..in order to free up the memory ?? like i would like to free up the memory for each iteration of the cluster ... – Vinay Jun 18 '13 at 16:20
  • You can call GC.start to start garbage collection. How big are your files? – Ermin Dedovic Jun 18 '13 at 16:32
  • 10-15mb on an average ...so should i put GC.start in the loop cluster.each at the end after all the steps ?? – Vinay Jun 18 '13 at 17:09
  • or should i make population_time as nil at the end in loop and then do GC.start ??? – Vinay Jun 18 '13 at 17:15
  • You can do that, but i think it isn't that important because one iteration shouldn't be so important, files aren't so large. – Ermin Dedovic Jun 18 '13 at 17:18
  • ok so you want me to make population_time to nil n then do GC.start ? files arent large ....but am not getting what else can be done here ...what am thinking is with each iteration classes and their active record associations are loaded in to the memory which is occupying the memory ...reference [link_for_reference](http://stackoverflow.com/questions/181406/ruby-memory-management). – Vinay Jun 18 '13 at 17:26
  • Well, it wouldn't hurt to experiment a bit and set the GC.start methods in few places and monitor memory consumption (using GC.stat or similar method). – Ermin Dedovic Jun 18 '13 at 17:40
  • Sure ...will give it a try with this and let you know what I get.Thanks! – Vinay Jun 18 '13 at 17:41
  • @Ermin could you please answer my question at [link_question](http://stackoverflow.com/questions/17194393/rexmldocument-new-can-we-give-encode-parameters-on-this-line) – Vinay Jun 19 '13 at 15:09
  • @Ermin same error again with GC.start :( ..... Thin web server (v1.5.1 codename Straight Razor) >> Maximum connections set to 1024 >> Listening on 0.0.0.0:3000, CTRL+C to stop !! Unexpected error while processing request: failed to allocate memory .... pls help me out solve this error – Vinay Jun 20 '13 at 11:38
  • Well, can you monitor and paste here your memory stats in each iteration? – Ermin Dedovic Jun 20 '13 at 12:15
  • Could you please let me know where we can find that ?? – Vinay Jun 20 '13 at 13:00
  • thanks ! will it display on the console ?? – Vinay Jun 20 '13 at 13:35
  • just put logger.info GC.stat.to_s – Ermin Dedovic Jun 20 '13 at 13:36
  • but the problem is that .... i dont see my displays which I gave in the loop if the loop is not finished successfully ....console just ends with Thin web server (v1.5.1 codename Straight Razor) >> Maximum connections set to 1024 >> Listening on 0.0.0.0:3000, CTRL+C to stop !! Unexpected error while processing request: failed to allocate memory – Vinay Jun 20 '13 at 13:50
  • sandbox/vsadhu/workspace/DemoA$ rails s => Booting Thin => Rails 3.2.13 application starting in development on http://0.0.0.0:3000 => Call with -d to detach => Ctrl-C to shutdown server /home/vsadhu/.rvm/gems/ruby-1.9.3-p392/gems/actionpack-3.2.13/lib/action_dispatch/http/mime_type.rb:102: warning: already initialized constant PDF >> Thin web server (v1.5.1 codename Straight Razor) >> Maximum connections set to 1024 >> Listening on 0.0.0.0:3000, CTRL+C to stop !! Unexpected error while processing request: failed to allocate memory --------------- this is what all i get ! – Vinay Jun 20 '13 at 13:51
  • Try logger.debug, or logger.fatal – Ermin Dedovic Jun 20 '13 at 13:54
  • ok thanks ... this script usually takes 8hrs to run .... so will get back to you once done ..thanks for your help ! – Vinay Jun 20 '13 at 13:55
  • Could you please answer my question at [link](http://stackoverflow.com/questions/17243333/csv-read-v-s-temp-table-read-from-database-optimization-of-the-loop-and-active) – Vinay Jun 21 '13 at 20:06
  • @ErminDedovic could you please answer my question at [link](http://stackoverflow.com/questions/17281285/using-closure-tree-gem-instead-of-awesome-nested-set-please-help-me-with-th) – Vinay Jun 24 '13 at 17:56

0 Answers0