-1

Please help me in solving this error.

I am getting this error while loading records from text files in to database using ruby scripts.

It just works fine if I use small number of records to load in to the database.But fails if there are large number of records.

CSV.foreach(fileName) do |line|
    completePath = line[0]                                                
    num_of_bps = line[1]

    completePath = cluster_path+ '/' + completePath
    inode = FileOrFolder.find_by_fullpath(completePath, :select=>"id") 

    metric_instance = MetricInstance.find(:first, :conditions=>["file_or_folder_id = ? AND dataset_id = ?", inode.id, dataset_id])
    add_entry(metric_instance.id, num_of_bps, num_of_bp_tests) 
end

def self.add_entry(metaid, num_of_bps, num_of_bp_tests)
    entry = Bp.new
    entry.metric_instance_id = metaid
    entry.num_of_bps = num_of_bps
    entry.num_of_bp_tests = num_of_bp_tests
    entry.save
    return entry
end
lurker
  • 56,987
  • 9
  • 69
  • 103
Vinay
  • 237
  • 2
  • 8
  • 17
  • You'll need to narrow this down to a reasonable size sample script that exhibits the error and show your script in your question. Otherwise, there's just not enough information here for anyone to help. – lurker Jun 14 '13 at 13:23
  • I have added the script code please review it.... there are several files in the same fashion that are being loaded in to database. – Vinay Jun 14 '13 at 13:36

1 Answers1

0

Try something like this:

File.open(fileName) do |csv|
  csv.each_line do |line|
    CSV.parse(line) do |values|
      # Here you can do your manipulation
    end
  end
end

This way is slower, but it should ensure you don't get out of memory.

Ermin Dedovic
  • 907
  • 4
  • 6
  • Thanks ! ... is the out-of-memory issue specific to the file size ? if so is there any way in ruby to check the file size within the script and call appropriate method? and also where can we check the maximum allowed size of the file? – Vinay Jun 14 '13 at 14:07
  • You have size method of File class: http://ruby-doc.org/core-2.0/File.html. You can also check available physical memory http://stackoverflow.com/questions/4029391/how-do-you-find-the-physical-memory-free-on-the-machine-in-ruby. Maximum memory that x86 programs can allocate is 2^32B=4GB and this can be a problem if you are using 32 bit Ruby. And the memory allocated in this case primarily depends of the file size (because it is a big file), and some memory is also used by the other variables on the heap. – Ermin Dedovic Jun 14 '13 at 14:26
  • @Ermin......Could you please answer this question ..... [link_to_question_2](http://stackoverflow.com/questions/17149761/failed-to-allocate-memory-on-the-console-and-internal-server-error-on-the-browse) – Vinay Jun 17 '13 at 15:35
  • @Ermin...could you please answer my question at ..... [link_question](http://stackoverflow.com/questions/17218324/is-there-an-efficient-way-than-csv-foreach-way-of-reading-files) – Vinay Jun 20 '13 at 18:18