Ruby Autovivification

Question

I've been trying to use autovivification in ruby to do simple record consolidation on this:

2009-08-21|09:30:01|A1|EGLE|Eagle Bulk Shpg|BUY|6000|5.03
2009-08-21|09:30:35|A2|JOYG|Joy Global Inc|BUY|4000|39.76
2009-08-21|09:30:35|A2|LEAP|Leap Wireless|BUY|2100|16.36
2009-08-21|09:30:36|A1|AINV|Apollo Inv Cp|BUY|2300|9.15
2009-08-21|09:30:36|A1|CTAS|Cintas Corp|SELL|9800|27.83
2009-08-21|09:30:38|A1|KRE|SPDR KBW Regional Banking ETF|BUY|9200|21.70
2009-08-21|09:30:39|A1|APA|APACHE CORPORATION|BUY|5700|87.18
2009-08-21|09:30:40|A1|FITB|Fifth Third Bancorp|BUY|9900|10.86
2009-08-21|09:30:40|A1|ICO|INTERNATIONAL COAL GROUP, INC.|SELL|7100|3.45
2009-08-21|09:30:41|A1|NLY|ANNALY CAPITAL MANAGEMENT. INC.|BUY|3000|17.31
2009-08-21|09:30:42|A2|GAZ|iPath Dow Jones - AIG Natural Gas Total Return Sub-Index ETN|SELL|6600|14.09
2009-08-21|09:30:44|A2|CVBF|Cvb Finl|BUY|1100|7.64
2009-08-21|09:30:44|A2|JCP|PENNEY COMPANY, INC.|BUY|300|31.05
2009-08-21|09:30:36|A1|AINV|Apollo Inv Cp|BUY|4500|9.15

so for example I want the record for A1 AINV BUY 9.15 to have a total of 6800. This is a perfect problem to use autovivification on. So heres my code:

#!/usr/bin/ruby

require 'facets'


h = Hash.autonew

File.open('trades_long.dat','r').each do |line|

        @date,@time,@account,@ticker,@desc,@type,amount,@price = line.chomp.split('|')
        if @account != "account"
           puts "#{amount}"
           h[@account][@ticker][@type][@price] += amount
         end

    #puts sum.to_s
end

The problem is no matter how I try to sum up the value in h[@account][@ticker][@type][@price] it gives me this error:

6000
/usr/local/lib/ruby/gems/1.9.1/gems/facets-2.7.0/lib/core/facets/hash/op_add.rb:8:in `merge': can't convert String into Hash (TypeError)
    from /usr/local/lib/ruby/gems/1.9.1/gems/facets-2.7.0/lib/core/facets/hash/op_add.rb:8:in `+'
    from ./trades_consolidaton.rb:13
    from ./trades_consolidaton.rb:8:in `each'
    from ./trades_consolidaton.rb:8

I've tried using different "autovivification" methods with no result. This wouldn't happen in perl! The autofvivification would know what you are trying to do. ruby doesn't seem to have this feature.

So my question really is, how do I perform simply "consolidation" of records in ruby. Specifically, how do I get the total for something like:

h[@account][@ticker][@type][@price]

Many thanks for your help!!

Just to clarify on glenn's solution. That would be perfect except it gives (with a few modifications to use the standard CSV library in ruby 1.9:

CSV.foreach("trades_long.dat", :col_sep => "|") do |row| 
     date,time,account,ticker,desc,type,amount,price = *row 
     records[[account,ticker,type,price]] += amount 
end

gives the following error:

TypeError: String can't be coerced into Fixnum
    from (irb):64:in `+'
    from (irb):64:in `block in irb_binding'
    from /usr/local/lib/ruby/1.9.1/csv.rb:1761:in `each'
    from /usr/local/lib/ruby/1.9.1/csv.rb:1197:in `block in foreach'
    from /usr/local/lib/ruby/1.9.1/csv.rb:1335:in `open'
    from /usr/local/lib/ruby/1.9.1/csv.rb:1196:in `foreach'
    from (irb):62
    from /usr/local/bin/irb:12:in `<main>'

Answer 1

I agree with Jonas that you (and Sam) are making this more complicated than it needs to be, but I think even his version is too complicated. I'd just do this:

require 'fastercsv'
records = Hash.new(0)
FasterCSV.foreach("trades_long.dat", :col_sep => "|") do |row|
  date,time,account,ticker,desc,type,amount,price = row.fields
  records[[account,ticker,type,price]] += amount.to_f
end

Now you have a hash with total amounts for each unique combination of account, ticker, type and price.

Answer 2

If you want a hash builder that works that way, you are going to have to redefine the + semantics.

For example, this works fine:

class HashBuilder
  def initialize
    @hash = {}
  end

  def []=(k,v)
    @hash[k] = v
  end

  def [](k)
    @hash[k] ||= HashBuilder.new
  end

  def +(val)
    val
  end

end


h = HashBuilder.new


h[1][2][3] += 1
h[1][2][3] += 3

p h[1][2][3]
# prints 4

Essentially you are trying to apply the + operator to a Hash.

>> {} + {}
NoMethodError: undefined method `+' for {}:Hash
        from (irb):1

However in facets{

>> require 'facets'
>> {1 => 10} + {2 => 20}
=> {1 => 10, 2 => 20} 
>> {} + 100
TypeError: can't convert Fixnum into Hash
        from /usr/lib/ruby/gems/1.8/gems/facets-2.7.0/lib/core/facets/hash/op_add.rb:8:in `merge'
        from /usr/lib/ruby/gems/1.8/gems/facets-2.7.0/lib/core/facets/hash/op_add.rb:8:in `+'
        from (irb):6
>> {} += {1 => 2}
=> {1=>2}
>>

If you want to redefine the + semantics for your hash in this occasion you can do:

class Hash; def +(v); v; end; end

Place this snippet before your original sample and all should be well. Keep in mind that you are changing the defined behavior for + (note + is not defined on Hash its pulled in with facets)

Answer 3

It looks like you are making it more complicated than it has to be. I would use the FasterCSV gem and Enumerable#inject something like this:

require 'fastercsv'

records=FasterCSV.read("trades_long.dat", :col_sep => "|")

records.sort_by {|r| r[3]}.inject(nil) {|before, curr|
   if !before.nil? && curr[3]==before[3]
    curr[6]=(curr[6].to_i+before[6].to_i).to_s
    records.delete(before)
  end
  before=curr
}

Answer 4

For others that find their way here, there is now also another option:

require 'xkeys' # on rubygems.org

h = {}.extend XKeys::Hash
...
# Start with 0.0 (instead of nil) and add the amount
h[@account, @ticker, @type, @price, :else => 0.0] += amount.to_f

This will generate a navigable structure. (Traditional keying with arrays of [@account, @ticker, @type, @price] as suggested earlier may be better this particular application). XKeys auto-vivifies on write rather than read, so querying the structure about elements that don't exist won't change the structure.