2

Can someone recommend a good data structure/way for this use case? I'm using Ruby/Rails with MySQL.

I have a product feed with tens of thousands of products (with price, url, product_code, etc). I access the feed once a day and want to store the product info into a quickly accessible hash that persists in the database once the program stops running.

Should this (potentially very large) hash be its own class, or perhaps a serialized property of Product class?

My pseudocode:

Saving product feed data into hash and storing in database:

all_products_hash = Hash.new
product_feed = get_feed_api()
product_feed.for_each |product| do
    updated_product = Product.new(product.url, product.code, product.price)
    all_products_hash[product.url] = updated_product
end

all_products_hash.save

Retrieving hash content (not sure how to retrieve hash from database):

a_product = all_products_hash.get['url']

The key to the hash is the product.url. Am I initializing the hash correctly? Many thanks for any advice!

heebee313
  • 325
  • 1
  • 5
  • 16
  • 2
    Can you expand on why it needs to be a hash? I would suggest storing each product as an ActiveRecord model. – A5308Y Feb 22 '15 at 07:22
  • How are you using the feed? Do you actually want to save the feed as a hash in your database, or is this really a question about how to cache the api response? – ptd Feb 23 '15 at 17:58

3 Answers3

2

Don't use hash. You can use SQLite with file storage, which will also allow you queries, basic indexing and more. If you are concerned with memory vs file storage performance, please read Quickly dumping a database in memory to file.

As a ruby sqlite connector, you can use https://github.com/sparklemotion/sqlite3-ruby

Hope that helps.

Community
  • 1
  • 1
Martin Macak
  • 3,507
  • 2
  • 30
  • 54
1

I think this is what your looking for, but you may have to elaborate on the problem if I'm wrong.

hash = {}
result = ActiveRecord::Base.connection.execute('YOUR QUERY HERE')
while((r = result).present?)
  hash[r.url] = r
end

Alternatively (but slowly)

hash = {}
Product.all.each do |p|
  hash[p.url] = p.attributes
end

Assuming the table holds id, url, code, price; it should look like this:

{
'www.whatever.com' => {id: 1, url: 'www.whatever.com', code:'123', price: '123'}
'www.whatever2.com' => {id: 2, url: 'www.whatever2.com', code:'321', price: '321'}
}
Camway
  • 1,010
  • 14
  • 23
1

Kind of depends on the data structure of that hash. Are we talking about a very complex structure (embedded structures) or a simple key/value.

If complex pull out the key information and store that in columns (id, sku, name, description..). The rest of it can be stored in a varchar field if large enough. Then you can just serialize/deserialize to a hash/json string.

If it's a simple structure I would create a Model to represent it.

Marty
  • 174
  • 1
  • 9