2

I've done something similar in Bash but not sure how to do this in 2-3 concise lines of Ruby, although it seems it can be done in a snap with the right coding Kung-Fu.

I have "file.csv" with a header row that looks like:

Ticker,"Price","Market Cap","Average Volume","Analyst Recom","Relative Strength Index (14)","Sector","Industry","Dividend Yield","Beta","52-Week Low","52-Week High","50-Day Low","50-Day High","Company","50-Day Simple Moving Average","Country","P/E","Forward P/E","PEG","P/S","P/B","P/Cash","P/Free Cash Flow","Payout Ratio","EPS (ttm)","EPS growth this year","EPS growth next year","EPS growth past 5 years","EPS growth next 5 years","Sales growth past 5 years","EPS growth quarter over quarter","Sales growth quarter over quarter","Shares Outstanding","Shares Float","Insider Ownership","Insider Transactions","Institutional Ownership","Institutional Transactions","Float Short","Short Ratio","Return on Assets","Return on Equity","Return on Investment","Current Ratio","Quick Ratio","LT Debt/Equity","Total Debt/Equity","Gross Margin","Operating Margin","Profit Margin","Performance (Week)","Performance (Month)","Performance (Quarter)","Performance (Half Year)","Performance (Year)","Performance (Year)","Average True Range","Volatility (Week)","Volatility (Month)","20-Day Simple Moving Average","200-Day Simple Moving Average","Change from Open","Gap","Relative Volume","Change","Volume","Earnings Date","No."

followed by about 7000 lines that look like:

FCD,27.89,,0.94,,66.75,"Financial","Exchange Traded Fund",3.13%,,19.75%,-0.36%,6.37%,-0.36%,"Focus Morningstar Consumer Defensive ETF",2.28%,"USA",,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,0.36%,3.07%,9.93%,10.85%,,2.01%,0.12,0.04%,0.21%,1.26%,6.69%,0.00%,-0.04%,0.96,-0.04%,900,,2186
FCE-A,14.59,2496.64,960.33,2.50,54.76,"Financial","Property Management",,2.83,56.55%,-24.87%,36.61%,-7.77%,"Forest City Enterprises Inc.",11.49%,"USA",,69.48,,2.2
5,1.58,10.87,,,-0.02,410.77%,250.00%,-10.06%,8.00%,1.54%,-28.77%,-9.00%,171.12,136.94,0.26%,-8.25%,74.80%,-0.13%,4.62%,6.59,0.46%,-0.12%,0.54%,,,4.35,4.35,39.54%,
4.82%,4.60%,-4.01%,8.96%,25.45%,13.10%,-22.80%,23.43%,0.44,3.07%,2.98%,-0.89%,1.49%,-1.62%,0.00%,0.47,-1.62%,449874,12/8/2010 4:30:00 PM,2187

Given a ticker symbol "FCD", I am trying to mass-assign about 30 new variables taken from the header fields to values matching the "FCD" row.

Each new variable is to be prefixed with fv_ and have the rest as in the field name minus all punctuation, spaces, quotes etc. (variable-unfriendly stuff).

So for "FCD" I am trying to give my script:

fv_Ticker="FCD"  
fv_Price=27.89  
fv_MarketCap=""  
fv_VolatilityMonth=0.21  # if get String not Float because of trailing % in "0.21%" that's okay, will deal with it later
etc.

I should note that I quit using any kind of CSV.read or CSV.foreach due to horrible slowdown using native Ruby 1.9.x CSV objects, which took minutes to read and is therefore unacceptable in a realtime application that runs repeatedly.

Instead I've been using a Ruby pipe to 'awk' to assign individual variables read from the file instantly like this:

$stock="FCD"
$dividend_yield = IO.readlines("|awk -F, '$1==\"#{$stock}\" {print $9}' finviz.AllStocks.csv")[0].to_f
$beta = IO.readlines("|awk -F, '$1==\"#{$stock}\" {print $10}' AllStocks.csv")[0].to_f

but now it's getting too hairy to not generalize. It needs to work with any CSV-like file with unknown fields until the first row it sees.

Community
  • 1
  • 1
Marcos
  • 4,796
  • 5
  • 40
  • 64
  • Why variables? Why not a hash of `{"FCD"=>someObject}`? – Linuxios Feb 29 '12 at 16:09
  • A hash would be fine too, if not better, but it needs to be fast. [Previously](http://stackoverflow.com/q/8476769/1069375) reading the CSV into Ruby failed me. – Marcos Feb 29 '12 at 16:23
  • Filling the symbol table with variables can't be less efficient than a hash. Also, using global variables for anything, especially this, is not a good idea. – Linuxios Feb 29 '12 at 16:30
  • I saw on the bash solution that you were using Google finance. Can Google finance give output in any more efficient format than CSV or XML? A binary format maybe? YAML? JSON? All of these are much more efficient. – Linuxios Feb 29 '12 at 16:32
  • Possibly of use http://snippets.dzone.com/posts/show/3899 – Mark Thomas Feb 29 '12 at 16:54
  • If you can make some assumptions about the CSV (such as data will not contain commas) then you can simply `split(",")` the lines and it will be fast. – Mark Thomas Feb 29 '12 at 16:56
  • @MarkThomas Unfortunately data does contain commas, like in company name `"Apple, Inc."`. Fortunately this should only happen inside double-quoted cells, so maybe some slick regex can handle that. – Marcos Feb 29 '12 at 17:57
  • @Linux_iOS.rb.cpp.c.lisp.m.sh Somewhere not far (on that [same link's](http://stackoverflow.com/a/8522791/1069375) page?) I've seen a Yahoo YQL or other online service that returns JSON from a web query like GF's xml, but not sure how that helps me read arbitrary fields from a single record in any CSV file defining its header row (not just about stocks) – Marcos Feb 29 '12 at 18:26

2 Answers2

1

This single-line version is even better than the last. It's shorter, simpler, and what's especially appealing, adapts all the header titles into (almost-)variable-ready unique names so I don't have to worry about parsing out most of that stuff later:

csv_data = CSV.parse IO.read(%`|sed -n "1p; /^#{$stock},/p" AllStocks.csv`), {:headers => true, :return_headers => false, :header_converters => :symbol, :converters => :all}

The results:

csv_data.size
=> 1

csv_data.to_a
=> [[:ticker, :price, :market_cap, :average_volume, :analyst_recom, :relative_strength_index_14, :sector, :industry, :dividend_yield, :beta, :"52week_low", :"52week_high", :"50day_low", :"50day_high", :company, :"50day_simple_moving_average", :country, :pe, :forward_pe, :peg, :ps, :pb, :pcash, :pfree_cash_flow, :payout_ratio, :eps_ttm, :eps_growth_this_year, :eps_growth_next_year, :eps_growth_past_5_years, :eps_growth_next_5_years, :sales_growth_past_5_years, :eps_growth_quarter_over_quarter, :sales_growth_quarter_over_quarter, :shares_outstanding, :shares_float, :insider_ownership, :insider_transactions, :institutional_ownership, :institutional_transactions, :float_short, :short_ratio, :return_on_assets, :return_on_equity, :return_on_investment, :current_ratio, :quick_ratio, :lt_debtequity, :total_debtequity, :gross_margin, :operating_margin, :profit_margin, :performance_week, :performance_month, :performance_quarter, :performance_half_year, :performance_year, :performance_year, :average_true_range, :volatility_week, :volatility_month, :"20day_simple_moving_average", :"200day_simple_moving_average", :change_from_open, :gap, :relative_volume, :change, :volume, :earnings_date, :no], ["ANAD", 2.57, 175.2, 442.65, 2.9, 38.21, "Technology", "Semiconductor - Integrated Circuits", nil, 2.3, "33.85%", "-52.84%", "27.23%", "-20.19%", "Anadigics, Inc.", "-2.63%", "USA", nil, nil, nil, 1.15, 1.05, 3.08, nil, nil, -0.73, "-4002.36%", "57.90%", "-7.14%", "15.67%", "-1.69%", "-564.96%", "-39.37%", 68.17, 66.25, "3.87%", "-7.51%", "47.64%", "-6.24%", "3.30%", 4.95, "-23.34%", "-26.70%", "-26.33%", 4.89, 3.9, 0.0, 0.0, "20.35%", "-32.79%", "-32.27%", "-5.17%", "-7.22%", "21.23%", "-6.20%", "-50.39%", "17.35%", 0.16, "5.14%", "5.51%", "-13.04%", "-3.71%", "-4.10%", "0.00%", 0.95, "-4.10%", 421200, "2/22/2012 7:00:00 AM", 280]]


$company = csv_data[:company][0]
=> "Anadigics, Inc."

csv_data[:volatility_month]
=> ["5.51%"]
Community
  • 1
  • 1
Marcos
  • 4,796
  • 5
  • 40
  • 64
0

Think I'm on to something. A middle-of-the-road way that lets Ruby see only a two-line CSV input, so native processing should be fast:

puts IO.readlines(%`|sed -n "1p; /^#{$stock},/p" AllStocks.csv`) # sed filters out the gazillion other lines that make Ruby slow

It works. Since Ruby only sees a short CSV, one line for header and one line for data thanks to sed, it loads in a split second, and I have my hash: (the associative array I was after)

$stock = "ANAD"

csv_data = CSV.parse IO.read(%`|sed -n "1p; /^#{$stock},/p" AllStocks.csv`)
headers = csv_data.shift.map {|i| i.to_s }
string_data = csv_data.map {|row| row.map {|cell| cell.to_s } }
tickers = array_of_hashes = string_data.map {|row| Hash[*headers.zip(row).flatten] }

pp tickers[0]    
=> {"Ticker"=>"ANAD", "Price"=>"2.57", "Market Cap"=>"175.20", "Average Volume"=>"442.65", "Analyst Recom"=>"2.90", "Relative Strength Index (14)"=>"38.21", "Sector"=>"Technology", "Industry"=>"Semiconductor - Integrated Circuits", "Dividend Yield"=>"", "Beta"=>"2.30", "52-Week Low"=>"33.85%", "52-Week High"=>"-52.84%", "50-Day Low"=>"27.23%", "50-Day High"=>"-20.19%", "Company"=>"Anadigics, Inc.", "50-Day Simple Moving Average"=>"-2.63%", "Country"=>"USA", "P/E"=>"", "Forward P/E"=>"", "PEG"=>"", "P/S"=>"1.15", "P/B"=>"1.05", "P/Cash"=>"3.08", "P/Free Cash Flow"=>"", "Payout Ratio"=>"", "EPS (ttm)"=>"-0.73", "EPS growth this year"=>"-4002.36%", "EPS growth next year"=>"57.90%", "EPS growth past 5 years"=>"-7.14%", "EPS growth next 5 years"=>"15.67%", "Sales growth past 5 years"=>"-1.69%", "EPS growth quarter over quarter"=>"-564.96%", "Sales growth quarter over quarter"=>"-39.37%", "Shares Outstanding"=>"68.17", "Shares Float"=>"66.25", "Insider Ownership"=>"3.87%", "Insider Transactions"=>"-7.51%", "Institutional Ownership"=>"47.64%", "Institutional Transactions"=>"-6.24%", "Float Short"=>"3.30%", "Short Ratio"=>"4.95", "Return on Assets"=>"-23.34%", "Return on Equity"=>"-26.70%", "Return on Investment"=>"-26.33%", "Current Ratio"=>"4.89", "Quick Ratio"=>"3.90", "LT Debt/Equity"=>"0.00", "Total Debt/Equity"=>"0.00", "Gross Margin"=>"20.35%", "Operating Margin"=>"-32.79%", "Profit Margin"=>"-32.27%", "Performance (Week)"=>"-5.17%", "Performance (Month)"=>"-7.22%", "Performance (Quarter)"=>"21.23%", "Performance (Half Year)"=>"-6.20%", "Performance (Year)"=>"17.35%", "Average True Range"=>"0.16", "Volatility (Week)"=>"5.14%", "Volatility (Month)"=>"5.51%", "20-Day Simple Moving Average"=>"-13.04%", "200-Day Simple Moving Average"=>"-3.71%", "Change from Open"=>"-4.10%", "Gap"=>"0.00%", "Relative Volume"=>"0.95", "Change"=>"-4.10%", "Volume"=>"421200", "Earnings Date"=>"2/22/2012 7:00:00 AM", "No."=>"280"}


tickers[0]["Volatility (Month)"].to_f
=> 5.51
Marcos
  • 4,796
  • 5
  • 40
  • 64