Ruby smarter_csv header issues

Question

I am trying to create a Ruby file that extracts data. I seem to run into an issue as the provided CSV files don't have headers attached. So, I am trying to tell smarter_csv that there are NO headers. The code, I use for this is:

transactions = SmarterCSV.process(file, {chunk_size: 20,remove_empty_values: false, remove_zero_values: false, headers_in_file: false, user_provided_headers:{timestamp: :timestamp, location: :loc, lat: :lat, long: :long, rfid: :rfid, start: :tart, ending: :end, type: :type , amount: :amount, status: :status}})

I also, add some new headers so they map right. The result unfortunately is an error, which is

/gems/smarter_csv-1.0.19/lib/smarter_csv/smarter_csv.rb:64:in process': undefined methodmap!' for nil:NilClass (NoMethodError)

I thought there might be something wrong with the passing of "false", so I changed it to a String, but that resulted unfortunately in that there were no custom headers added. I was wondering if anyone else is struggling with the headers in CSV. Maybe there is an easy solution, or maybe we need to contribute to the GEM. Thanks!

Answer 1

I also, add some new headers so they map right.

user_provided_headers:{timestamp: :timestamp, location: :loc, lat: :lat, long: :long, rfid: :rfid, start: :tart, ending: :end, type: :type , amount: :amount, status: :status}

:user_provided_headers is supposed to be an Array not a Hash:

user provided Array of header strings or symbols, to define
what headers should be used, overriding any in-file headers.
You can not combine the :user_provided_headers and :key_mapping options.

...

the provided CSV files don't have headers attached

If the csv file has no headers, the following makes no sense:

location: :loc

What would :location refer to? That seems to be saying that the header called :location in the file should be called :loc in the output. smarter_csvhas a setting for mapping existing headers in a file to new names in the output:

:key_mapping -- hash which maps headers from the CSV file to keys in the result hash

But you claim your csv file has no headers, and you are also telling smarter_csv that your file has no headers: headers_in_file: false.

Here are some examples of how things work:

csv.csv:

0,1,2
3,,5

And some code:

require 'smarter_csv'

data = SmarterCSV.process(
  'csv.csv',
  {
    headers_in_file: false,
    user_provided_headers: %i[x y z],
    remove_empty_values: false,
    remove_zero_values: false,
  }
)

p data

--output:--
[{:x=>0, :y=>1, :z=>2}, {:x=>3, :y=>"", :z=>5}]

If you tell smarter_csv that your file doesn't contain headers with headers_in_file: false, and then you provide a :key_mapping, which tells smarter csv to map the headers in the file to new names in the output, then you will get an error:

require 'smarter_csv'

data = SmarterCSV.process(
  'csv.csv',
  {
    headers_in_file: false,
    key_mapping: {'x' => 'x_val', 'y' => 'y_val', 'z' => 'z_val'},
    remove_empty_values: false,
    remove_zero_values: false,
  }
)

p data

--output:--
/Users/7stud/.rvm/gems/ruby-2.1.2/gems/smarter_csv-1.0.19/lib/smarter_csv/smarter_csv.rb:64:in `process': undefined method `map!' for nil:NilClass (NoMethodError)
    from 1.rb:3:in `<main>'

Now, if your csv file actually contains headers, headers_in_file: true, then providing a :key_mapping option makes sense:

csv.csv:

x,y,z
0,1,2
3,,5

The code:

require 'smarter_csv'

data = SmarterCSV.process(
  'csv.csv',
  {
    headers_in_file: true,
    key_mapping: {x: 'x_val', y: 'y_val', z: 'z_val'},  #keys must be symbols
    remove_empty_values: false,
    remove_zero_values: false,
  }
)

p data

--output:--
[{:x_val=>0, :y_val=>1, :z_val=>2}, {:x_val=>3, :y_val=>"", :z_val=>5}]

Based on your current understanding of smarter_csv, I suggest you reconsider whether you should be using the :chunk_size option. If you don't know, don't.