invalid byte sequence in UTF-8 CSV Rails 4

Question

I'm getting:

ArgumentError invalid byte sequence in UTF-8

With my Resque job

Below is my stack trace:

C:/BitNami/rubystack-2.0.0-4/ruby/lib/ruby/2.0.0/csv.rb:1780:in `sub!'
C:/BitNami/rubystack-2.0.0-4/ruby/lib/ruby/2.0.0/csv.rb:1780:in `block in shift'
C:/BitNami/rubystack-2.0.0-4/ruby/lib/ruby/2.0.0/csv.rb:1774:in `loop'
C:/BitNami/rubystack-2.0.0-4/ruby/lib/ruby/2.0.0/csv.rb:1774:in `shift'
C:/BitNami/rubystack-2.0.0-4/ruby/lib/ruby/2.0.0/csv.rb:1716:in `each'
C:/BitNami/rubystack-2.0.0-4/ruby/lib/ruby/2.0.0/csv.rb:1730:in `to_a'
C:/BitNami/rubystack-2.0.0-4/ruby/lib/ruby/2.0.0/csv.rb:1730:in `read'
C:/BitNami/rubystack-2.0.0-4/ruby/lib/ruby/2.0.0/csv.rb:1291:in `parse'
C:/BitNami/rubystack-2.0.0-4/projects/virtual_exhibition/app/jobs/users.rb:14:in `parse_csv'
C:/BitNami/rubystack-2.0.0-4/projects/virtual_exhibition/app/jobs/users.rb:6:in `perform'
C:/BitNami/rubystack-2.0.0-4/ruby/lib/ruby/gems/2.0.0/gems/resque-status-0.4.2/lib/resque/plugins/status.rb:161:in `safe_perform!'
C:/BitNami/rubystack-2.0.0-4/ruby/lib/ruby/gems/2.0.0/gems/resque-status-0.4.2/lib/resque/plugins/status.rb:137:in `perform'

Also below is my job getting called

class UserJob
  include Resque::Plugins::Status

  def perform
    puts "Parsing CSV and updating..."
    parse_csv
    puts "Update finished..."
  end

  def parse_csv
    @counter = 0 
    @row = []
    csv_text = File.read("#{Rails.public_path}/careersfair.csv").encode('UTF-8')
    csv = CSV.parse(csv_text, headers: false)
    csv.each do |row|
      user = User.find_by_email row[3]
      puts user.inspect
      if user.present?
        user.update(:first_name => row[0], :last_name => row[1], :industry => row[2], :event_ids => 1, :skip_invitation => true) 
        puts @counter += 1
      else
        puts "Not found - #{row[3]}"
      end
    end
  end
end

It seems CSV.parse is failing.

Is there a reason why this is happening?

csv_text = `File.read("#{Rails.public_path}/careersfair.csv").encode!('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '')` — Yana Agun Siswanto, Mar 04 '14 at 06:25
http://stackoverflow.com/questions/2982677/ruby-1-9-invalid-byte-sequence-in-utf-8 — Yana Agun Siswanto, Mar 04 '14 at 06:26

score 1 · Answer 1 · answered Mar 04 '14 at 06:24

I think your csv file has some invalid characters. Change "csv_text" into the follwing line.

csv_text = File.read("#{Rails.public_path}/careersfair.csv")encode("UTF-8", invalid: :replace, undef: :replace, replace: "?")

If you couldn't solve that by this way, I assume the csv file is not UTF-8.

If you're on Linux, try file -i filename.txt. You can see encoding of the file.

require "iconv"
conv = Iconv.new("UTF-8//IGNORE","ENCODING_OF_YOUR_FILE")
csv_text = File.read("#{Rails.public_path}/careersfair.csv")
text = conv.iconv(csv_text)

invalid byte sequence in UTF-8 CSV Rails 4

1 Answers1