invalid byte sequence in UTF-8 CSV Rails 4 - ruby-on-rails

Question

I'm getting:
ArgumentError invalid byte sequence in UTF-8
With my Resque job
Below is my stack trace:
C:/BitNami/rubystack-2.0.0-4/ruby/lib/ruby/2.0.0/csv.rb:1780:in `sub!'
C:/BitNami/rubystack-2.0.0-4/ruby/lib/ruby/2.0.0/csv.rb:1780:in `block in shift'
C:/BitNami/rubystack-2.0.0-4/ruby/lib/ruby/2.0.0/csv.rb:1774:in `loop'
C:/BitNami/rubystack-2.0.0-4/ruby/lib/ruby/2.0.0/csv.rb:1774:in `shift'
C:/BitNami/rubystack-2.0.0-4/ruby/lib/ruby/2.0.0/csv.rb:1716:in `each'
C:/BitNami/rubystack-2.0.0-4/ruby/lib/ruby/2.0.0/csv.rb:1730:in `to_a'
C:/BitNami/rubystack-2.0.0-4/ruby/lib/ruby/2.0.0/csv.rb:1730:in `read'
C:/BitNami/rubystack-2.0.0-4/ruby/lib/ruby/2.0.0/csv.rb:1291:in `parse'
C:/BitNami/rubystack-2.0.0-4/projects/virtual_exhibition/app/jobs/users.rb:14:in `parse_csv'
C:/BitNami/rubystack-2.0.0-4/projects/virtual_exhibition/app/jobs/users.rb:6:in `perform'
C:/BitNami/rubystack-2.0.0-4/ruby/lib/ruby/gems/2.0.0/gems/resque-status-0.4.2/lib/resque/plugins/status.rb:161:in `safe_perform!'
C:/BitNami/rubystack-2.0.0-4/ruby/lib/ruby/gems/2.0.0/gems/resque-status-0.4.2/lib/resque/plugins/status.rb:137:in `perform'
Also below is my job getting called
class UserJob
include Resque::Plugins::Status
def perform
puts "Parsing CSV and updating..."
parse_csv
puts "Update finished..."
end
def parse_csv
#counter = 0
#row = []
csv_text = File.read("#{Rails.public_path}/careersfair.csv").encode('UTF-8')
csv = CSV.parse(csv_text, headers: false)
csv.each do |row|
user = User.find_by_email row[3]
puts user.inspect
if user.present?
user.update(:first_name => row[0], :last_name => row[1], :industry => row[2], :event_ids => 1, :skip_invitation => true)
puts #counter += 1
else
puts "Not found - #{row[3]}"
end
end
end
end
It seems CSV.parse is failing.
Is there a reason why this is happening?

Answer 1

I think your csv file has some invalid characters. Change "csv_text" into the follwing line.
csv_text = File.read("#{Rails.public_path}/careersfair.csv")encode("UTF-8", invalid: :replace, undef: :replace, replace: "?")
If you couldn't solve that by this way, I assume the csv file is not UTF-8.
If you're on Linux, try file -i filename.txt. You can see encoding of the file.
require "iconv"
conv = Iconv.new("UTF-8//IGNORE","ENCODING_OF_YOUR_FILE")
csv_text = File.read("#{Rails.public_path}/careersfair.csv")
text = conv.iconv(csv_text)