Is there a way to remove the BOM from a UTF-8 encoded file?

Question

I know that all of my JSON files are encoded in UTF-8, but the data entry person who edited the JSON files saved it as UTF-8 with the BOM.

When I run my Ruby scripts to parse the JSON, it is failing with an error. I don't want to manually open 58+ JSON files and convert to UTF-8 without the BOM.

possible duplicate of [How to avoid tripping over UTF-8 BOM when reading files](http://stackoverflow.com/questions/543225/how-to-avoid-tripping-over-utf-8-bom-when-reading-files) — knut, Mar 27 '12 at 10:15

knut · Answer 1 · 2017-06-26T20:15:55.950

40

With ruby >= 1.9.2 you can use the mode r:bom|utf-8

This should work (I haven't test it in combination with json):

json = nil #define the variable outside the block to keep the data
File.open('file.txt', "r:bom|utf-8"){|file|
  json = JSON.parse(file.read)
}

It doesn't matter, if the BOM is available in the file or not.

Andrew remarked, that File#rewind can't be used with BOM.

If you need a rewind-function you must remember the position and replace rewind with pos=:

#Prepare test file
File.open('file.txt', "w:utf-8"){|f|
  f << "\xEF\xBB\xBF" #add BOM
  f << 'some content'
}

#Read file and skip BOM if available
File.open('file.txt', "r:bom|utf-8"){|f|
  pos =f.pos
  p content = f.read  #read and write file content
  f.pos = pos   #f.rewind  goes to pos 0
  p content = f.read  #(re)read and write file content
}

edited Jun 26 '17 at 20:15

answered Oct 15 '11 at 20:42

knut

27,320
6
84
112

one-liner for ruby (takes file paths on STDIN): `ruby -ne 'File.open($_.strip, "r:bom|utf-8") {|f| content=f.read(f.size); f.close; File.open($_.strip, "w").write(content) }` – Markus Strauss Oct 04 '12 at 17:18
So this failed for me because I used File#rewind. At least in ruby 1.9.2, using "bom|UTF-8" simply advances the file pointer when you open it, and this gets forgotten when you rewind. Seems to me like a poor implementation of open given this encoding parameter. – Andrew Schwartz Nov 14 '14 at 20:14
@AndrewSchwartz You are right. I extended my answer with a work around for your problem. – knut Nov 14 '14 at 22:19

Abe · Accepted Answer · 2011-02-16T02:08:12.603

29

So, the solution was to do a search and replace on the BOM via gsub! I forced the encoding of the string to UTF-8 and also forced the regex pattern to be encoded in UTF-8.

I was able to derive a solution by looking at http://self.d-struct.org/195/howto-remove-byte-order-mark-with-ruby-and-iconv and http://blog.grayproductions.net/articles/ruby_19s_string

def read_json_file(file_name, index)
  content = ''
  file = File.open("#{file_name}\\game.json", "r") 
  content = file.read.force_encoding("UTF-8")

  content.gsub!("\xEF\xBB\xBF".force_encoding("UTF-8"), '')

  json = JSON.parse(content)

  print json
end

edited Feb 16 '11 at 02:08

answered Feb 16 '11 at 02:00

Abe

6,386
12
46
75

14

I would recommend `sub!(/^\xEF\xBB\xBF/, '')` as the BOM occures only once at the beginning of the string. – Jan Nov 26 '12 at 10:08
I would actually suggest `String#delete!` over `String#gsub!(..., '')` – Farley Knight Dec 18 '13 at 21:28
@Jan your suggestion is good unless your setup is defaulting to UTF-8 or at least that's what I believe to be my problem because your command does not work on mine. Just doing `content.sub!(/^\xEF\xBB\xBF/, '')` gives an "invalid UTF-8 byte sequence" error. – Ryan Jan 12 '17 at 08:20
3

@Ryan I think @Jan simply meant using `sub` instead of `gsub`, but didn't necessarily mean to drop the `force_encoding`. – Bo Jeanes Oct 24 '17 at 00:51

score 12 · Answer 3 · answered May 10 '13 at 16:02

12

You can also specify encoding with the File.read and CSV.read methods, but you don't specify the read mode.

File.read(path, :encoding => 'bom|utf-8')
CSV.read(path, :encoding => 'bom|utf-8')

answered May 10 '13 at 16:02

Tim Uruski

141
2
3

it works well 10 years later – Vadim Eremeev May 12 '23 at 07:49

Andrew Schwartz · Answer 4 · 2014-11-14T20:43:59.293

the "bom|UTF-8" encoding works well if you only read the file once, but fails if you ever call File#rewind, as I was doing in my code. To address this, I did the following:

def ignore_bom
  @file.ungetc if @file.pos==0 && @file.getc != "\xEF\xBB\xBF".force_encoding("UTF-8")
end

which seems to work well. Not sure if there are other similar type characters to look out for, but they could easily be built into this method that can be called any time you rewind or open.

score 2 · Answer 5 · answered Aug 28 '18 at 19:36

2

Server side cleanup of utf-8 bom bytes that worked for me:

csv_text.gsub!("\xEF\xBB\xBF".force_encoding(Encoding::BINARY), '')

answered Aug 28 '18 at 19:36

UdiP

21
1

Tilo · Answer 6 · 2023-03-19T08:08:18.137

I just implemented this for the smarter_csv gem, and want to share this in case someone comes across this issue.

The problem is to remove byte-sequences independent of the string encoding. The solution is to use the methods bytes and byteslice from the String class.

See: https://ruby-doc.org/core-3.1.1/String.html#method-i-bytes

    UTF_8_BOM = %w[ef bb bf].freeze

    def remove_bom(str)
      str_as_hex = str.bytes.map{|x| x.to_s(16)}
      return str.byteslice(3..-1) if str_as_hex[0..2] == UTF_8_BOM

      str
    end

Is there a way to remove the BOM from a UTF-8 encoded file?

6 Answers6

Linked