Encode string as \uXXXX

Question

I am trying to port a code from python to ruby, and having difficulties in one of the functions that encodes a UTF-8 string to JSON.

I have stripped down the code to what I believe is my problem.

I would like to make ruby output the exact same output as python.

The python code:

#!/usr/bin/env python
# encoding: utf-8

import json
import hashlib

text = "ÀÈG"
js = json.dumps( { 'data': text } )

print 'Python:'
print js
print hashlib.sha256(js).hexdigest()

The ruby code:

#!/usr/bin/env ruby
require 'json'
require 'digest'

text = "ÀÈG"
obj = {'data': text}
# js = obj.to_json # not using this, in order to get the space below
js = %Q[{"data": "#{text}"}]

puts 'Ruby:'
puts js
puts Digest::SHA256.hexdigest js

When I run both, this is the output:

$ ./test.rb && ./test.py
Ruby:
{"data": "ÀÈG"}
6cbe518180308038557d28ecbd53af66681afc59aacfbd23198397d22669170e
Python:
{"data": "\u00c0\u00c8G"}
a6366cbd6750dc25ceba65dce8fe01f283b52ad189f2b54ba1bfb39c7a0b96d3

What do I need to change in the ruby code to make its output identical to the python output (at least the final hash)?

Notes:

• I have tried things from this SO question (and others) without success.
• The code above produces identical results when using only english characters, so I know the hashing is the same.

Answer 1

Surely someone will come along with a more elegant (or at least a more efficient and robust) solution, but here's one for the time being:

#!/usr/bin/env ruby

require 'json'
require 'digest'

text = 'ÀÈG'
  .encode('UTF-16')                          # convert UTF-8 characters to UTF-16
  .inspect                                   # escape UTF-16 characters and convert back to UTF-8
  .sub(/^"\\u[Ff][Ee][Ff][Ff](.*?)"$/, '\1') # remove outer quotes and BOM
  .gsub(/\\u\w{4}/, &:downcase!)             # downcase alphas in escape sequences

js = { data: text }                          # wrap in containing data structure
  .to_json(:space=>' ')                      # convert to JSON with spaces after colons
  .gsub(/\\\\u(?=\w{4})/, '\\u')             # remove extra backslashes

puts 'Ruby:', js, Digest::SHA256.hexdigest(js)

Output:

$ ./test.rb 
Ruby:
{"data": "\u00c0\u00c8G"}
a6366cbd6750dc25ceba65dce8fe01f283b52ad189f2b54ba1bfb39c7a0b96d3