perl & python writing out non-ASCII characters into JSON differently

Question

I have hash keys that look like this:

1я310яHOM_REF_truth:HOM_ALT_test:discordant_hom_ref_to_hom_altяAяC

this is a string that is joined by the Cyrillic letter я which I chose as a delimiter because it will never appear in this files.

I write this to a JSON file in Perl 5.30.2 thus:

use JSON 'encode_json';

sub hash_to_json_file {
    my $hash     = shift;
    my $filename = shift;
    my $json = encode_json $hash;
    open my $out, '>', $filename;
    say $out $json
}

and in python 3.8:

use json
def hash_to_json_file(hashtable,filename):
    json1=json.dumps(hashtable)
    f = open(filename,"w+")
    print(json1,file=f)
    f.close()

when I try to load a JSON written by Python back into a Perl script, I see a cryptic error that I don't know how to solve:

Wide character in say at read_json.pl line 27.

Reading https://perldoc.perl.org/perlunifaq.html I've tried adding use utf8 to my script, but it doesn't work. I've also tried '>:encoding(UTF-8)' within my subroutine, but the same error results.

Upon inspection of the JSON files, I see keys like "1Ñ180ÑHET_ALT_truth:HET_REF_test:discordant_het_alt_to_het_refÑAÑC,G" where ÑAÑ substitutes я. In the JSON written by python, I see \u044f I think that this is the wide character, but I don't know how to change it back.

I've also tried changing my subroutine:

use Encode 'decode';
sub json_file_to_hash {
   my $file = shift;
   open my $in, '<:encoding(UTF-8)', $file;
   my $json = <$in>;
   my $ref = decode_json $json;
   $ref = decode('UTF-8', $json);
   return %{ $ref }
}

but this gives another error:

Wide character in hash dereference at read_json.pl line 17, <$_[...]> line 1

How can I get python JSON read into Perl correctly?

Answer 1

use utf8;                               # Source is encoded using UTF-8
use open ':std', ':encoding(UTF-8)';    # For say to STDOUT.  Also default for open()

use JSON qw( decode_json encode_json );

sub hash_to_json_file {
    my $qfn = shift;
    my $ref = shift;
    my $json = encode_json($ref);       # Produces UTF-8
    open(my $fh, '>:raw', $qfn)         # Write it unmangled
       or die("Can't create \"$qfn\": $!\n");

    say $fh $json;
}

sub json_file_to_hash {
    my $qfn = shift;
    open(my $fh, '<:raw', $qfn)         # Read it unmangled
       or die("Can't create \"$qfn\": $!\n");

    local $/;                           # Read whole file
    my $json = <$fh>;                   # This is UTF-8
    my $ref = decode_json($json);       # This produces decoded text
    return $ref;                        # Return the ref rather than the keys and values.
}

my $src = { key => "1я310яHOM_REF_truth:HOM_ALT_test:discordant_hom_ref_to_hom_altяAяC" };
hash_to_json("a.json", $src);
my $dst = hash_to_json("a.json");
say $dst->{key};

---

You could also avoid using :raw by using from_json and to_json.

use utf8;                               # Source is encoded using UTF-8
use open ':std', ':encoding(UTF-8)';    # For say to STDOUT. Also default for open()

use JSON qw( from_json to_json );

sub hash_to_json_file {
    my $qfn  = shift;
    my $hash = shift;
    my $json = to_json($hash);          # Produces decoded text.
    open(my $fh, '>', $qfn)             # "use open" will add :encoding(UTF-8)
       or die("Can't create \"$qfn\": $!\n");

    say $fh $json;                      # Encoded by :encoding(UTF-8)
}

sub json_file_to_hash {
    my $qfn = shift;
    open(my $fh, '<', $qfn)             # "use open" will add :encoding(UTF-8)
       or die("Can't create \"$qfn\": $!\n");

    local $/;                           # Read whole file
    my $json = <$fh>;                   # Decoded text thanks to "use open".
    my $ref = from_json($json);         # $ref contains decoded text.
    return $ref;                        # Return the ref rather than the keys and values.
}

my $src = { key => "1я310яHOM_REF_truth:HOM_ALT_test:discordant_hom_ref_to_hom_altяAяC" };
hash_to_json("a.json", $src);
my $dst = hash_to_json("a.json");
say $dst->{key};

Answer 2

I like the ascii option so that the JSON output is all 7-bit ASCII

my $json = JSON->new->ascii->encode($hash);

Both the Perl and Python JSON modules will be able to read it.