0

I've been using strings to represent decoded JSON integers larger than 32 bits. It seems the string_of_int is capable of dealing with large integer inputs. So a decoder, written (in the Json.Decode namespace):

id: json |> field("id", int) |> string_of_int,  /* 'id' is string */

is succefully dealing with integers of at least 37 bits.

Encoding, on the other hand, is proving troublesome for me. The remote server won't accept a string representation, and is expecting an int64. Is it possible to make bs-json support the int64 type? I was hoping something like this could be made to work:

type myData = { id: int64 };

let encodeMyData = (data:myData) => Json.Encode.(object_([("id", int64(myData.id)]))

Having to roll my own encoder is not nearly as formidable as a decoder, but ... I'd rather not.

Jamie
  • 7,075
  • 12
  • 56
  • 86
  • @glennsl Would it be "safe" to just add `external int64 : int64 -> Js.Json.t = "%identity"` to the encoder files? – Jamie Mar 15 '21 at 02:16

1 Answers1

0

You don't say exactly what problem you have with encoding. The int encoder does literally nothing except change the type, trusting that the int value is actually valid. So I would assume it's the int_of_string operation that causes problems. But that begs the question, if you can successfully decode it as an int, why are you then converting it to a string?

The underlying problem here is that JavaScript doesn't have 64 bit integers. The max safe integer is 253 - 1. JavaScript doesn't actually have integers at all, only floats, which can represent a certain range of integers, but can't efficiently do integer arithmetic unless they're converted to either 32-bit or 64-bit ints. And so for whatever reason, probably consistent overflow handling, it was decided in the EcmaScript specification that binary bitwise operations should operate on 32-bit integers. And so that opened the possibility for an internal 32-bit representation, a notation for creating 32-bit integers, and the possibility of optimized integer arithmetic on those.

So to your question:

Would it be "safe" to just add external int64 : int64 -> Js.Json.t = "%identity" to the encoder files?

No, because there's no 64-bit integer representation in JavaScript, int64 values are represented as an array of two Numbers I believe, but is also an internal implementation detail that's subject to change. Just casting it to Js.Json.t will not yield the result you expect.

So what can you do then?

I would recommend using float. In most respects this will behave exactly like JavaScript numbers, giving you access to its full range.

Alternatively you can use nativeint, which should behave like floats except for division, where the result is truncated to a 32-bit integer.

Lastly, you could also implement your own int_of_string to create an int that is technically out of range by using a couple of lightweight JavaScript functions directly, though I wouldn't really recommend doing this:

let bad_int_of_string = str => 
  str |> Js.Float.fromString |> Js.Math.floor_int;
glennsl
  • 28,186
  • 12
  • 57
  • 75
  • That was very informative, thank you. I was aware that in JS, int was limited to the significant of a IEEE-754 double, and it seems that `string_of_int` handles the 53 (ish) bits. What wasn't understood was the `int` in Re seems to be limited to 32bits - at least that's as far as my investigation got. I'll try your suggestions. Again, thank you. – Jamie Mar 15 '21 at 11:39
  • 1
    `int` itself isn't limited to 32 bits. It's just an ordinary JavaScript `Number` underneath, effectively a `float`. But some `int` __operations__ are limited to 32-bits, probably because they're optimized for integer arithmetic. – glennsl Mar 15 '21 at 11:57
  • `nativeint` + the OCaml documentation allowed me to _store_ an int of the 37 bits I required, but serializing is proving difficult during compile: `Json.Encode.(object_([("id", int(myData.id)),]))` The compiler is asking for a `nativeint()` method. Using the `int64` hack I suggested, unfortunately did - as you predicted - serialize the number into an array of two. Are you able to suggest a strategy for the final step of serializing? – Jamie Mar 15 '21 at 12:05
  • BTW: if you try to assign a greater than 32 bit number to an `int` type, the value is truncated to 32 bits in JS code. – Jamie Mar 15 '21 at 12:09
  • For encoding a `nativeint` you'd have to add an encoder like you suggested for `int64`. It won't work for `int64` as that has a different representation, but it should work for `nativeint`. I'd recommend just using `float` instead though. You can hide the implementation behind an abstract type if you want, and provide just the operations you need for that. – glennsl Mar 15 '21 at 12:16
  • 1
    Ahh ... thank you so much. Using float (which, obviously I didn't expect) worked. Your library is made all the better by your informative and timely answers. – Jamie Mar 15 '21 at 12:29
  • 1
    Happy to help!! – glennsl Mar 15 '21 at 13:02