Put byte array to JSON and vice versa

Question

Is it possible to put a byte[] (byte array) to JSON?

if so, how can I do that in java? then read that JSON and convert that field again to byte[]?

Answer 1

Here is a good example of base64 encoding byte arrays. It gets more complicated when you throw unicode characters in the mix to send things like PDF documents. After encoding a byte array the encoded string can be used as a JSON property value.

Apache commons offers good utilities:

 byte[] bytes = getByteArr();
 String base64String = Base64.encodeBase64String(bytes);
 byte[] backToBytes = Base64.decodeBase64(base64String);

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Base64_encoding_and_decoding

Java server side example:

public String getUnsecureContentBase64(String url)
        throws ClientProtocolException, IOException {

            //getUnsecureContent will generate some byte[]
    byte[] result = getUnsecureContent(url);

            // use apache org.apache.commons.codec.binary.Base64
            // if you're sending back as a http request result you may have to
            // org.apache.commons.httpclient.util.URIUtil.encodeQuery
    return Base64.encodeBase64String(result);
}

JavaScript decode:

//decode URL encoding if encoded before returning result
var uriEncodedString = decodeURIComponent(response);

var byteArr = base64DecToArr(uriEncodedString);

//from mozilla
function b64ToUint6 (nChr) {

  return nChr > 64 && nChr < 91 ?
      nChr - 65
    : nChr > 96 && nChr < 123 ?
      nChr - 71
    : nChr > 47 && nChr < 58 ?
      nChr + 4
    : nChr === 43 ?
      62
    : nChr === 47 ?
      63
    :
      0;

}

function base64DecToArr (sBase64, nBlocksSize) {

  var
    sB64Enc = sBase64.replace(/[^A-Za-z0-9\+\/]/g, ""), nInLen = sB64Enc.length,
    nOutLen = nBlocksSize ? Math.ceil((nInLen * 3 + 1 >> 2) / nBlocksSize) * nBlocksSize : nInLen * 3 + 1 >> 2, taBytes = new Uint8Array(nOutLen);

  for (var nMod3, nMod4, nUint24 = 0, nOutIdx = 0, nInIdx = 0; nInIdx < nInLen; nInIdx++) {
    nMod4 = nInIdx & 3;
    nUint24 |= b64ToUint6(sB64Enc.charCodeAt(nInIdx)) << 18 - 6 * nMod4;
    if (nMod4 === 3 || nInLen - nInIdx === 1) {
      for (nMod3 = 0; nMod3 < 3 && nOutIdx < nOutLen; nMod3++, nOutIdx++) {
        taBytes[nOutIdx] = nUint24 >>> (16 >>> nMod3 & 24) & 255;
      }
      nUint24 = 0;

    }
  }

  return taBytes;
}

Answer 2

The typical way to send binary in json is to base64 encode it.

Java provides different ways to Base64 encode and decode a byte[]. One of these is DatatypeConverter.

Very simply

byte[] originalBytes = new byte[] { 1, 2, 3, 4, 5};
String base64Encoded = DatatypeConverter.printBase64Binary(originalBytes);
byte[] base64Decoded = DatatypeConverter.parseBase64Binary(base64Encoded);

You'll have to make this conversion depending on the json parser/generator library you use.

Answer 3

In line with @Qwertie's suggestion, but going further on the lazy side, you could just pretend that each byte is a ISO-8859-1 character. For the uninitiated, ISO-8859-1 is a single-byte encoding that matches the first 256 code points of Unicode.

So @Ash's answer is actually redeemable with a charset:

byte[] args2 = getByteArry();
String byteStr = new String(args2, Charset.forName("ISO-8859-1"));

This encoding has the same readability as BAIS, with the advantage that it is processed faster than either BAIS or base64 as less branching is required. It might look like the JSON parser is doing a bit more, but it's fine because dealing with non-ASCII by escaping or by UTF-8 is part of a JSON parser's job anyways. It could map better to some formats like MessagePack with a profile.

Space-wise however, it is usually a loss. With UTF-8 each non-ASCII byte would occupy 2 bytes, while BAIS uses (2+4n + r?(r+1):0) bytes for every run of 3n+r such bytes (r is the remainder). It would be a win on UTF-16, but who uses that for JSON?

(This encoding trick works on any language -- 8859-1 is very widely supported.)

Answer 4

If your byte array may contain runs of ASCII characters that you'd like to be able to see, you might prefer BAIS (Byte Array In String) format instead of Base64. The nice thing about BAIS is that if all the bytes happen to be ASCII, they are converted 1-to-1 to a string (e.g. byte array {65,66,67} becomes simply "ABC") Also, BAIS often gives you a smaller file size than Base64 (this isn't guaranteed).

After converting the byte array to a BAIS string, write it to JSON like you would any other string.

Here is a Java class (ported from the original C#) that converts byte arrays to string and back.

import java.io.*;
import java.lang.*;
import java.util.*;

public class ByteArrayInString
{
  // Encodes a byte array to a string with BAIS encoding, which 
  // preserves runs of ASCII characters unchanged.
  //
  // For simplicity, this method's base-64 encoding always encodes groups of 
  // three bytes if possible (as four characters). This decision may 
  // unfortunately cut off the beginning of some ASCII runs.
  public static String convert(byte[] bytes) { return convert(bytes, true); }
  public static String convert(byte[] bytes, boolean allowControlChars)
  {
    StringBuilder sb = new StringBuilder();
    int i = 0;
    int b;
    while (i < bytes.length)
    {
      b = get(bytes,i++);
      if (isAscii(b, allowControlChars))
        sb.append((char)b);
      else {
        sb.append('\b');
        // Do binary encoding in groups of 3 bytes
        for (;; b = get(bytes,i++)) {
          int accum = b;
          if (i < bytes.length) {
            b = get(bytes,i++);
            accum = (accum << 8) | b;
            if (i < bytes.length) {
              b = get(bytes,i++);
              accum = (accum << 8) | b;
              sb.append(encodeBase64Digit(accum >> 18));
              sb.append(encodeBase64Digit(accum >> 12));
              sb.append(encodeBase64Digit(accum >> 6));
              sb.append(encodeBase64Digit(accum));
              if (i >= bytes.length)
                break;
            } else {
              sb.append(encodeBase64Digit(accum >> 10));
              sb.append(encodeBase64Digit(accum >> 4));
              sb.append(encodeBase64Digit(accum << 2));
              break;
            }
          } else {
            sb.append(encodeBase64Digit(accum >> 2));
            sb.append(encodeBase64Digit(accum << 4));
            break;
          }
          if (isAscii(get(bytes,i), allowControlChars) &&
            (i+1 >= bytes.length || isAscii(get(bytes,i), allowControlChars)) &&
            (i+2 >= bytes.length || isAscii(get(bytes,i), allowControlChars))) {
            sb.append('!'); // return to ASCII mode
            break;
          }
        }
      }
    }
    return sb.toString();
  }

  // Decodes a BAIS string back to a byte array.
  public static byte[] convert(String s)
  {
    byte[] b;
    try {
      b = s.getBytes("UTF8");
    } catch(UnsupportedEncodingException e) { 
      throw new RuntimeException(e.getMessage());
    }
    for (int i = 0; i < b.length - 1; ++i) {
      if (b[i] == '\b') {
        int iOut = i++;

        for (;;) {
          int cur;
          if (i >= b.length || ((cur = get(b, i)) < 63 || cur > 126))
            throw new RuntimeException("String cannot be interpreted as a BAIS array");
          int digit = (cur - 64) & 63;
          int zeros = 16 - 6; // number of 0 bits on right side of accum
          int accum = digit << zeros;

          while (++i < b.length)
          {
            if ((cur = get(b, i)) < 63 || cur > 126)
              break;
            digit = (cur - 64) & 63;
            zeros -= 6;
            accum |= digit << zeros;
            if (zeros <= 8)
            {
              b[iOut++] = (byte)(accum >> 8);
              accum <<= 8;
              zeros += 8;
            }
          }

          if ((accum & 0xFF00) != 0 || (i < b.length && b[i] != '!'))
            throw new RuntimeException("String cannot be interpreted as BAIS array");
          i++;

          // Start taking bytes verbatim
          while (i < b.length && b[i] != '\b')
            b[iOut++] = b[i++];
          if (i >= b.length)
            return Arrays.copyOfRange(b, 0, iOut);
          i++;
        }
      }
    }
    return b;
  }

  static int get(byte[] bytes, int i) { return ((int)bytes[i]) & 0xFF; }

  public static int decodeBase64Digit(char digit)
    { return digit >= 63 && digit <= 126 ? (digit - 64) & 63 : -1; }
  public static char encodeBase64Digit(int digit)
    { return (char)((digit + 1 & 63) + 63); }
  static boolean isAscii(int b, boolean allowControlChars)
    { return b < 127 && (b >= 32 || (allowControlChars && b != '\b')); }
}

See also: C# unit tests.

Answer 5

Amazingly now org.json now lets you put a byte[] object directly into a json and it remains readable. you can even send the resulting object over a websocket and it will be readable on the other side. but i am not sure yet if the size of the resulting object is bigger or smaller than if you were converting your byte array to base64, it would certainly be neat if it was smaller.

It seems to be incredibly hard to measure how much space such a json object takes up in java. if your json consists merely of strings it is easily achievable by simply stringifying it but with a bytearray inside it i fear it is not as straightforward.

stringifying our json in java replaces my bytearray for a 10 character string that looks like an id. doing the same in node.js replaces our byte[] for an unquoted value reading <Buffered Array: f0 ff ff ...> the length of the latter indicates a size increase of ~300% as would be expected

Answer 6

what about simply this:

byte[] args2 = getByteArry();
String byteStr = new String(args2);