Parsing Byte Array in Apps Script

Question

I get the following data from a stock API:

0, 0, 0, 1, 0, 4, 71, 79, 79, 71, 0, 0, 0, 0, 4, 68, 93, -17, -65, -67, 82, 68, 95, 5, 31, >68, 93, 64, 0, 68, 93, -17, -65, -67, -17, -65, -67, 70, -17, -65, -67, 52, -17, -65, -67, >0, 0, 1, 63, -17, -65, -67, 99, 30, 0, 68, 92, -17, -65, -67, -17, -65, -67, 68, 94, -17, >-65, -67, 0, 68, 91, 81, 72, 68, 94, -17, -65, -67, -17, -65, -67, 70, -17, -65, -67, -17, >-65, -67, 87, 0, 0, 1, 63, -17, -65, -67, -17, -65, -67, 122, 0, 68, 93, -17, -65, -67, >-17, -65, -67, 68, 94, 74, -17, -65, -67, 68, 91, -17, -65, -67, 0, 68, 91, -17, -65, -67, >70, 34, 107, 10, 0, 0, 1, 63, -17, -65, -67, -17, -65, -67, -17, -65, -67, 0, 68, 95, 95, >92, 68, 95, -17, -65, -67, 61, 68, 93, -17, -65, -67, -17, -65, -67, 68, 94, -17, -65, >-67, 0, 70, 127, -17, -65, -67, 0, 0, 1, 63, -17, -65, -67, -17, -65, -67

This is the piece of binary read succesfully by the code below which returns the array {error=0.0, symbol=GOOG, count=1, bars=4, length=4, bar=[]}:

0, 0, 0, 1, 0, 4, 71, 79, 79, 71, 0, 0, 0, 0, 4

It is from this point forward I'm struggling:

68, 93, -17, -65, -67, 82, 68, 95, 5, 31, >68, 93, 64, 0, 68, 93, -17, -65, -67, -17, -65, >-67, 70, -17, -65, -67, 52, -17, -65, -67, >0, 0, 1, 63, -17, -65, -67, 99, 30, 0, 68, 92, >-17, -65, -67, -17, -65, -67, 68, 94, -17, >-65, -67, 0, 68, 91, 81, 72, 68, 94, -17, -65, >-67, -17, -65, -67, 70, -17, -65, -67, -17, >-65, -67, 87, 0, 0, 1, 63, -17, -65, -67, >-17, -65, -67, 122, 0, 68, 93, -17, -65, -67, >-17, -65, -67, 68, 94, 74, -17, -65, -67, >68, 91, -17, -65, -67, 0, 68, 91, -17, -65, -67, >70, 34, 107, 10, 0, 0, 1, 63, -17, -65, >-67, -17, -65, -67, -17, -65, -67, 0, 68, 95, 95, >92, 68, 95, -17, -65, -67, 61, 68, 93, >-17, -65, -67, -17, -65, -67, 68, 94, -17, -65, >-67, 0, 70, 127, -17, -65, -67, 0, 0, 1, >63, -17, -65, -67, -17, -65, -67

The above is four "bars" of code including close, high, low, open, volume and timestmap. All of which are 4 byte Floats except Volume which is a Long with length of 8. Length described by the API as 8-bit bytes.

I'm not having issues parsing the first bit of the code (though it may not be perfect). I'm having issues though with the rest. From the API website, this is how the data is structured:

Field Type Length(8 bit bytes) Description

Symbol Count Integer 4 Number of symbols for which data is being returned. The subsequent sections are repeated this many times

REPEATING SYMBOL DATA

Symbol Length Short 2 Length of the Symbol field

Symbol String Variable The symbol for which the historical data is returned

Error Code Byte 1 0=OK, 1=ERROR

Error Length Short 2 Only returned if Error Code=1. Length of the Error string

Error Text String Variable Only returned if Error Code=1. The string describing the error

Bar Count Integer 4 # of chart bars; only if error code=0

REPEATING PRICE DATA

close Float 4 high Float 4 low Float 4 open Float 4 volume Float 4 (in 100's) timestamp Long 8 time in milliseconds from 00:00:00 UTC on January 1, 1970

END OF REPEATING PRICE DATA

Terminator Bytes 2 0xFF, 0XFF

END OF REPEATING SYMBOL DATA The PriceHistory response is in binary format represented by a symbol, number of chart

END OF REPEATING SYMBOL DATA

I'm able to extract count, length, symbol, error code and bar count. But it's parsing the bars I'm having issues. I'm not getting any data back but rather gibberish such as D]�R9.

Code is available here if you want to see how crappy I've gotten so far :) : http://pastebin.com/5eq9XPjT

  for (i=0;i<=dataArray.length;i++) {
if (i<=5) {
  symbolDetails['count'] = Utilities.newBlob(dataArray[0] + dataArray[1] + dataArray[2] + dataArray[3]).getDataAsString(); //Symbol count
  symbolDetails['length'] = Utilities.newBlob(dataArray[4] + dataArray[5]).getDataAsString(); // Length of record
  i=5; // jump ahead
} else if (i>5 && i<6+Number(symbolDetails['length'])) {
  for (j=0;j<Number(symbolDetails['length']);j++) {
    symbolDetails['symbol'][j] = dataArray[i];
    i++
  }
  i = 10;
  Logger.log(symbolDetails['symbol']);
  symbolDetails['symbol'] = Utilities.newBlob(symbolDetails['symbol']).getDataAsString();
  // partTwo is the sequence of data after the symbol. As symbol can be of varying length, this will help us know where we are.
  var partTwo = 5 + Number(symbolDetails['length']);
} else {
  // Get Error Code
  if (i == partTwo) {
    symbolDetails['error'] = Utilities.newBlob(dataArray[i]).getDataAsString();
  } else if (i>= Number(partTwo + 1) && i<=Number(partTwo+5)) {
    // Get Bar Count
    for (j=0;j<4;j++) {
      symbolDetails['bars'] =+ dataArray[i];
      i++;
    }
    symbolDetails['bars'] = Utilities.newBlob(symbolDetails['bars']).getDataAsString();
  }
}

}

Hoping someone can assist with this. Thank you very much.

It's currently very difficult to read your question. You can [format your question](http://stackoverflow.com/help/formatting) with backticks (`) and four-space margins to annotate the code in your answer. — apsillers, Jul 12 '13 at 15:22
Could You insert the whole code into the question? If it's to be useful the code should be immortalized here. — Grzegorz Piwowarek, Jul 12 '13 at 15:37
attempted edit to make it look better and added code. Code is not "required" as the issue is just transcribing part of the binary. Code as is returns the JS array {error=0.0, symbol=GOOG, count=1, bars=4, length=4, bar=[]}. It's running through the bars lines that I'm suddenly having issues. From what I can read, the Java class DataInputStream would help tremendously but I don't see a similar class for Google Apps. I'm limited to the Utilities class. — timtrice, Jul 12 '13 at 15:52

score 0 · Accepted Answer · edited May 23 '17 at 12:08

I'm not totally sure about your data, but I'm going to assume it's a series of signed 8-bit values.

First let's look at this as hex since that's much easier to deal with than signed values:

00 00 00 01 00 04 47 4f 4f 47 00 00 00 00 04 44 5d ef bf bd
52 44 5f 05 1f 44 5d 40 00 44 5d ef bf bd ef bf bd 46 ef bf
bd 34 ef bf bd 00 00 01 3f ef bf bd 63 1e 00 44 5c ef bf bd
ef bf bd 44 5e ef bf bd 00 44 5b 51 48 44 5e ef bf bd ef bf
bd 46 ef bf bd ef bf bd 57 00 00 01 3f ef bf bd ef bf bd 7a
00 44 5d ef bf bd ef bf bd 44 5e 4a ef bf bd 44 5b ef bf bd
00 44 5b ef bf bd 46 22 6b 0a 00 00 01 3f ef bf bd ef bf bd
ef bf bd 00 44 5f 5f 5c 44 5f ef bf bd 3d 44 5d ef bf bd ef
bf bd 44 5e ef bf bd 00 46 7f ef bf bd 00 00 01 3f ef bf bd

Let's go in order:

# Symbol Count = 1
00 00 00 01

# Symbol Length = 4
00 04

# Symbol = 'GOOG'
47 4f 4f 47

# Error Code = 0
00

# Error Length & Error Text are not present since Error Code == 0

# Bar Count = 4
00 00 00 04

Now to the problem area. The next value is a float. Specifically it's a float in little endian format. This question can help convert the bytes to a float. Note that your endianness is different than that in the question, so you'll need to reverse the bytes.

# Close = $887.74
44 5d ef bf

Here's some Javascript code to decode this value and print it out. It uses the previously referenced SO answer's Bytes2Float32 function.

function Bytes2Float32(bytes)
{
    var sign = (bytes & 0x80000000) ? -1 : 1;
    var exponent = ((bytes >> 23) & 0xFF) - 127;
    var significand = (bytes & ~(-1 << 23));

    if (exponent == 128) 
        return sign * ((significand) ? Number.NaN : Number.POSITIVE_INFINITY);

    if (exponent == -127) {
        if (significand == 0) return sign * 0.0;
        exponent = -126;
        significand /= (1 << 22);
    } else significand = (significand | (1 << 23)) / (1 << 23);

    return sign * significand * Math.pow(2, exponent);
}

var bytes = new Array(0x44, 0x5D, 0xEF, 0xBF);
var asInt = (bytes[3] & 0xFF) 
          | ((bytes[2] & 0xFF) << 8)  
          | ((bytes[1] & 0xFF) << 16) 
          | ((bytes[0] & 0xFF) << 24);

write("As Int: " + asInt + "\n");
var asFloat = Bytes2Float32(asInt);
write("Closing Price: " + asFloat);

Indeed this prints 887.74603 which sounds about right for the GOOG closing price.

You'll do the same for the high low, open and volume values (remember to multiple volume by 100 per the API spec). For the timestamp it's a long, so convert those 8 bytes to a long and find the correct Date API to convert the value to a Date object.

This is the result I got when attempted to write JS for Google Script: `code`[13-07-24 10:16:18:432 CDT] Closing Price: 71589359 code with : var asInt = (dataArray[17] & 0xFF) | ((dataArray[16] & 0xFF) << 8) | ((dataArray[15] & 0xFF) << 16) | ((dataArray[14] & 0xFF) << 24); var asFloat = parseFloat(asInt); Logger.log("Closing Price: " + asFloat); Not sure what the difference is why I wouldn't be getting the same result — timtrice, Jul 24 '13 at 17:02

Parsing Byte Array in Apps Script

1 Answers1