1

I thought I'd ask after hours of inconclusive research and tests:

Introduction

I'm trying to send very large arrays of Doubles from an app to a server, naturally, I want to compress this as much as possible. Specifically, these array contain CMDeviceMotion components (acceleration, x, y, z, gyroscope, etc...), but this question should apply to any large array of numbers (over 100K or a million values)

What I've tried and found by researching options

Say I have a large array of Double (There are many others) :

var CMX = CM.map({$0.userAcceleration.x})

here, CMX is of type [Double] and CM is [CMDeviceMotion]

I've tried making POST requests to my server by sending CMX in different ways, then calculating the total size after I receive it on the server :

  1. First, as a single comma separated string : {"AX":"-0.0441827848553658,-0.103976868093014,-0.117475733160973,-0.206566318869591,-0.266509801149368,-0.282151937484741,-0.260240525007248,-0.266505032777786,-0.315020948648453,-0.305839896202087,0.0255246963351965,0.0783950537443161,0.0749507397413254,0.0760494321584702,-0.0101579604670405,0.106710642576218,0.131824940443039,0.0630970001220703,0.21177926659584,0.27022996544838,0.222621202468872,0.234281644225121,0.288497060537338,0.176655143499374,0.193904414772987,0.169417425990105,0.150193274021149,0.00871349219232798,-0.0270088445395231,-0.0 ....

Size 153 Kb.

It makes sense that this is larger than sending as binary data, since a single number here is 64 bits (8 bytes), and becomes 17 bytes long (one byte per character) +1 = 18 (added a character for the comma).

With this reasoning, sending the array as binary data should be smaller.

  1. Base 64 encoding

Here, I convert the array to a Data object using NSKeyedArchiver and base 64 encode the data before sending it :

["AX":NSKeyedArchiver.archivedData(withRootObject:CM.map({$0.userAcceleration.x})).base64EncodedString()]

This made the file size 206 Kb

  1. Sending the data as a JSON array

By just sending :

["AX": CM.map({$0.userAcceleration.x})]

It turned out that this array of numbers was practically converted to a comma separated string, the size ended up being the same as in trial 1 (160Kb)

  1. Sending as Data without base 64 encoding

Doing this:

["AX":NSKeyedArchiver.archivedData(withRootObject:CM.map({$0.userAcceleration.x}))

made the application crash at runtime, so I can't send a Data object as a value in a JSON

Question

How can I send these array in a more condensed way in a JSON object ?

Note that I already have downsampling in mind, and using 32 bit floats to reduce the size.

Vadim Kotov
  • 8,084
  • 8
  • 48
  • 62
kmn
  • 2,615
  • 4
  • 18
  • 28
  • You could try and zip the data string, and send it base64 encoded, or try to get the binary representation of the Double (like here: https://stackoverflow.com/questions/26953591/how-to-convert-a-double-into-a-byte-array-in-swift ) and send it base64 encoded – Andreas Oetjen Mar 29 '18 at 11:48
  • Any compression methods would request to know the potential patterns of your data. e.g. data range, does order matter, fixed decimal number, etc. – zc246 Mar 29 '18 at 11:49
  • 1
    You may look at [Protocol Buffers](https://en.wikipedia.org/wiki/Protocol_Buffers), there's even [Apple's implementation for Swift](https://github.com/apple/swift-protobuf). – user28434'mstep Mar 29 '18 at 12:46

1 Answers1

2

Simple way would be to do this:

let data: Data = CMX.withUnsafeBufferPointer { pointer in
    return Data(buffer: pointer)
}

And you have binary buffer with all your Doubles/Floats combined.

But because HTTP is text-based protocol you will have to convert this data to base64 string:

let base64String = data.base64EncodedString()

And this base64String should be passed for AX parameter of your POST(?) HTTP request.

EDIT:

To convert it back you may use code like this:

extension Array {
    init?(data: Data) {
        // This check should be more complex, but here we just check if total byte count divides to one element size in bytes
        guard data.count % MemoryLayout<Element>.size == 0 else { return nil }

        let elementCount = data.count / MemoryLayout<Element>.size
        let buffer = UnsafeMutableBufferPointer<Element>.allocate(capacity: elementCount)
        data.copyBytes(to: buffer)

        self = buffer.map({$0})
        buffer.deallocate()
    }

   // Wrapped here code above
    var data: Data {
        return self.withUnsafeBufferPointer { pointer in
            return Data(buffer: pointer)
        }
    }
}

let converted: [Double]? = Array(data: CMX.data) // converted now should be equal to CMX
user28434'mstep
  • 6,290
  • 2
  • 20
  • 35