7

How should I go about compressing a very large JSON string to transmit over websockets? (Also later to store in localStorage)

It's already minified, but I need something that can do this: http://www.unit-conversion.info/texttools/compress/ (I tried poking about in the source there and couldn't figure it out)

Shaun Dreclin
  • 292
  • 1
  • 3
  • 10
  • The real question is, why would you send a very large string over websockets, and then store it in localstorage, what's the use case for this, there usually are better ways to store and access data. – adeneo Jan 17 '16 at 15:05
  • You shouldn't worry about it. Data sent over websockets will be automatically compressed with [`permessage-deflate`](https://tools.ietf.org/html/draft-ietf-hybi-permessage-compression-28) extension. – Lewis Jan 17 '16 at 15:42
  • @adeneo The data I'm transmitting over websockets and the data to store locally are not the same data, I'm just saying I will later also need to compress strings for localstorage. – Shaun Dreclin Jan 18 '16 at 02:09
  • @Tresdin any good way to find out what's actually being transmitted over a websocket connection? I'm using the [ws](https://github.com/websockets/ws) module in node and vanilla websockets in the client. – Shaun Dreclin Jan 18 '16 at 02:10
  • 6
    why is the first comment always an unimaginative 'why'? – john k Jun 13 '18 at 15:34

1 Answers1

7

You can compress any kind of binary data (strings, Buffers) with Node.js, no external deps required, using the zlib module.

You can use either gzip or deflate compression algorithms, depending on your needs.

Example

(shamelessly stolen from Node.js' website)

const zlib = require('zlib')
const input = JSON.stringify({ some: 'json-data' })

zlib.deflate(input, (err, buffer) => {
  if (err) {
    console.log('u-oh')
  }

  // Send buffer as string to client using my imaginary io object
  io.send(buffer.toString('base64'))
})

Update: It might be better to just enable HTTP compression on the transport layer instead of compressing and decompressing the data on your own.

Robert Rossmann
  • 11,931
  • 4
  • 42
  • 73
  • 2
    I'll need to decompress that on the client side, does the zlib library also exist in a way that I could include it on the client webpage? And I'll want to compress data coming from the client and heading to the server as well. – Shaun Dreclin Jan 18 '16 at 02:11
  • You might try to use [this](https://www.npmjs.com/package/gzip-js). I wonder, however - would it not make more sense to just enable gzip compression on the HTTP transport layer, instead of implementing compression at API layer? I have updated the answer with relevant note. – Robert Rossmann Jan 18 '16 at 09:11
  • Not totally sure how I'd do that, I'm using [ws](https://github.com/websockets/ws) in node and websockets in javascript, all the networking stuff is handled for me haha – Shaun Dreclin Jan 18 '16 at 12:23
  • There are extensions for websockets that you could use - see [this SO question and answer](http://stackoverflow.com/questions/19298651/how-does-websocket-compress-messages). – Robert Rossmann Jan 18 '16 at 18:25