9

Possible Duplicate:
Receiving image through websocket

Using 

imageData = context.getImageData(0, 0, width, height);
JSON.stringify(imageData.data);

I grab the pixel data, convert it to a string, and then send it over the wire via websockets. However, this string can be pretty large, depending on the size of the canvas object. I tried using the compression technique found here: JavaScript implementation of Gzip but socket.io throws the error Websocket message contains invalid character(s). Is there an effective way to compress this data so that it can be sent over websockets?

Community
  • 1
  • 1
user730569
  • 3,940
  • 9
  • 42
  • 68
  • @Esailija Maybe... is the consensus to base64 encode strings? – user730569 Jun 25 '12 at 20:11
  • How about this http://stackoverflow.com/questions/6869926/websockets-and-binary-data . Really, I'm just searching google here :D – Esailija Jun 25 '12 at 20:12
  • @Esailija yeah but none of these are really about the best way to compress a string... just about how to send images or binary data – user730569 Jun 25 '12 at 20:15
  • If you send a png/jpg image as raw binary, then that's pretty good compression already. – Esailija Jun 25 '12 at 20:16
  • @Esailija and for converting a string to raw binary? – user730569 Jun 25 '12 at 20:24
  • If you have the raw png binary as base64, you can get decode it with [`btoa`](https://developer.mozilla.org/en/DOM/window.btoa) – Esailija Jun 25 '12 at 20:25
  • @Esailija ah ok got it, but does that actually compress strings? seems like the number of characters is about the same after using `btoa` – user730569 Jun 25 '12 at 20:27
  • Don't do base64 strings or utf-8 strings, because 1) browsers supports extracting canvas as compressed JPEG or PNG image in raw binary data format (blob) 2) WebSockets support sending binary blobs. Dealing with base64 is only needed if you are dealing with legacy (in)compatibility. – Mikko Ohtamaa Jun 25 '12 at 22:05
  • @Mikko, actually the native browser support for canvas to Blob is somewhat limited right now. On the other hand, pretty much every browser that supports canvas supports the toDataURL method. Also, base64 encode/decode is pretty efficient CPU-wise. – kanaka Jun 25 '12 at 22:55

4 Answers4

7

There are several ways I would recommend depending on which axis of efficiency you are wanting (bandwidth vs CPU efficiency).

Option 1: You can use the canvas toDataURL method. This returns a base64 encoded image of the canvas image data. It will be compressed using the image format you specify (or PNG for the default) and it will be pre-encoded to base64 for sending over WebSocket.

canvas = document.getElementById("mycanvas");
b64png = canvas.toDataURL();

ws.send(b64png);

Option 2: If you can tolerate lossy compression then you can ask for the image as a base64 encoded JPEG from the toDataURL method:

canvas = document.getElementById("mycanvas");
b64jpeg = canvas.toDataURL("image/jpeg");

ws.send(b64jpeg);

Option 3: If you are using a browser that supports binary WebSocket data (Chrome, Firefox, IE 10) then you can just send the canvas arraybuffer directly over WebSocket

canvas = document.getElementById("mycanvas");
ctx = canvas.getContext('2d');
imgdata = ctx.getImageData(0,0, width, height).data; // This is a Uint8ClampedArray
ws.send(imgdata.buffer); // Send the ArrayBuffer from the Uint8ClampedArray

Option 3 will likely be the least efficient in terms of bandwidth, but the most efficient in terms of processing power on the client and server side because the image data is sent raw with little pre/post processing required.

The most bandwidth efficient option will likely be #2 but you will lose some image quality during conversion of the image data to JPEG format. You could even go further and base64 decode the data into an arraybuffer or blob and send that via binary WebSocket so that you don't get the 33% base64 bandwidth overhead, but this adds even more CPU overhead.

If you want efficient bandwidth without losing any image quality then option #2 is your best bet.

Some notes/caveats:

The toDataURL prefixes the base64 data something like this:

"data:image/png;base64,iVBORw0KGgoAAAA..."

One nice thing about the data URL format is that you can take the whole thing and paste it into your browsers address bar and the browser will render the image.

See the MDN Canvas page for more info about toDataURL.

kanaka
  • 70,845
  • 23
  • 144
  • 140
  • Better to send as binary instead of base64 encoding? – Mikko Ohtamaa Jun 25 '12 at 21:51
  • @MikkoOhtamaa, I think I addressed that in the answer. It's better in terms of bandwidth efficiency, but there is no API I am aware of that will give you a compressed image that is not base64 encoded. This means you have to manually decode it to a binary data type before sending it which will add CPU overhead to the client side. – kanaka Jun 25 '12 at 21:54
  • See my answer you to get image as binary blob – Mikko Ohtamaa Jun 25 '12 at 21:59
  • Can the down-voter please indicate what the problem is with the answer so that it can be fixed and have the downvote removed? – kanaka Jun 25 '12 at 22:40
  • How would you decode Option 1? – user730569 Jun 26 '12 at 03:37
  • I assume you mean how do you deocde the data on the server? Well, if you are using python it would be something like this: import base64; base64.b64decode(data.split(",")[1]). Which splits the data at the comma, takes the second part and base64 decodes it. That gives you the raw PNG/JPEG image. – kanaka Jun 26 '12 at 14:05
  • For Option 3 I'm getting an error: `NS_ERROR_CANNOT_CONVERT_DATA: Component returned failure code: 0x80460001 (NS_ERROR_CANNOT_CONVERT_DATA)`. It tries to send an Uint8ClampedArray (Firefox 17). Shouldn't it be an ArrayBuffer or Uint8Array instead? – dforce Nov 30 '12 at 09:44
  • @dforce, true, an older version of the spec required ArrayBuffers, but it now allows full typed arrays. I made a change to send the buffer which should work with all browsers versions that support binary data over WebSockets – kanaka Nov 30 '12 at 15:42
  • @kanaka: thanks that work fine. however i'm not able to receive it via websocket. i had success with this example [mozilla Sending_and_Receiving_Binary_Data](https://developer.mozilla.org/en-US/docs/DOM/XMLHttpRequest/Sending_and_Receiving_Binary_Data) and sending via websocket. but if i use the canvas arraybuffer it didn't work :(. it seems that i recevei an empty arraybuffer with the correct size. – dforce Dec 03 '12 at 17:24
  • [example code here](http://jsfiddle.net/xULGE/4/) – dforce Dec 03 '12 at 17:36
  • WORKS now: updates [here](http://jsfiddle.net/xULGE/9/) – dforce Dec 04 '12 at 08:46
  • @dforce: you shouldn't have to copy the canvas data manually. If you do then that indicates a browser bug. Please file a bug with the browser vendors that have this issue and then post back here for reference. – kanaka Dec 04 '12 at 15:39
4

The most bandwidth efficient way is to to send photo like data is JPEG encoded binary as blob

You can get <canvas> data as binary JPEG blob:

https://github.com/miohtama/Krusovice/blob/master/src/tools/resizer.js#L51

(For non-photo like content you can also get PNG blob)

Blob is always raw binary, no UTF-8 or base64 crap involved.

WebSocket.send() supports blobs as input:

https://developer.mozilla.org/en/WebSockets/WebSockets_reference/WebSocket

HTTP Blob sending:

https://developer.mozilla.org/en/DOM/XMLHttpRequest/Sending_and_Receiving_Binary_Data

Your mileage with different browsers may vary.

Mikko Ohtamaa
  • 82,057
  • 50
  • 264
  • 435
  • Unless you happen to be running Firefox with support for canvas.mozGetAsFile (which will likely go away) this is going to be one of the least CPU efficient methods because you have to base64 decode the result of toDataURL to an ArrayBuffer and then construct a Blob out of the arraybuffer. Also, in this case the Blob conversion is just overhead and the ArrayBuffer could be sent over the WebSocket channel directly. – kanaka Jun 25 '12 at 22:38
  • 2
    -1 for a couple reasons: JPEG is not always the most bandwidth efficient way to encode image data (often for clipart type images, PNG is far more efficient) And Blob is no more bandwidth efficient than sending ArrayBuffer over WebSockets (and less CPU efficient in this case because you are doing an unnecessary ArrayBuffer to Blob conversion under most browsers). Fix the definitive (but wrong) leading statement and I'll remove the downvote. – kanaka Jun 25 '12 at 22:50
  • It really depends on so many variables. For smaller images, this seems to be true. For larger images, my ad hoc testing shows that a base64-string can potentially compress to fewer bytes when gzipped that its' equivalent JPEG. But that, of course, depends on the server accepting gzipped http requests. – nikc.org Jun 26 '12 at 04:46
  • For photo-like image data, there is no way that base64-string would be smaller than JPEG. JPEG already does its own entropy encoding interenally. If you need lossless encoding of image data, just use PNG which may work better for graphics-like content as mentioned in the answer. – Mikko Ohtamaa Jun 26 '12 at 06:41
  • Blob is more bandwidth efficient than sending base64-strings. Also, *as mentioned in the answer*, PNG is better compression for graphics like image data. – Mikko Ohtamaa Jun 26 '12 at 06:42
  • Also new APIs for blob extraction of data are available in browsers - your mileage may vary. – Mikko Ohtamaa Jun 26 '12 at 06:43
3

I disagree with the attempts to close since you asked for a more efficient way. The least we can do is help you come up with a more efficient way.

It really depends on what you're doing though. Can you make the client do more work?

Do you really have to send all of the canvas pixel data? Can you instead send only the pixels that have changed? (Or is that nearly all of them?)

Sending only the changes back and forth would push it into a computing problem rather than a large-amount-of-data-over-the-wire problem.

Depending on your app, can you keep track of regions that have changed? If you have 2-3 small rectangles of that have changed on the canvas then that ought to be much smaller to send than the entire canvas.

As with any efficiency question its worth asking if you're doing the right thing in the first place. Do you really need to throw large amounts of pixel data over the wire? Often with canvas it is easier to recreate the scene on the server by sending over the commands that have changed the scene than it is to send over the bitmap itself. Websockets ought to be well suited to this. This could be a good solution for a lot of drawing applications and games, but again it really depends on what you're trying to accomplish here.

Simon Sarris
  • 62,212
  • 13
  • 141
  • 171
  • Within the constrains of "sending a full canvas image to the server using websocket in the most bandwidth efficient way " there is a definite answer which I have written below – Mikko Ohtamaa Jun 25 '12 at 22:03
0

I realized a way to significantly trim down the data that was being sent over the wire.

The getImageData method returns an object with a data property that is itself an object with keys as the index of the pixel and the value as the individual red, green, blue, or alpha. The keys were making the object really big, especially since a 300x300 canvas object would have 300x300x4 = 360,000 object key/value pairs.

So by extracting just the values of the colors and pushing them into an array:

function extractValues(obj){
  var array = [];
  for (var key in obj){
    array.push(obj[key]);
  }
  return array;
}

I was able to reduce the data by over 50%, which led to significant performance boosts.

user730569
  • 3,940
  • 9
  • 42
  • 68