3

I am working on a plugin for an existing web-based tool in JavaScript. We are collecting data and store it in a string like this: "2.545,3.552,8.568;2.553,9.898,6.542;..." and so on.

The problem is, that we reach an export limit of 64k characters too fast. I wanted to ask - and excuse me if it is a dumb question - if someone knows of an algorithm or method we could use to compress the string before we export it. I am sure that it is technically possible but it certainly exceeds my skills as a programmer.

Thanks for any tips, link or suggestion.

HenryChinaski
  • 137
  • 1
  • 8
  • 1
    Possible duplicate of [How to compress a string?](https://stackoverflow.com/questions/3640357/how-to-compress-a-string) – Markus Safar Nov 15 '18 at 00:05
  • I might should have mentioned that I am an amateur programmer, not familiar with web programming in general. As I said, sorry if it is a dumb question. @MarkusSafar Thanks for the link. I actually found this when I googled the question but I wasn't sure if it is really the same one. The OP asked something about a "SVG path String" and I am unfamiliar with that term. Maybe I should ask the question somewhere more appropriate for noobs? I dont really understand how I would implement such a solution. – HenryChinaski Nov 15 '18 at 00:12
  • @HenryChinaski, _SVG path String_ just means that the OP has a _string_ which contains _path data_ from the _SVG_ format. However the problem seems to be the same as your goal is to _compress data_ which is stored in a _string_. Therefore you need some kind of compression algorithm and the linked question contains a bunch of answers pointed to them. So I hope this helps you ;-) – Markus Safar Nov 15 '18 at 00:16
  • Thanks @MarkusSafar! The Huffmann JavaScript Compression looks promising, but even the author states that JavaScript based compression is very slow in general. If I try it on his web page the site just freezes. So maybe someone can tell me if this is a bad idea in general? – HenryChinaski Nov 15 '18 at 00:17
  • @HenryChinaski you can send the string to a Web Worker and let it compress it there. Then give it back to the main thread. That's only if you exclusively need to compress it in JS. – Abana Clara Nov 15 '18 at 00:20
  • Rather than attempting to compress, could you instead batch process your requests? E.g. reach the 64K limit, send that much, then keep going into another? – Luke Joshua Park Nov 15 '18 at 00:20
  • In my opinion that depends totally on the use case. Where does the data come from, where does the data go and how important is it to you... I would suggest compressing the data on the server because everything you do in the client can be easily terminated by "just closing the browser" – Markus Safar Nov 15 '18 at 00:21
  • Thanks for the tip @AbanaClara. I believe the problem with this solution is that I have no control over the client-server since the plugin is used in a web tool by another company. The company allows you to modify the JavaScript code but I don't believe that I can send/request data from another server, or am I wrong? – HenryChinaski Nov 15 '18 at 00:23
  • Thanks @LukeJoshuaPark. Thats a good suggestion and thats how we deal with it at the moment. The problem is, that the user needs to export the data in the final step by himself and to have to do this with douzen of strings seems not to be very user friendly. – HenryChinaski Nov 15 '18 at 00:25
  • @HenryChinaski Look up Web Worker. It isn't a server or anything. It's a multi-threading API accessible in JS and built in most if not all modern browsers. You can compress the string there so the page won't freeze – Abana Clara Nov 15 '18 at 00:26
  • @MarkusSafar The problem is that we dont own the server the client is connected to. In the final step he exports the data by himself and thats how we get it. I already googled if it is possible to use JavaScript to even send the data to our servers directly from the client, but it doesn't seem to be possible out of security reasons? – HenryChinaski Nov 15 '18 at 00:27
  • @AbanaClara Thank you very much for the suggestion. Will definitely look into that! – HenryChinaski Nov 15 '18 at 00:28
  • @HenryChinaski `I already googled if it is possible to use JavaScript to even send the data to our servers directly from the client.` <-- it's called http request or ajax, and it's definitely possible. Otherwise all web products are just static contact me about us websites. – Abana Clara Nov 15 '18 at 00:29
  • @HenryChinaski `that the user needs to export the data in the final step by himself and to have to do this with douzen of strings seems not to be very user friendly` <-- Also you can still slice and dice your requests at the final step. If you can't send the whole load of data in a single string then batch them and send one by one. You can easily create a logic that will do this in a single click – Abana Clara Nov 15 '18 at 00:34
  • @HenryChinaski, I see so you need a way to do the workload within the browser but you don't want it to freeze of course. In this case you could either fiddle with a bunch of callbacks and make sure you split the workload up into small peaces at a time (which will be pretty complicated) or you take into consideration to use [Web Workers](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers) which should allow you to do you heavy calculations async. – Markus Safar Nov 15 '18 at 00:35
  • @AbanaClara Just so I don't misunderstand you: We are implementing a JavaScript on a domain we don't own and where people might not want us to send data from to our own servers. I thought that this shouldn't be possible. Isn't that a huge security risk? – HenryChinaski Nov 15 '18 at 00:36
  • @MarkusSafar Thank you very much! All of your really pointed me in the right direction. – HenryChinaski Nov 15 '18 at 00:37
  • @HenryChinaski Ah yes that's taboo without them knowing. Not sure if that's possible either. Have not worked in something with that pattern. – Abana Clara Nov 15 '18 at 00:37
  • 1
    @AbanaClara Thanks anyway for the hint with web workers! – HenryChinaski Nov 15 '18 at 00:39
  • Compression works, but if you also use a base 255 number system, or even hexidecimal to store those numbers, It can help to reduce memory overhead. – Eric Petersen Apr 29 '20 at 10:01

2 Answers2

4

lz-string looks like it will work.

var string = "2.545,3.552,8.568;2.553,9.898,6.542";
alert("Size of sample is: " + string.length);
var compressed = LZString.compress(string);
alert("Size of compressed sample is: " + compressed.length);
string = LZString.decompress(compressed);
alert("Sample is: " + string);
<script src="https://cdn.jsdelivr.net/gh/pieroxy/lz-string/libs/lz-string.js"></script>
Will
  • 3,201
  • 1
  • 19
  • 17
  • Thank you so much @Will. Really helping me out with your snippet! I am used to C# code in stand-alone applications and I have a hard time to understand trivial things in web developing. – HenryChinaski Nov 15 '18 at 00:38
  • @HenryChinaski I forgot to ask. If you have no way of modifying the server code. How are you going to decompress this string once they receive it? Your client will be handling a string not formatted in a way they expect it to be – Abana Clara Nov 15 '18 at 00:40
  • @AbanaClara they send the text file to us, we decompress it and send them the results after we worked with it. – HenryChinaski Nov 15 '18 at 00:42
  • @HenryChinaski ah great. You shouldn't have problems then. Good luck – Abana Clara Nov 15 '18 at 00:43
  • @AbanaClara Perfect! Thanks! – HenryChinaski Nov 15 '18 at 00:44
  • If you want to store this data in localStorage - make sure you have `compressToUTF16()` and `decompressFromUTF16()` for cross-browser compatibility. `compress`/`decompress` works only for webkit localStorage. – 350D Sep 13 '20 at 08:51
0

I would suggest you use a dedicated Web Worker for this. There is a pretty good explanation on Using Web Worker on the mozilla developers page.

Markus Safar
  • 6,324
  • 5
  • 28
  • 44
  • Thank you very much for your help! – HenryChinaski Nov 15 '18 at 00:40
  • @HenryChinaski, you are very welcome. If you need any further assistance, just let me know ;-) – Markus Safar Nov 15 '18 at 00:41
  • @HenryChinaski be mindful that a web worker will not have access to the DOM, window, or any library. You need to copy paste the source code of your compression library inside the web worker script, or you could conceptually send the function string and evaluate inside the worker. Web workers accepts and sends nothing but strings. I'm not sure how multi-threading works in C# or Java but this is how it currently works for JS – Abana Clara Nov 15 '18 at 00:47
  • @AbanaClara, actually if you are used to C#, JS is a pain in the ass :D – Markus Safar Nov 15 '18 at 00:50
  • @MarkusSafar It's the same for me when I code in C#. C# is a PITA for me. I'm only recently starting to get a hang of it. – Abana Clara Nov 15 '18 at 00:51
  • @AbanaClara, hahaha - I can imagine ;-) If you need any hints on C#, just let me know ;-) – Markus Safar Nov 15 '18 at 00:52
  • 1
    @AbanaClara Thanks for the tip. I will keep it in mind and just try a bit for a day or two. Actually, JS is my smaller problem. My problems are all the seemingly minor things you need to pay attention to when you are using databases, servers, html implementation and so on. I believe that most of it is really trivial stuff for experienced web developers but in stand-alone applications, there's just you and the code. You can test everything easily in your safe space. – HenryChinaski Nov 15 '18 at 00:59
  • @HenryChinaski you'll be able to get a hang of web workers for like an hour or two, 30 minutes even. Pretty easy API to use and potentially game changing if you need to do some really heavy calculations – Abana Clara Nov 15 '18 at 01:01
  • 2
    @AbanaClara Worked perfectly! Thanks! – HenryChinaski Nov 15 '18 at 01:57