Compressing and decompressing the Stack Overflow homepage HTML

Question

I'm trying to compress HTML with JavaScript and decompress it with Ruby. Some carachters are not being processed correctly however and I'm looking for a way to fix this.

My compression function first turns the html in to a byte array using this function. It then compresses the array with the js-deflate library. Finally, the output from that is base64 encoded using window.btoa().

var compress = function(htmlString) {
  var compressed, originalBytes;

  originalBytes = Utils.stringToByteArray(htmlString);
  compressed = RawDeflate.deflate(originalBytes.join(''));
  return window.btoa(compressed);
};

On the Ruby end of things I have a Decompression class which first base64 decodes the compressed html. It then uses the Ruby Zlib standard library to decompress the html. This process is described in this Stack Overflow question thread.

require "base64"
require "zlib"

class Decompression
  def self.decompress(string)
    decoded = Base64.decode64(string)
    inflate(decoded)
  end

private

  def self.inflate(string)
    zstream = Zlib::Inflate.new(-Zlib::MAX_WBITS)
    buf = zstream.inflate(string)
    zstream.finish
    zstream.close
    buf
  end
end

I'm using this class to inflate compressed html, which was sent to a local server, and write it to a file.

decompressed_content = Decompression.decompress(params["compressed_content"])
File.write('decompressed.html', decompressed_content)

Then I open the file in the browser to see if it looks correct.

For the most part, this works fine. I can process the Stack Overflow homepage and it comes out like this:

processed Stack Overflow homepage

You can see there is some problems though. Some characters are not coming out correctly, most notably the down arrow beside my name in the header

down arrow

and the multiplication signs in the recent tags list

recent tags list

How can I fix my code so that these parts of the page are processed properly?

I have tried to force the encoding of the inflated html to UTF-8 but it doesn't change anything.

  def self.decompress(string)
    decoded = Base64.decode64(string)
    # Forcing the encoding of the output doesn't do anything.
    inflate(decoded).force_encoding('UTF-8')
  end

  def self.decompress(string)
    decoded = Base64.decode64(string)
    # Either does forcing the encoding of the inflate input.
    inflate(decoded.force_encoding('UTF-8'))
  end

One key is that the encoding of the string seems to change to ASCII-8BIT after it is Base64 decoded:

def self.decompress(string)
  p "Before decode: #{string.encoding}"
  decoded = Base64.decode64(string)
  p "After decode: #{decoded.encoding}"
  inflated = inflate(decoded)
  p "After inflate: #{inflated.encoding}"
  inflated
end

# Before decode: UTF-8
# After decode: ASCII-8BIT
# After inflate: ASCII-8BIT

Edits

Someone asked for the method I use to get the html in the first place. I simply pull it off the page with jQuery:

$('html')[0].outerHTML

Edit to show effect of adding a Content-Type meta tag to the inflated html

I added <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> to the inflated html. And I now get question mark boxes like this (Chrome browser by the way):

question mark box .

If I inspect the source of my inflated html and compare it to the source of the actual Stack Overflow html I can see that there is a different character being used for the upside-down triangle beside my name.

Actual SO Source: ▾
Inflated Source without meta Content-Type: ¾
Inflated source with meta Content-Type: �

Can you also show us how you get the HTML in the first place? — parasietje, Oct 23 '12 at 14:53

alexandernst · Accepted Answer · 2012-10-25T19:23:06.950

2

Try adding UTF8 encoding to the HTML you get. Like in

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" charset="UTF-8" />

I suggest this because I can't actually see it in the source code (ctrl+u on this same page).

EDIT: Turns out you were missing the charset!

edited Oct 25 '12 at 19:23

answered Oct 23 '12 at 14:49

alexandernst

14,352
22
97
197

But how would my decompression function be aware of that? All it can see at the input is a compressed string. It doesn't know whether or not the string contains a meta content type tag. It doesn't even know that the string contains html. – David Tuite Oct 24 '12 at 03:59
Ah.. of course. Unfortunately it doesn't seem to fix the problem. I edited the question with some results. – David Tuite Oct 24 '12 at 07:40
You were really close but it turns out I actually needed to add a `` instead of the `http-equiv` tag. – David Tuite Oct 25 '12 at 14:53
Uh, that was really close. :) – alexandernst Oct 25 '12 at 15:30
Done :p In fact, you're OK writing the http-equiv and the content values too! – alexandernst Oct 25 '12 at 19:23

score 2 · Answer 2 · answered Oct 23 '12 at 17:37

Usually, the content type of HTML is determined by both HTTP headers and the HTML code itself. If you send over only the HTML code, the HTTP header information is lost.

Look at the HTTP headers sent over by StackOverflow:

Cache-Control:public, max-age=60
Content-Encoding:gzip
Content-Length:33200
Content-Type:text/html; charset=utf-8
Date:Tue, 23 Oct 2012 17:35:02 GMT
Expires:Tue, 23 Oct 2012 17:36:02 GMT
Last-Modified:Tue, 23 Oct 2012 17:35:02 GMT
Vary:*

As you can see, the Content-type is specified as being utf-8. If creating a file, the content-type needs to be set manually using a HTML tag in the HEAD, as @alexandernst suggests.

score 0 · Answer 3 · edited May 23 '17 at 11:55

0

alexandernst was on the right track but what I actually needed to add to the HTML output was a meta charset tag:

<meta charset="UTF-8">

Once I put that into the HTML output of the Stack Overflow homepage it looks perfect.

edited May 23 '17 at 11:55

Community

1
1

answered Oct 24 '12 at 12:21

David Tuite

22,258
25
106
176

Compressing and decompressing the Stack Overflow homepage HTML

3 Answers3