How to detect zstd compression?

Question

I am currently working on a python application, that works with facebook api's. As we all know, facebook loves their own technology and is working with zstd for data compression.

The problem: facebook is returning either a uncompressed response with normal json or if the response is longer, it is responding with a zstd compressed json.

My current code is something like this:

import zstd
import json


def handle_response(response)
    json = None
    try:
        json = json.loads(zstd.decompress(response.content))
    except:
        json = json.loads(response.text)

    return json

I am currently wondering, if there is a more clean way to do this and even detect zstd.

What you have would be "cleaner" if you replace the bare `except` with some specific type of exception. — martineau, Aug 03 '21 at 14:49
Compressed data will have a [file signature](https://en.wikipedia.org/wiki/List_of_file_signatures), sometimes called magic number that identifies it. There is a [document that describes the format](https://github.com/facebook/zstd/blob/master/doc/zstd_compression_format.md) and gives the `zstd` magic number. There are [ways to check the magic number in python](https://stackoverflow.com/q/10937350/3279716). — Alex, Aug 03 '21 at 14:53

score 6 · Accepted Answer · answered Aug 03 '21 at 15:25

6

What you're doing is fine.

You could, I suppose, check to see if the stream starts with the four bytes 28 b5 2f fd. If it doesn't, it's not a zstd stream. If it does, it may be a zstd stream. In the latter case, you would try to decompress and if it fails, you would fall back to just copying the input.

That turns out to be exactly the same as what you're already doing, since the first thing that zstd.decompress is doing is to look for that signature.

answered Aug 03 '21 at 15:25

Mark Adler

101,978
13
118
158

So basically: `request.content.startswith(b"\x28\xb5\x2f\xfd")` thanks! :) – mynameisgod Aug 03 '21 at 22:10
Or just do what you put in the question. – Mark Adler Aug 03 '21 at 23:55

James Wang · Answer 2 · 2023-03-09T09:45:06.763

After excute python clause 'b = zstd.compress(str)',the byte array always starts with 'b'(\xb5/\xfd \x03\x19\x00' . So you can use 'index()' function to judge.

------------------
Does anyone know how to decompress the content which is the http response output type 'applicaton/zstd' when using internet browser? like below:

ache-control: no-store
cf-cache-status: DYNAMIC
cf-ray: 7a51c7dbad5efb2c-SJC
content-length: 1591
content-type: application/zstd
date: Thu, 09 Mar 2023 07:59:49 GMT
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} report-to: {"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v3?s=K3U4RKcx6XOP1ekNoJGTilOIZ%2FR4f43q%2BBsvbmmbEkQFHWMJQ5JvhDFDbZFHqVczdsR0rzY24pO9h4kjeehrn3fs0H76%2FO2F612s%2F7%2FjQ%2F6LRjYIb%2BOsPsFOEzIzJTU0NwSyUEmTPW0%3D"}],"group":"cf-nel","max_age":604800}
server: cloudflare

How to detect zstd compression?

2 Answers2

Linked