Compress data in Java and decompress in Python

Question

So I am trying to compress (gzip or similar format) a JSON object before I throw it in my MySQL database. I am currently storing the data as BLOB. I have tried to use the following Java method to compress the data:

public static byte[] compress(String str) throws Exception {
    if (str == null || str.length() == 0) {
        return null;
    }

    ByteArrayOutputStream obj = new ByteArrayOutputStream();
    GZIPOutputStream gzip = new GZIPOutputStream(obj);
    gzip.write(str.getBytes("UTF-8"));
    gzip.close();
    return obj.toByteArray();
}

and then store it in the database using setBytes() with a PreparedStatement and am having no issues with this. What I am having issues with is decrypting the data in Python 2.7 I have tried using zlib.decompress() to no avail. It can't seem to read the data that Java is storing. I also need to write a conversion script in Python to compress the old rows into this new format. So whatever format I needs to be readable by the Python decompress() whether it was compressed with Java or Python 2.7

I am happy to provide anymore information that can assist in helping to find a solution to my dilemma.

Thanks.

EDIT: Some of the Python Code:

class KitPvPMatch(Base):
    """ The SQLAlchemy declarative model class for a User object. """
    __tablename__ = 'kit_pvp_matches'
    __table_args__ = {
        'mysql_engine': 'InnoDB',
        'mysql_charset': 'utf8'
    }

    match_id = Column(INTEGER(11), autoincrement=True, primary_key=True, nullable=False)
    season = Column(Unicode(5), nullable=False)
    winner = Column(Unicode(16), nullable=False)
    loser = Column(Unicode(16), nullable=False)
    ladder_id = Column(TINYINT(4), nullable=False)
    data = Column(BLOB, nullable=False)

# The line in question
jsonData = json.loads(zlib.decompress(match.data))

# The error
error: Error -3 while decompressing data: incorrect header check

The gzip module seems to only be for files? I am using a database. — MasterGberry, Dec 06 '13 at 22:14
Ahh, right you are. Sorry. I clearly didn't read your post very carefully. — mgilson, Dec 06 '13 at 22:16

score 1 · Accepted Answer · edited May 23 '17 at 11:56

1

Here is a post that goes over unzipping using zlib with a stream.

Otherwise, have you tried the gzip docs for gzip.py. You may need a temp file. The documentation for gzip is here. There is a fairly decent solution for this approach in the following post on decompression..

If you haven't already, ensure that you are getting bytes back from SQL. Python is flexible so it may be a string. Call bytearray(string) on your string if this is the case.

If that doesn't work:

What format is the data in when returned by your SQL command?
What error, if any are you getting?

edited May 23 '17 at 11:56

Community

1
1

answered Dec 06 '13 at 22:09

Andrew Scott Evans

1,003
12
26

The gzip module seems to only be for files? I am using a database. – MasterGberry Dec 06 '13 at 22:15
you could try pumping to a temp file and then unzipping. I added some beef but without your python code (where the problem seems to be), I am a bit at a loss, – Andrew Scott Evans Dec 06 '13 at 22:19
try this post on unzipping gzipped data as well http://stackoverflow.com/questions/12571913/python-unzipping-stream-of-bytes – Andrew Scott Evans Dec 06 '13 at 22:25
i would have dropped this in the comment field fyi, but I don't have the rep yet. – Andrew Scott Evans Dec 06 '13 at 22:33
I added some of the Python code to help give some more information as to whats going on. – MasterGberry Dec 06 '13 at 23:03
ill get on it but it might be a few hours. im at work – Andrew Scott Evans Dec 06 '13 at 23:09
ever get a chance to look over this again? – MasterGberry Dec 09 '13 at 05:09
sorry, I've been busy its finals week and I work 40-60 hours as well. I can get back to it Tuesday after my last final. The other links I posted should get you pretty close to a solution. – Andrew Scott Evans Dec 09 '13 at 05:51
so sorry about not getting up code. I'm finishing a major project for work. ill edit when I can. – Andrew Scott Evans Dec 15 '13 at 06:36

Compress data in Java and decompress in Python

1 Answers1