how to decompress .tar.bz2 in memory with python

Question

How to decompress *.bz2 file in memory with python? The bz2 file comes from a csv file.

I use the code below to decompress it in memory, it works, but it brings some dirty data such as filename of the csv file and author name of it, is there any other better way to handle it?

#!/usr/bin/python
# -*- coding: utf-8 -*-
import StringIO
import bz2


with open("/app/tmp/res_test.tar.bz2", "rb") as f:
    content = f.read()

    compressedFile = StringIO.StringIO(content)
    decompressedFile = bz2.decompress(compressedFile.buf)
    compressedFile.seek(0)

    with open("/app/tmp/decompress_test", 'w') as outfile:
        outfile.write(decompressedFile)

I found this question, it is in gzip, however my data is in bz2 format, I try to do as instructed in it, but it seems that bz2 could not handle it in this way.

Edit:

No matter the answer of @metatoaster or the code above, both of them will bring some more dirty data into the final decompressed file. For example: my original data is attached below and in csv format with the name res_test.csv:

Then I cd into the directory where the file is in and compress it with tar -cjf res_test.tar.bz2 res_test.csv and get the compressed file res_test.tar.bz2, this file could simulate the bz2 data that I will get from internet and I wish to decompress it in memory without cache it into disk first, but what I get is data below and contains too much dirty data:

The data is still there, but submerged in noise, does it possible to decompress it into pure data just the same as the original data instead of decompress it and extract real data from too much noise?

metatoaster · Accepted Answer · 2017-09-19T07:15:40.213

For generic bz2 decompression, BZ2File class may be used.

from bz2 import BZ2File
with BZ2File("/app/tmp/res_test.tar.bz2") as f:
    content = f.read()

content should contain the decompressed contents of the file.

However, given that this is a tar file (an archive file that is normally extracted to disk as a directory of files), the tarfile module could be used instead, and it has extended mode flags for handling bz2. Assuming the target file contains a res_test.csv, the following can be used

tf = tarfile.open('/app/tmp/res_test.tar.bz2', 'r:bz2')
csvfile = tf.extractfile('res_test.csv').read()

The r:bz2 flag opens the tar archive in a way that makes it possible to seek backwards, which is important as the alternative method r|bz2 makes it impractical to call extract files from the members it return by extractfile. The second line simply calls extractfile to return the contents of 'res_test.csv' from the archive file as a string.

The transparent open mode ('r:*') is typically recommended, however, so if the input tar file is compressed using gzip instead no failure will be encountered.

Naturally, the tarfile module has a lower level open method which may be used on arbitrary stream objects. If the file was already opened using BZ2File already, this can also be used

with BZ2File("/app/tmp/res_test.tar.bz2") as f:
    tf = tarfile.open(fileobj=f, mode='r:')
    csvfile = tf.extractfile('res_test.csv').read()

Hi, thank you for your answer, I have tried your method, however it still contains too much noise in the final decompressed data, I have edited the question according to this test, and I just wonder could you please share some more idea? — buxizhizhoum, Sep 19 '17 at 06:48
@buxizhizhoum my bad, I didn't clue into the fact that you are really working with a tarfile. — metatoaster, Sep 19 '17 at 07:07
And one more question, does it possible to read the content if the tar.bz2 file is get with from web directly without buffer it to disk? — buxizhizhoum, Sep 19 '17 at 08:59
You can just simply use [`requests.get`](http://docs.python-requests.org/en/master/user/quickstart/) and read the `response.content` into a [`io.BytesIO`](https://docs.python.org/2/library/io.html#io.BytesIO). Example: `fileobj = BytesIO(requests.get(url).content)` and then `contents = tarfile.open(fileobj=fileobj).extractfile(filename).read()` — metatoaster, Sep 19 '17 at 09:11

how to decompress .tar.bz2 in memory with python

1 Answers1