2

I'm trying to uncompress a file from www.dukascopy.com using lzma (files seems to be binary, it's in their .bi5 format). It's forex data for the EUR/USD pair. Excuse me, but I'm not very familiar with how to read binary files. Did I decompress it right? If so, how should I convert my resulting data that I have decompressed.

Test file: https://dl.dropboxusercontent.com/u/11283578/00h_ticks.bi5

My test code:

import lzma

with open('/Users/me/Downloads/dukascopy_php_scripts_v0.27/EURUSD/2008/05/16/00h_ticks.bi5','rb') as f:
    infile = f.read()

decompressor = lzma.LZMADecompressor()
print(lzma.decompress(infile))

Which prints:

(starting with).....b'\x00\x00\x16\xd4\x00\x02X\xf6\x00\x02X\xe6@\x19\x99\x9a?\xe6ff\x00\x00 (\x00\x02X\xf7\x00\x02X\xe3A\x10\x00\x00@\x19\x99\x9a\x00\x00)c\x00\x02X\xed\x00\x02X\xe8@\xc0\x00\x00@\x9333\x00\x00.\x87\x00\x02X\xe9\x00\x02X\xe4@\x99\x99\x9a@\xc0\x00\x00\x00\x00.\xfa\x00\x02X\xe8\x00\x02X\xe3A\x10\x00\x00@\x19\x99\x9a\x00\x000N\x00\x02X\xe8\x00\x02X\xe3Aq\x99\x9a@\x99\x99\x9a\x00\x000\xc0\x00\x02X\xee\x00\x02X\xe4@\xc0\x00\x00?\xe6ff\x00\x001<\x00\x02X\xf4\x00\x02X\xeaA\x14\xcc\xcd@\x19\x99\x9a\x00\x001\xb3\x00\x02X\xf9\x00\x02X\xef@\xac\xcc\xcd@\xe6ff\x00\x002&\x00\x02X\xfb\x00\x02X\xf6@\x99\x99\x9a@\x19\x99\x9a\x00\x00T\xd7\x00\x02Y\x03\x00\x02X\xef@\x19\x99\x9a?\x99\x99\x9a\x00\x00U\x1b\x00\x02X\xfa\x00\x02X\xf5A\x14\xcc\xcd?\x80\x00\x00\x00\x00U?\x00\x02X\xf6\x00\x02X\xf1@\x99\x99\x9aAS33\x00\x00Uc\x00\x02X\xf3\x00\x02X\xe9@\x99\x99\x9a@\xac\xcc\xcd\x00\x00U\xef\x00\x02X\xe8\x00\x02X\xde@\x86ffA#33\x00\x00V\x13\x00\x02X\xdd\ ......(ending with)..... \x00\x02Y+\x00\x02Y!?\x80\x00\x00@\x99\x99\x9a'

Emmanuel Mudiay
  • 123
  • 6
  • possible duplicate of [Reading data from Dukascopy tick binary file](http://stackoverflow.com/questions/14035808/reading-data-from-dukascopy-tick-binary-file) – Peter Wood May 22 '15 at 06:22
  • @PeterWood No, that's in C and doesn't talk about the decompression at all. – unwind May 22 '15 at 06:35
  • Sorry, I was wearing my C++ hat. It should give you some ideas about the format though. You can use [`struct`](https://docs.python.org/2/library/struct.html) to extract the values. – Peter Wood May 22 '15 at 06:38

1 Answers1

1

You can iteratively decompress the file without reading it entirely into memory like this:

Example:

#!/usr/bin/env python

URL = "https://dl.dropboxusercontent.com/u/11283578/00h_ticks.bi5"

from requests import get
from lzma import LZMADecompressor

decompressor = LZMADecompressor()
response = get(URL, stream=True)

with open("foo.bin", "wb") as f:
    for chunk in response.iter_content(chunk_size=1024):
        if chunk:
            result = decompressor.decompress(chunk)
            if result != b"":
                f.write(result)

We can then confirm and verify the output by comparing a download of the file and decompression with the lzma command line tool:

Output:

$ python foo.py

$ curl -q -o - https://dl.dropboxusercontent.com/u/11283578/00h_ticks.bi5 | lzma -d - > test.bin
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11600  100 11600    0     0   7509      0  0:00:01  0:00:01 --:--:--  7508

$ shasum foo.bin
c36d2314e43f3c71ad85e75e748b4f54edc8d74a  foo.bin

$ shasum test.bin
c36d2314e43f3c71ad85e75e748b4f54edc8d74a  test.bin

Since I don't have local access to the same file as you nor do I have the same environment the above example downloads the file you put on Dropbox and decompresses it iteratively with Python 3.4 and the lzma.LZMADecompressor library/class.

James Mills
  • 18,669
  • 3
  • 49
  • 62