I'm trying to read Dukascopy Data from the uncompressed data formatted (already un-lzma-ed XXh_ticks.bi5) with plain Java and I don't find the right way to do. Is there an example on that topic?
AFAIK the order of the fields is:
- milliseconds since epoch
- Ask price
- Bid price
- Ask volume
- Bid volume
Unfortunately I have no clue which JAVA data types and which byte order I would be looking at.
I'd be grateful for any help which you could provide me with.
==EDIT (from the comments for better readability)
int size_milliseconds = Integer.BYTES;
int size_ask = Integer.BYTES;
int size_bid = Integer.BYTES;
int size_ask_volume = Float.BYTES;
int size_bid_volume = Float.BYTES;
int buffersize = size_milliseconds + size_ask + size_bid + size_ask_volume + size_bid_volume;
==EDIT (first attempt ... result looking good)
String fileName = "11h_ticks.bi5";
try {
int size_milliseconds = Integer.BYTES;
int size_ask = Integer.BYTES;
int size_bid = Integer.BYTES;
int size_askv = Float.BYTES;
int size_bidv = Float.BYTES;
int buffersize = size_milliseconds + size_ask + size_bid + size_askv + size_bidv;
int nRead = 0;
double PriceFactor = 100000.00000;
FileInputStream inputStream = new FileInputStream(fileName);
byte[] buffer = new byte[buffersize];
ByteBuffer bb = ByteBuffer.wrap(buffer);
bb.order(ByteOrder.BIG_ENDIAN);
while ((nRead = inputStream.read(buffer)) != -1) {
bb.rewind();
int millisec = bb.getInt();
int ask = bb.getInt();
int bid = bb.getInt();
float askv = bb.getFloat();
float bidv = bb.getFloat();
double dask = ask / PriceFactor;
double dbid = bid / PriceFactor;
String rowstring="Millisec=" + (String)Integer.toString(millisec) + ", ask=" + (String)Double.toString(dask) + ", bid=" + (String)Double.toString(dbid) + ", ask_vol=" + (String)Float.toString(askv) + ", bid_vol=" + (String)Float.toString(bidv);
System.out.println(rowstring);
}
inputStream.close();
} catch (FileNotFoundException ex) {
System.out.println(
"Unable to open file '" +
fileName + "'");
} catch (IOException ex) {
System.out.println(
"Error reading file '" + fileName + "'");
}
with the following result:
Millisec=269, ask=1.11645, bid=1.11643, ask_vol=1.87, bid_vol=3.0
Millisec=320, ask=1.11645, bid=1.11643, ask_vol=2.25, bid_vol=3.0
Millisec=574, ask=1.11645, bid=1.11643, ask_vol=3.0, bid_vol=3.0
Millisec=777, ask=1.11645, bid=1.11643, ask_vol=3.0, bid_vol=2.25
Millisec=942, ask=1.11645, bid=1.11643, ask_vol=3.0, bid_vol=3.0
Millisec=1634, ask=1.11645, bid=1.11643, ask_vol=2.25, bid_vol=3.0
Millisec=1685, ask=1.11646, bid=1.11644, ask_vol=4.95, bid_vol=4.0
Millisec=1736, ask=1.11646, bid=1.11645, ask_vol=2.32, bid_vol=1.0
Millisec=1787, ask=1.11647, bid=1.11644, ask_vol=4.87, bid_vol=3.0
Millisec=1838, ask=1.11647, bid=1.11644, ask_vol=2.62, bid_vol=4.12
Millisec=1889, ask=1.11647, bid=1.11644, ask_vol=2.25, bid_vol=4.5
Millisec=1940, ask=1.11647, bid=1.11644, ask_vol=1.12, bid_vol=4.5
Millisec=2004, ask=1.11648, bid=1.11644, ask_vol=6.4, bid_vol=5.25
Millisec=2055, ask=1.11648, bid=1.11644, ask_vol=7.15, bid_vol=5.25
Millisec=2157, ask=1.11648, bid=1.11644, ask_vol=6.55, bid_vol=5.25
Millisec=2209, ask=1.11648, bid=1.11644, ask_vol=9.4, bid_vol=5.25
Millisec=2260, ask=1.11647, bid=1.11644, ask_vol=5.62, bid_vol=5.25
So far, so good. The prices for July 1st 2016, 11:00 AM seem to be correct. But here's my questions:
A. The volume, I cannot judge whether it is good or not. Or doesn't it matter and only the relative size matters? Any advise on that?
B. How about the ByteOrder? Is that correct? Using LITTLE_ENDIAN (as advised in different forum entries for dukascopy data) only produces garbled output.
C. The above output is for EURUSD. If I run that on USA30IDXUSD then the output is not usable. What do I have to change? Output for USA30IDXUSD:
Millisec=158, ask=184.52136, bid=184.50411, ask_vol=5.5E-5, bid_vol=2.5E-4
Millisec=377, ask=184.52506, bid=184.50381, ask_vol=5.5E-5, bid_vol=2.5E-4
Millisec=842, ask=184.52342, bid=184.50336, ask_vol=5.5E-5, bid_vol=2.5E-4
Millisec=2139, ask=184.52328, bid=184.50465, ask_vol=5.5E-5, bid_vol=2.5E-4
Millisec=2636, ask=184.51599, bid=184.50234, ask_vol=3.0E-6, bid_vol=9.4E-5
Millisec=3129, ask=184.5154, bid=184.50316, ask_vol=3.0E-6, bid_vol=9.4E-5
Millisec=3349, ask=184.51607, bid=184.49866, ask_vol=3.0E-6, bid_vol=9.4E-5
Millisec=3679, ask=184.51599, bid=184.49944, ask_vol=3.0E-6, bid_vol=9.4E-5
Millisec=4009, ask=184.5155, bid=184.4998, ask_vol=3.0E-6, bid_vol=9.4E-5
Millisec=4449, ask=184.51532, bid=184.50248, ask_vol=3.0E-6, bid_vol=9.4E-5
Millisec=4779, ask=184.51502, bid=184.50385, ask_vol=3.0E-6, bid_vol=9.4E-5
Millisec=5267, ask=184.51586, bid=184.50095, ask_vol=3.0E-6, bid_vol=9.4E-5
Millisec=5659, ask=184.51569, bid=184.50094, ask_vol=3.0E-6, bid_vol=9.4E-5
Millisec=6099, ask=184.51579, bid=184.50268, ask_vol=3.0E-6, bid_vol=9.4E-5
According to my broker's history, the Dow Jones index was not over 18000 on that day (O:17882.6, H:17997.1, L:17857.1, C:17963.1), so using a different PriceFactor fot that symbol wouldn't be a solution. And what about volume with USA30IDXUSD? I think I'm damn close ... ;-)
