2

I am playing around with interactive brokers Java API to get real time market data. My stupid approach in storing the data is to write them to a file (using Java) and read the file every 30 minutes (by calling Python from within Java). This is inefficient.

I am wondering is there a way to actually avoid reading/writing to files and just "stream" the data collected within those 30 minutes and pipe them into Python. Forgive me as I do not know if "stream" is the correct word here. What is the efficient approach to this problem ? Note that I am not looking for code to call a Python program inside Java.

user1769197
  • 2,132
  • 5
  • 18
  • 32
  • This link below might be help. But I would flip it around. It might make sense to call the Java code from within a Python script and all be called from a cron job on a 30 min, or whatever interval. https://stackoverflow.com/questions/3356554/ipc-inter-process-communication-between-python-and-java – Omeri Sep 06 '21 at 04:34
  • If you're just playing around, then play around in the python API instead. Your question only makes sense if you have a large java code base already. – brian Sep 06 '21 at 14:32

1 Answers1

3

One approach that I have worked on is to use a message queue(like Kafka, Rabbit MQ, etc) to 'stream' the data between the services.

You can have a scheduler at your Java service which will produce messages containing your data every 30 minutes and on the other end, your Python program consumes the same by subscribing to the particular Topic.

Communication using Message queue

The advantage here is Message queues, especially Kafka, are designed for high-rate data aggregation. Also, you need not call the python program explicitly from your Java code.

Dhaval Simaria
  • 1,886
  • 3
  • 28
  • 36