Python Interactive brokers IB API very very slow

Question

I am trying out the new Python Interactive Broker API, but I am experiencing some serious speed issues at the very first step...

The following code (see below) times

0:00:08.832813 until the data is done being received

0:00:36.000785 until the app is fully disconnected...

Why is it so slow? What would be the best way to speed it up?

from ibapi import wrapper
from ibapi.client import EClient
from ibapi.utils import iswrapper #just for decorator
from ibapi.common import *
from ibapi.contract import *
import datetime
from datetime import timedelta


class DataApp(wrapper.EWrapper, EClient):
    def __init__(self):
        wrapper.EWrapper.__init__(self)
        EClient.__init__(self, wrapper=self)

    @iswrapper
    def historicalData(self, reqId: TickerId, date: str, open: float, high: float,
                            low: float, close: float, volume: int, barCount: int,
                            WAP: float, hasGaps: int):
        super().historicalData(reqId, date, open, high, low, close, volume,
                                barCount, WAP, hasGaps)
        print("HistoricalData. ", reqId, " Date:", date, "Open:", open,
               "High:", high, "Low:", low, "Close:", close, "Volume:", volume)

    @iswrapper
    def historicalDataEnd(self, reqId: int, start: str, end: str):
        super().historicalDataEnd(reqId, start, end)
        print("HistoricalDataEnd ", reqId, "from", start, "to", end)
        print(datetime.datetime.now()-startime)
        self.done = True # This ends the messages loop - this was not in the example code...

    def get_data(self):        
        self.connect("127.0.0.1", 4002, clientId=10)
        print("serverVersion:%s connectionTime:%s" % (self.serverVersion(),
                                                self.twsConnectionTime()))

        cont = Contract()
        cont.symbol = "ES"
        cont.secType = "FUT"
        cont.currency = "USD"
        cont.exchange = "GLOBEX"
        cont.lastTradeDateOrContractMonth = "201706"
        self.reqHistoricalData(1, cont, datetime.datetime.now().strftime("%Y%m%d %H:%M:%S"),
                               "1800 S", "30 mins", "TRADES", 0, 1, [])
        self.run()        
        self.disconnect()
        print(datetime.datetime.now()-startime)

global starttime
startime = datetime.datetime.now()
DA = DataApp()
DA.get_data()

I also tried to continually run it is the background, in order to only submit requests on the fly with

def runMe():
    app.run() # where run() has be removed from the class definition

import threading
thread = threading.Thread(target = runMe)
thread.start()

But it was also incredibly slow. Any suggestions appreciated

score 2 · Accepted Answer · answered Jun 20 '17 at 16:35

I would recommend you modify the connection socket lock within the connection class in the ibapi module. The recommendation came from heshiming on github; if you have access to the private interactive brokers repo you can access the discussion here https://github.com/InteractiveBrokers/tws-api/issues/464

I did this and it improved performance significantly.

Heshiming recommends you reduce the timeout on the socket lock object, which is called every time you send or receive a message. To modify the socket lock, Go to the site-packages folder for ibapi and modify the connect function within connection.py, changing "self.socket.settimeout(1)" to "self.socket.settimeout(0.01)". This is line 48 in connection.py for the version I have.

In case you can't see heshiming's post, I've included it at the bottom of this post.

Alternative Option: another interesting solution would be to leverage asyncio for an asynchronous event loop. I have not done this but it looks promising. See the example Ewald put together https://github.com/erdewit/tws_async

Heshiming Comment:

The implementation of Connection /ibapi/connection.py has a Lock object shared in both sendMsg and recvMsg. Since connect, self.socket.settimeout(1) is called, therefore the underlying self.socket.recv(4096) only times out once per second.

Such implementation creates a performance problem. Since the lock is shared, the socket cannot send data while receiving. In the scenario where the message received is less than 4k bytes long, the recvMsg function will wait for 1 second before releasing the lock, making subsequent sendMsg wait. In my experiment, most messages appear to be shorter than 4k bytes. In other words, this imposes a cap of one recvMsg per second.

There are couple strategies to mitigate this. One can reduce the receive buffer to a number much less than 4k, or reduce the socket timeout to something like 0.001 second to make it block less.

Or according to http://stackoverflow.com/questions/1981372/are-parallel-calls-to-send-recv-on-the-same-socket-valid , the socket itself is actually thread-safe. Thus no locks are necessary.

I tried all three strategies. Removing the lock works the best. And reducing the timeout to 0.001 works in similar ways.

I can only vouch for linux/unix platforms, and I haven't tried it on Windows. Would you consider to change the implementation to improve this?

The latest version of the api (May 2019) no longer has a lock in `recvMsg`; it's only used in `sendMsg` which seems reasonable. The timeout is still 1s but absent the lock that shouldn't cause issues considering that `sendMsg` and `recvMsg` would typically be running in different threads. — fantabolous, Oct 25 '19 at 08:40

score 0 · Answer 2 · answered Oct 25 '19 at 09:00

The current API (May 2019) has improved speed by removing the lock in recvMsg as @BenM proposed in his answer. However, it can still be slow depending on the type of request.

Removing most of their logging helped, presumably because some msg types are quite large. I initially tried filtering it out using the logging library, but flat-out removing the code worked better speed-wise, and I put this down to the extra processing required to generate the larger strings before even being passed to logging.

Replace Queue with deque: It might also be worth replacing the Queue in client.py with a collections.deque because deque is >10x faster. I've done this elsewhere for big speed ups but haven't gotten to it yet for this. deque should be thread-safe without locks the way it would be used here: "Deques support thread-safe, memory efficient appends and pops from either side of the deque" (from the docs).

Queue vs deque speed comparison

score -1 · Answer 3 · answered Jun 12 '17 at 03:52

app.run() is an infinite loop while app.done == False, but it do not stop immediately when app.done is set to True. (I don't know why).

What I have done is to write a new method instead of using app.run().

Here's my solution:

import time
from ibapi import (decoder, reader, comm)

and put this function into your Client Class.

def getMessage(self, wait=3):
    # wait 3 secs for response to come in
    time.sleep(wait)
    # get everything in app.msg_queue
    while not self.msg_queue.empty():
        text = self.msg_queue.get(block=True, timeout=0.2)
        fields = comm.read_fields(text)
        self.decoder.interpret(fields)

the usage is simple. Just use app.getMessage() instead of app.run()

This exits as soon as the queue is empty, so would require you to keep calling it in presumably an infinite loop. — fantabolous, Oct 25 '19 at 08:34

Python Interactive brokers IB API very very slow

3 Answers3