reqHistoricalData in IBPY doesn't return anything [python]

Question

I am trying to obtain historical data from Interactive Brokers (IB) through Ibpy. I have tried several scripts for this task, which I have adapted from others who indicate that it should work. None of them work for me, however! I am new to python, so I admit that I do not have complete insight into the workings of these methods - however, I should have tried the most obvious fixes. Below I have listed two of the scripts I have tried. I'm using python 2x.

In TWS I have the following settings:

checked: Enable ActiveX and Socket Clients. unchecked: Enable DDE clients. unchecked: Read-Only API. checked: Download open orders on connection. checked: Include FX posistions when sending portfolio. checked: Send status updates for EEP. Socket port = 7496. checked: Use negative numbers to bind automatic orders. unchecked: Create API message log file. unchecked: Include market data in API log file. Logging Level = Error. Master API client ID = 222. Timeout to send bulk data to API is 30 seconds. Component Exch Separator = Blank. checked: Allow connections from localhost only.

API - Precautions checked: Bypass Order Precautions for API Orders. everything else all unchecked in this tab.

I have got TWS logged in and running when I run the python scripts and the TWS API settings above seem correct compared with what everyone else is saying online. I have a real IB account subscribed to US equity data. It should further be mentioned that I tried to run another script placing an order through IBPY as well - this worked, so the problem seems to exist only (at the moment at least) regarding obtaining the historical data.

Script 1:

from time import sleep, strftime, localtime  
from ib.ext.Contract import Contract  
from ib.opt import ibConnection, message  


new_symbolinput = ['AAPL']
newDataList = []  
dataDownload = []  

def historical_data_handler(msg):  
    global newDataList  
    print (msg.reqId, msg.date, msg.close)
    if ('finished' in str(msg.date)) == False:  
        new_symbol = new_symbolinput[msg.reqId]  
        dataStr = '%s, %s, %s' % (new_symbol, strftime("%Y-%m-%d", localtime(int(msg.date))), msg.close)  
        newDataList = newDataList + [dataStr]
    else:  
        new_symbol = new_symbolinput[msg.reqId]  
        filename = 'minutetrades' + new_symbol + '.csv'  
        csvfile = open('IBdata/' + filename,'w')
        for item in newDataList:  
            csvfile.write('{} \n'.format(item))
        csvfile.close()  
        newDataList = []  
        global dataDownload  
        dataDownload.append(new_symbol)  


con = ibConnection(port=7496, clientId=222)  
con.register(historical_data_handler, message.historicalData)  
con.connect()  

symbol_id = 0  
for i in new_symbolinput:  
    print (i)  
    qqq = Contract()  
    qqq.m_symbol = i  
    qqq.m_secType = 'STK'  
    qqq.m_exchange = 'SMART'  
    qqq.m_currency = 'USD'
    con.reqHistoricalData(symbol_id, qqq, '20161101', '1 W', '1 D', 'MIDPOINT', 1, 2)  

    symbol_id = symbol_id + 1  
    sleep(10)  

print (dataDownload) 
filename = 'downloaded_symbols.csv'  
csvfile = open('IBdata/' + filename,'w')  
for item in dataDownload:  
    csvfile.write('%s \n' % item)  
csvfile.close()

this should return the data in a csv file. The csv file is created, but it is empty.

Response:

Server Version: 76
TWS Time at connection:20170315 14:18:06 CET
AAPL
[]

So it clearly doesn't return anything.

Script 2:

from time import sleep, strftime
from ib.ext.Contract import Contract
from ib.opt import ibConnection, message

def my_account_handler(msg):
    print(msg)

def my_tick_handler(msg):
    print(msg)

def my_hist_data_handler(msg):
    print(msg)


if __name__ == '__main__':

    con = ibConnection(port=7496,clientId=222)
    con.register(my_account_handler, 'UpdateAccountValue')
    con.register(my_tick_handler, message.tickSize, message.tickPrice)
    con.register(my_hist_data_handler, message.historicalData)
    con.connect()

    print(con.isConnected())

    def inner():

        qqqq = Contract()
        qqqq.m_secType = "STK" 
        qqqq.m_symbol = "AAPL"
        qqqq.m_currency = "USD"
        qqqq.m_exchange = "SMART"
        endtime = strftime('%Y%m%d %H:%M:%S')
        print(endtime)
        print(con.reqHistoricalData(1,qqqq,endtime,"1 W","1 D","MIDPOINT",1,2))



    sleep(10)

    inner()
    sleep(5)
    print('disconnected', con.disconnect())
    print(con.isConnected())

The response here:

Server Version: 76
TWS Time at connection:20170315 14:29:53 CET
True
20170315 14:30:05
None
('disconnected', True)
False

Again nothing is returned. I have no idea why, as it seems to work for others. I might have missed something fundamental as I'm quite new to Python?

Any help is very much appreciated.

Answer 1

Always implement an error handler and the API will tell you what's wrong. In this case it says use "1 day" for a bar size.

There's no need to sleep. Use nextValidId to know when the connection is ready. Use the different end methods to know when you're done. historicalDataEnd doesn't seem to be implemented yet in IBpy so just look for 'finished'

Don't shut off api logging, it would have shown the error as well as all the different messages sent to and from TWS. You can shut off market data in the log file as it's quite a lot. Look for a file 'api.222.Wed.log' in your jts dir.

from time import sleep, strftime
from ib.ext.Contract import Contract
from ib.opt import ibConnection, message
import pandas as pd
import numpy as np

def nextValidId_handler(msg):
    print(msg)
    inner()

hist = []

def my_hist_data_handler(msg):
    print(msg)
    if "finished" in msg.date:
        print('disconnecting', con.disconnect())
        df = df = pd.DataFrame(index=np.arange(0, len(hist)), columns=('date', 'close', 'volume'))
        for index, msg in enumerate(hist):
            df.loc[index,'date':'volume'] = msg.date, msg.close, msg.volume
        print(df )
    else:
        hist.append(msg)    

def error_handler(msg):
    print(msg)

if __name__ == '__main__':

    con = ibConnection(port=7497,clientId=222)
    con.register(error_handler, message.Error)
    con.register(nextValidId_handler, message.nextValidId)
    con.register(my_hist_data_handler, message.historicalData)
    con.connect()

    print(con.isConnected())

    def inner():

        qqqq = Contract()
        qqqq.m_secType = "STK" 
        qqqq.m_symbol = "AAPL"
        qqqq.m_currency = "USD"
        qqqq.m_exchange = "SMART"
        endtime = strftime('%Y%m%d %H:%M:%S')
        print(endtime)
        con.reqHistoricalData(1,qqqq,endtime,"1 W","1 day","MIDPOINT",1,2)

    print(con.isConnected())