1

I took the below code from one of the answered queries in Stackoverflow (unfortunately i cannot give full credit as i cannot locate the page anymore). I changed it a bit to fit my purpose. I want to extract historical Reuters data (fundamentalData) for a list of tickers. The below code works fine but it only grabs the last ticker data. I know i need to build a while loop but i tried many times and none worked. I'm sure this is a quick fix but since I am new at coding and python in general I just can't find the solution. Any help would be appreciated!

#Import all libriaries 
from ib.opt import ibConnection, message
from time import sleep
import lxml.etree
from bs4 import BeautifulSoup
import pandas as pd
import numpy as np
from IPython.core.debugger import set_trace
from ibapi import wrapper
from ibapi.client import EClient
from ibapi.common import *  
from ibapi.contract import Contract 

#upload excel file of list of company tickers you want to review
us_list= pd.read_excel(r'C:\TWS API\Request Reuters data\Quant Project IB TWS_test.xlsx', engine='openpyxl')

stocksList = us_list[['TICKER']]
stocksList

def fundamentalData_handler(msg):
    global imported
    imported = msg.data

def error_handler(msg):
    pass

# processing of the lines in financial statements
def extractVauleFromLineItem(fiscalperiod, code): 
    stage1 = fiscalperiod.find(name='lineitem', coacode=code)
    if (not stage1 is None):
        stage2 = stage1.get_text()
        if (stage2 == stage2):
            stage3 = float(stage2)
            if (stage3 == stage3):
                return (stage3)
            else:
                return (0.0)
                        
result = pd.DataFrame(columns =['Year', 'Ticker','control','TotalRevenue', 'GrossProfit', 'CommonSharesOutstanding','DilutedNormalizedEPS', 'totalCash', 'TotalDebt','Dividends'])

outcomes = []

for i, row in stocksList.iterrows():
    contract = Contract()
    contract.symbol = row['TICKER']
    contract.secType = "STK"
    contract.currency = "USD"
    contract.exchange = "SMART"


    tws = ibConnection("127.0.0.1",port=7497, clientId=901)
    tws.register(error_handler, message.Error)
    tws.register(fundamentalData_handler, message.fundamentalData)
    tws.connect()
    tws.reqFundamentalData(1,contract,'ReportsFinStatements')
    sleep(1)
    tws.disconnect()
    print(contract.symbol) 
    
    soup = BeautifulSoup(imported) # library for processing of the obtained XML data
    data = []
    print(soup.find(name='issueid', type="Ticker").get_text())
    print(soup.find(name='coid', type="CompanyName").get_text()) 
    
    # I found that IB API is not very stable. 
    # Sometimes it returns data of the wrong company. 
    # So the control is important
    print('Control -',contract.symbol == soup.find(name='issueid', type="Ticker").get_text())
    print()
    
    for fiscalperiod in soup.find_all(name="fiscalperiod", type="Annual"):
        year = fiscalperiod['fiscalyear']
        TotalRevenue = extractVauleFromLineItem(fiscalperiod, 'RTLR')
        GrossProfit = extractVauleFromLineItem(fiscalperiod, 'SGRP')
        CommonSharesOutstanding = extractVauleFromLineItem(fiscalperiod, 'QTCO')
        DilutedNormalizedEPS = extractVauleFromLineItem(fiscalperiod, 'VDES')
        totalCash = extractVauleFromLineItem(fiscalperiod, 'OTLO') 
        TotalDebt = extractVauleFromLineItem(fiscalperiod, 'STLD')
        Dividends = extractVauleFromLineItem(fiscalperiod, 'FCDP')
        
    
        thisYearData = (year,contract.symbol, (contract.symbol == soup.find(name='issueid', type="Ticker").get_text()),TotalRevenue , GrossProfit, CommonSharesOutstanding, totalCash, TotalDebt, Dividends)    
        data.append(thisYearData)
        
    df_data = pd.DataFrame(data, columns =['Year','control','TotalRevenue', 'GrossProfit', 'CommonSharesOutstanding','DilutedNormalizedEPS', 'totalCash', 'TotalDebt','Dividends'])

    df_data = df_data.sort_values(by=['Year'])
Carol
  • 11
  • 2
  • FYI, there is a new api http://interactivebrokers.github.io/ IbPy isn't updated anymore. However, to find your errors don't just pass in the error callback and turn on api logging in TWS. – brian Jan 23 '21 at 16:52
  • Thanks Brian, i have changed the code and updated the libraries. Obviously i still have the same issue. Not all the tickers are in the dataframe, only the last one. – Carol Jan 24 '21 at 08:29
  • You didn't paste the new code so I still can't run it but your problem is `data = []` erases your data for every row and then `df_data = pd.DataFrame(data`... makes a new df with the new data and overwrites the old df. – brian Jan 24 '21 at 12:32
  • thanks Brian, i'll take a look and amend the code accordingly. – Carol Jan 24 '21 at 16:42
  • Your edit has just added imports without changing code. There are 2 examples I found. https://stackoverflow.com/q/49329139/2855515 and https://stackoverflow.com/a/58918593/2855515 – brian Jan 24 '21 at 20:07
  • thanks Brian, i'm working on it based on the above questions answered but i keep getting an error when i run the code: NameError: name 'imported' is not defined. the 'imported' is defined so i'm not sure why i'm getting this error. – Carol Feb 05 '21 at 09:44
  • You are using the global variable 'imported' but it is not defined anywhere in the global scope. – brian Feb 05 '21 at 12:40

0 Answers0