Getting timeseries data from Tradermade with executor.map

Question

I am pulling rates from Tradermade in timeseries format and adding the timeseries object to a Rates class with some other data, the Rates class performs analysis on the timeseries data and the Rates object is then added to a dictionary with the symbol as its key.

Being relatively new to programming I followed the synchronous method of getting the data from Tradermade. It's added 66 seconds to the time - which I want to avoid.

Today I learned about using concurrent.futures and ThreadPoolExecutor.

The problem I have run into is passing the kwarg arguments to executor.map using a list of symbols as the iterator.

I have read the QnA on SO and other sites though I have not found a way to solve this problem - it's probably glaringly obvious, though I am a noob - so please be kind :)

My previous code is pasted below. I will be taking the tm.timeseries part and executing it before I add the symbols and their timeseries objects to the Rates class.

Though, I have no idea how to pass the arguments required for tm.timeseries into executor.map.

Any advice?

Note: from_date and to_datetime are the same for every symbol

    symbols = [

        'AUDUSD', 'EURUSD', 'GBPUSD', 'USDCAD', 'USDJPY', 'USDCHF', 'AUDCAD', 'AUDCHF', 'AUDJPY', 'AUDNZD', 'CADCHF', 'CADJPY',
        'CHFJPY', 'EURAUD', 'EURCHF', 'EURCAD', 'EURGBP', 'EURJPY', 'EURNZD', 'EURZAR', 'GBPAUD', 'GBPCAD', 'GBPCHF', 'GBPJPY',
        'GBPNZD', 'NZDCAD', 'NZDCHF', 'NZDJPY', 'NZDUSD', 'USDMXN', 'USDNOK', 'USDSEK', 'USDSGD', 'USDTRY', 'USDZAR'
    ]

    rates = {}
    for symbol in symbols:
        if "JPY" in symbol:
            point = 3
        else:
            point = 5
        rates[symbol] = Rates(

            symbol,
            point,
            tm.timeseries(currency=symbol, start=from_date, end=to_datetime,
                          interval="hourly", fields=["open", "high", "low", "close"])

        )

JessicaRyan · Accepted Answer · 2021-10-24T08:19:12.767

After reading the accepted answer on OS I implemented the solution and it worked, just don't use zip if you want to avoid param error. Here is the post:

Solution

I managed to bring the time down from 68 seconds to 7 seconds.

The function I used:

def get_timeseries_rates(psymbol, pstart, pend):
    
    currency=psymbol
    start_date= pstart.strftime("%Y-%m-%d-%H:%M")
    end_date= pend.strftime("%Y-%m-%d-%H:%M")
    api_key = ""
    format="split"
    interval="daily"

    df =pd.read_json('https://marketdata.tradermade.com/api/v1/timeseries?currency='
                    +currency+'&api_key='+api_key+'&start_date='+start_date+'&end_date='
                    +end_date+'&format='+format+'&interval='+interval)
    df = pd.DataFrame(df.quotes['data'], columns=df.quotes['columns'])
    
    return df

executor.map using repeat (I did not use zip as it threw an error for expected params)

rates_timeseries = []

with concurrent.futures.ThreadPoolExecutor() as executor:
    results = executor.map(get_timeseries_rates, symbols, repeat(
        from_date), repeat(to_datetime))

    for result in results:
        rates_timeseries.append(result)

Getting timeseries data from Tradermade with executor.map

1 Answers1