Getting Pool apply_async return too slow

Question

I'm trying to make a bot for IQ Option.

I already did it, but i did it one by one, like, i had to open 10 bots so i could check 10 pairs.

I've been trying all day long doing with ThreadPool, Threadings, map and starmap (i think i didn't use them as good as they can be).

The thing is: i'm checking pairs (EURUSD, EURAUD...) values of the last 100 minutes. When i do it one by one, it takes between 80 and 300ms to return each. I'm trying now to do this in a way that i could do like all the calls at the same time and get their results around the same time to their respective var.

Atm my code is like this:

from iqoptionapi.stable_api import IQ_Option
from functools import partial
from multiprocessing.pool import ThreadPool as Pool
from time import *
from datetime import datetime, timedelta
import os
import sys
import dados #my login data
import config #atm is just payoutMinimo = 0.79

parAtivo = {}

class PAR:
    def __init__(self, par, velas):
        self.par = par
        self.velas = velas
        self.lucro = 0
        self.stoploss = 50000
        self.stopgain = 50000

def verificaAbertasPayoutMinimo(API, payoutMinimo):
    status = API.get_all_open_time()
    profits = API.get_all_profit()
    abertasPayoutMinimo = []

    for x in status['turbo']:        
        if status['turbo'][x]['open'] and profits[x]['turbo'] >= payoutMinimo:
            abertasPayoutMinimo.append(x)
    return abertasPayoutMinimo

def getVelas(API, par, tempoAN, segundos, numeroVelas):
    return API.get_candles(par, tempoAN*segundos, numeroVelas, time()+50)


def logVelas(velas, par):
    global parAtivo
    parAtivo[par] = PAR(par, velas)


def verificaVelas(API, abertasPayoutMinimo, tempoAN, segundos, numeroVelas):    
    pool = Pool()
    global parAtivo
    
    for par in abertasPayoutMinimo:
        print(f"Verificando par {par}")

        pool = Pool()

        if par not in parAtivo:
            callbackFunction = partial(logVelas, par=par)
            
            pool.apply_async(
                getVelas,
                args=(API, par, tempoAN, segundos, numeroVelas),
                callback=callbackFunction
            )
    
    pool.close()
    pool.join()

def main():
    tempoAN = 1
    segundos = 60
    numeroVelas = 20
    tempoUltimaVerificacao = datetime.now() - timedelta(days=99)
    global parAtivo
    
    conectado = False

    while not conectado:
        API = IQ_Option(dados.user, dados.pwd)
        API.connect()

        if API.check_connect():
            os.system("cls")
            print("Conectado com sucesso.")
            sleep(1)
            conectado = True
        else:
            print("Erro ao conectar.")
            sleep(1)
            conectado = False
    
    API.change_balance("PRACTICE")
    
    while True:
        if API.get_balance() < 2000:
            API.reset_practice_balance()

        if datetime.now() > tempoUltimaVerificacao + timedelta(minutes=5):
            abertasPayoutMinimo = verificaAbertasPayoutMinimo(API, config.payoutMinimo)
            tempoUltimaVerificacao = datetime.now()

        verificaVelas(API, abertasPayoutMinimo, tempoAN, segundos, numeroVelas)
        
        for item in parAtivo:
            print(parAtivo[item])

        break #execute only 1 time for testing
    

if __name__ == "__main__":
    main()

@edit1: just complemented with more info, actually this is the whole code right now.

@edit2: when i print it like this:

for item in parAtivo:
    print(parAtivo[item].velas[-1]['close']

I get:

It's correct, the problem is it takes too long, like almost 3 seconds, the same as if i was doing without ThreadPool.

Please add more information on what packages you are using, and some of the more pressing objects (e.g. `API`, etc.), without revealing sensitive info. Hard to help without knowing where these functions come from. — felipe, Apr 11 '21 at 04:45
If IQ Option officially supports non-browser/non-human access to its API, it is not apparent on its website. It's entirely possible that IQ option only processes your requests serially, especially as you seem to be using a test account. So at a minimum you would need to have multiple clients to be able parallelise your requests. edit: or it might be all the processing time is on your end (perhaps the client is inefficient), in which case a ThreadPool won't help because of the Global Interpreter Lock (GIL). — Dunes, Apr 11 '21 at 09:44
@Dunes as long as i know, at the moment i could be doing a browser app. I'm just looking for coin pairs values and i could be showing them all side by side in real time. — Maicon Araújo, Apr 11 '21 at 14:35
I'm not sure I understand your comment. I was just trying to suggest some reasons why multithreading might not be helping. How have you timed your code? Is it just from the start of the program to the end of the program? I've noticed that you always call `verificaAbertasPayoutMinimo`, and then only run the while loop once (with a break). Is it maybe that `verificaAbertasPayoutMinimo` is taking most of the three seconds? — Dunes, Apr 11 '21 at 16:39
@Dunes sorry, i'm freaking out with this lol. Yes, verificaAbertasPayoutMinimo will return from the API all the pairs that are active and with the minimum payout that i set. But the func that is taking almost all of it is `verificaVelas`, when i get the values from the API for each pair on list abertasPayoutMinimo. — Maicon Araújo, Apr 11 '21 at 16:42
In that case, do you know how much data you are asking to fetch? At a vague guess it looks to be the last 20 60-second candles for 7 symbols. That shouldn't be a huge amount of data. But I am guessing about what your code is doing. So I could be wrong. You could be asking for a lot of data. Deserialising that data will be CPU intensive. Python is **really really bad** at using threads to parallelise CPU intensive tasks. If the size of the data is not the problem. Have you tried having each job create its own API client rather than share a global one? — Dunes, Apr 11 '21 at 17:00
@Dunes actually i'm just learning as i'm doing, so tbh i have no idea on how would i do it, it's also my first time working with an API — Maicon Araújo, Apr 11 '21 at 20:57

score 0 · Answer 1 · answered Apr 11 '21 at 22:07

0

Solved.

Did it using threadings.Thread, like this:

for par in abertasPayoutMinimo:        
        t = threading.Thread(
            target=getVelas,
            args=(API, par, tempoAN, segundos)
        )
        t.start()
        t.join()

answered Apr 11 '21 at 22:07

Maicon Araújo

21
2

This will run sequentially, one by one. – HTF Apr 13 '21 at 07:53
@HTF really? that's odd, because now i'm getting all the calls' results in less than 50ms – Maicon Araújo Apr 13 '21 at 12:57

Getting Pool apply_async return too slow

1 Answers1