Stack Overflow
Stack Overflow

Questions tagged [starmap]

46 questions
41
votes
2 answers

python struct.error: 'i' format requires -2147483648 <= number <= 2147483647

Problem I'm willing to do a feature engineering using multiprocessing module (multiprocessing.Pool.starmap(). However, it gives an error message as follows. I guess this error message is about the size of inputs (2147483647 = 2^31 − 1?), since the…
SUNDONG
  • 2,501
  • 5
  • 21
  • 37
12
votes
2 answers

How to use boto3 client with Python multiprocessing?

Code looks something like this: import multiprocessing as mp from functools import partial import boto3 import numpy as np s3 = boto3.client('s3') def _something(**kwargs): # Some mixed integer programming stuff related to the variable…
RNHTTR
  • 2,235
  • 2
  • 15
  • 30
11
votes
1 answer

How to get result from Pool.starmap_async()?

I have program which computes the index of array*value and returns a string. I use .starmap_async() because I must pass two arguments to my async function. The program looks as follows: import multiprocessing as mp from multiprocessing import…
ElConrado
  • 1,477
  • 4
  • 20
  • 46
7
votes
3 answers

Common Lisp equivalent of Python's itertools.starmap?

Python's Itertools has what is called starmap. Given a collection of collections and a function, it applies the function to each collection strictly inside the collection, using the elements of said internal collection as arguments to the function.…
J. Mini
  • 1,868
  • 1
  • 9
  • 38
4
votes
1 answer

Terminate all processes when condition is met

I am using starmap to run testing function. Whats the best/safest way to terminate all processes when a process first finds the permutation [5,2,4,3,1]? import multiprocessing as mp import time def testing(lts): # code .... start_time =…
Panos Kalatzantonakis
  • 12,525
  • 8
  • 64
  • 85
3
votes
2 answers

Python - Multiprocessing StarMap with two arguments

I've a function that runs multiple queries in parallel but I'm having some troubles to run my function using multprocessing with more than argument. I've this code: def run(args): query, cursor = args cursor.execute(query) with…
Pedro Alves
  • 1,004
  • 1
  • 21
  • 47
3
votes
0 answers

multiprocessing starmap AttributeError: 'NoneType' object has no attribute 'write'

My parallelized script was working for a while. I used the multiprocessing library and the function starmap for it. But now when I'm trying to run it the following AttributeErroris displayed: Python 3.6.5 | packaged by conda-forge | (default, Apr 6…
Robo
  • 51
  • 3
2
votes
2 answers

Python - multiprocessing a nested dictionary

I have tried using this question to answer my problem, but I haven't had any success. I'm using Python 3.10. My dictionary is structured like this (where each list of string is a review of the product): {storeNameA : {productA : 0 [string, string,…
Kav
  • 25
  • 4
2
votes
1 answer

Python Pool.Starmap Not Terminating Or Outputting on Print

I have attempted in a few different ways to perform Pool.starmap. I have tried various different suggestions and answers, and to no avail. Below is a sample of the code I am trying to run, however it gets caught and never terminates. What am I doing…
rly
  • 21
  • 2
2
votes
1 answer

Getting Pool apply_async return too slow

I'm trying to make a bot for IQ Option. I already did it, but i did it one by one, like, i had to open 10 bots so i could check 10 pairs. I've been trying all day long doing with ThreadPool, Threadings, map and starmap (i think i didn't use them as…
Maicon Araújo
  • 21
  • 2
2
votes
1 answer

Converting "Pre-Zipped" list values to integers

I frequently have a list of "pre-zipped" (elements are lists) data. I'm trying to convert each item to an integer (from an input string, usually). How can I make each element in a complex list into an integer? I initially attempted a list…
Zara
  • 86
  • 1
  • 7
2
votes
0 answers

Why do we need map() and apply() if we already have starmap()?

import multiprocessing as mp pool = mp.Pool() As I understand it (please correct me if wrong): map works on functions that take only one input parameter, and you can pass in a list of such single parameter to make multiple function calls apply…
Indominus
  • 1,228
  • 15
  • 31
1
vote
0 answers

python - Difference in CPU cores used when using Pool map and Pool starmap

I want to use Pool to split a task among n workers. What happens is that when I'm using map with one argument in the task function, I observe that all the cores are used, all tasks are launched simultaneously. On the other hand, when I'm using…
tzoukritzou
  • 337
  • 1
  • 4
  • 16
1
vote
1 answer

Is my understanding of multiprocessing's starmap correct?

I wrote a little program to check a bunch of hashes, and I was wondering if this is a correct application of starmap and if it should be making other processes. My current understanding is that starmap is supposed to divide the task amongst the…
NeffPeff
  • 46
  • 5
1
vote
0 answers

multiprocessing a for loop in python using pool.starmap and multiple arguments

from multiprocessing import Pool pending_integrations = DataService().get_pending_integrations(args.integration) for integration in pending_integrations: integration_id = integration.get('id') …
karthik kancharla
  • 11
  • 3
1
2 3 4