Python multiprocessing struct.error: 'i' format requires -2147483648 <= number <= 2147483647

Asked Jun 12 '19 at 07:47

Active Jun 12 '19 at 08:50

Viewed 1,753 times

I'm creating a lookup dictionary with the help of other dictionaries. I have 32 cores and 64GB RAM instance where I'm running it. names is a huge numpy array with around 28184057 elements. When I try to apply pool.map on 'names', I get the following error:

struct.error: 'i' format requires -2147483648 <= number <= 2147483647

This is my code:

pool = mp.Pool(32)
uni_lookup = pool.map(self.create_lookup_dict, names)
pool.close()

And create_lookup_dict contains the following code:

def create_lookup_dict(self, word):
    lookup_word_level = {word: list(set(i[0] for i in self.groups.items() \
                                if word in i[1]))}
    final = []

    if self.tag_dict.get(word, 0) is 0:
        print("No tag")
        return (word, lookup_word_level[word])
    if self.secondary_category_tag_dict.get(word, 0) is 0:
        print("No secondary category tag")
        return (word, lookup_word_level[word])
    for i in lookup_word_level[word]:
        if self.tag_dict[i].intersection(self.tag_dict[word]) and \
        self.secondary_category_tag_dict[i].intersection(self.secondary_category_tag_dict[word]):
            final.append(i)
    return (word, final)

What am I missing?

edited Jun 12 '19 at 08:50

asked Jun 12 '19 at 07:47

Mayur Bhangale

1

which line is the error coming from – Devesh Kumar Singh Jun 12 '19 at 07:48
@DeveshKumarSingh pool.map – Mayur Bhangale Jun 12 '19 at 07:49
I would suggest you check if the create_lookup_dict works independently, and then try putting . it in pool.map also look at https://docs.python.org/3/library/multiprocessing.html?highlight=process#multiprocessing.pool.Pool.imap – Devesh Kumar Singh Jun 12 '19 at 07:51
@DeveshKumarSingh It takes really long time without pool.map – Mayur Bhangale Jun 12 '19 at 07:51
@DeveshKumarSingh Let me check `pool.imap`, it seems promising – Mayur Bhangale Jun 12 '19 at 07:54
Where is `dish_lookup_word_level` coming from? – ChatterOne Jun 12 '19 at 08:15
@ChatterOne my bad, it was lookup_word_level – Mayur Bhangale Jun 12 '19 at 08:50
1

Can you please add the full traceback? – rtoijala Jun 12 '19 at 10:10
Possibly relevant: https://stackoverflow.com/questions/7894791 and https://stackoverflow.com/questions/21139911 – rtoijala Jun 12 '19 at 10:37
1

This one can be useful too: https://stackoverflow.com/questions/47776486/python-struct-error-i-format-requires-2147483648-number-2147483647?rq=1 Looks like its a problem about the size of data, returned from pool.map. But I still can't figure out how to solve it. – Mikhail_Sam Jun 27 '19 at 15:59

Python multiprocessing struct.error: 'i' format requires -2147483648 <= number <= 2147483647

0 Answers0