how to reduce memory usage in kaggle for python code

Question

import itertools

deck = ['AD', '2D', '3D', '4D', '5D', '6D', '7D', '8D', '9D', '10D', 'JD', 'QD', 'KD', 
        'AC', '2C', '3C', '4C', '5C', '6C', '7C', '8C', '9C', '10C', 'JC', 'QC', 'KC',    
        'AH', '2H', '3H', '4H', '5H', '6H', '7H', '8H', '9H', '10H', 'JH', 'QH', 'KH',      
        'AS', '2S', '3S', '4S', '5S', '6S', '7S', '8S', '9S', '10S', 'JS', 'QS', 'KS']

combinations = list(itertools.combinations(deck, 9))

i try to find all this combinations then i will load this combinations in a csv file but kaggle gives me this error message:

Your notebook tried to allocate more memory than is available. It has restarted.

I would suggest [using a generator](https://stackoverflow.com/a/37240126/17200348). — B Remmelzwaal, Feb 13 '23 at 23:24
`def combinations_generator(deck, k): for combination in itertools.combinations(deck, k): yield combination combinations = combinations_generator(deck, 9)` i am trying this one i hope it works. Dank u wel — sefa, Feb 13 '23 at 23:33
There are **3679075400** combinations. If you need one combination at a time then there is no point in creating a List to store them all. As @ B Remmelzwaal says - use a generator. — user19077881, Feb 13 '23 at 23:34
If you need all 3.6Billion **at the same time** then you can see how much memory it takes from the number of combinations. — user19077881, Feb 13 '23 at 23:43

Dorian Turba · Accepted Answer · 2023-02-14T01:26:10.493

Don't use a list, just write line after line.

itertools.combinations create an iterator that allows you to iterate over each value without having to create a list and store each value in memory.

You can use the csv module to write each combination as a line. If you don't want an empty line between each combination, don't forget to use the newline='' in open: https://stackoverflow.com/a/3348664/6251742.

import csv
import itertools

deck = ['AD', '2D', '3D', '4D', '5D', '6D', '7D', '8D', '9D', '10D', 'JD', 'QD', 'KD',
        'AC', '2C', '3C', '4C', '5C', '6C', '7C', '8C', '9C', '10C', 'JC', 'QC', 'KC',
        'AH', '2H', '3H', '4H', '5H', '6H', '7H', '8H', '9H', '10H', 'JH', 'QH', 'KH',
        'AS', '2S', '3S', '4S', '5S', '6S', '7S', '8S', '9S', '10S', 'JS', 'QS', 'KS']

combinations = itertools.combinations(deck, 9)

with open('combinations.csv', 'w', newline='') as file:
    writer = csv.writer(file, delimiter=',')
    for combination in combinations:
        writer.writerow(combination)

Result after some time:

AD,2D,3D,4D,5D,6D,7D,8D,9D
AD,2D,3D,4D,5D,6D,7D,8D,10D
AD,2D,3D,4D,5D,6D,7D,8D,JD
AD,2D,3D,4D,5D,6D,7D,8D,QD
AD,2D,3D,4D,5D,6D,7D,8D,KD
...  # 3679075395 more lines, 98.3 GB

how to reduce memory usage in kaggle for python code

1 Answers1

Don't use a list, just write line after line.