How to select a range of columns when using the replace function on a large dataframe?

Question

I have a large dataframe that consists of around 19,000 rows and 150 columns. Many of these columns contain values with -1s and -2s. When I try to replace the -1s and -2s with 0 using the following code, Jupyter times out on me and says no memory left. So, I am curious if you can select a range of columns and apply the replace function. This way I can replace in batches since I cant seem to replace in one pass based on my available memory.

Here is the code a tried to use that timed out on me when first replacing the -2s:

df.replace(to_replace=-2, value="0").

Thank you for any guidance!

Sean

This post might be of interest. https://stackoverflow.com/a/27759140/8230574 — FizzyGood, Dec 16 '22 at 23:53
Please provide enough code so others can better understand or reproduce the problem. — Community, Dec 17 '22 at 05:02

score 0 · Answer 1 · answered Dec 17 '22 at 00:23

Let's say you want to divide your columns in chunks of 10, then you should try something like this:

columns = your_df.columns
division_num = 10
chunks_num = int(len(columns)/division_num)

index = 0
for i in range(chunks_num):
    your_df[columns[index: index+10]].replace(to_replace=-2, value="0")
    index += division_num

If your memory keeps overflowing then maybe you can try with loc function to divide the data by rows instead of columns.

How to select a range of columns when using the replace function on a large dataframe?

1 Answers1