Why that code snippet could not run at Google Colab?

Question

I use TPU which provides 35 GB RAM. Before I try to run this code snippet I had 20 GB free RAM. But I could not run that code snippet because of low ram(it says more than 35 GB required) I used a 6 GB dataset. Why this code takes so much place? I used garbage collector and deleted the dataframe that I used before. What can I do to make that code working?

Gen RAM Free: 26.4 GB | Proc size: 156.4 MB GPU RAM Free: 16280MB | Used: 0MB | Util 0% | Total 16280MB

seed = 7
test_size = 0.4
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=test_size)
gc.collect()
model=KNeighborsClassifier(n_neighbors=6,algorithm='kd_tree',n_jobs=25)
model.fit(X_train, y_train)

https://stackoverflow.com/questions/48750199/google-colaboratory-misleading-information-about-its-gpu-only-5-ram-available — Celius Stingher, Dec 19 '19 at 17:00
Comments did not work for me and colab does not ask if I want more ram. — Estheralda, Dec 19 '19 at 17:16

score 1 · Accepted Answer · answered Dec 20 '19 at 00:51

Data science workflows in Python tend to be quite memory hungry. For example, if your input data X, Y consumes 6GB, then the line

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=test_size)

will consume another 6 GB (each created subset is not a view into the original array, but a new array object with a new memory footprint).

Inside the KNeighborsClassifier other copies of the data may be made (e.g. the code in that algorithm requires input arrays to be contiguous C-ordered float64 arrays, and if your data does not meet that criteria a copy will be made).

In general, a good rule-of-thumb for data science workflows based on numpy is you'll need memory of about 4-5x the size of the raw data you're working with, unless you're very careful about preventing data copies.

Why that code snippet could not run at Google Colab?

1 Answers1