I work with AI Platform Python3 Jupyter notebooks on a daily basis and I noticed an inconsistent behavior at memory error situations among different VM types.
Say I'm trying to use more memory than available, while on "standard" VM types I would get a "Memory error" stack trace as cell output, on the new "Efficient instance" type VMs, the kernel will restart automatically without any notifications (even the cell will remain marked as still executing); the only way to figure out what has happened is by looking at the htop core/mem usage.
It took me some time to realize what was going on, and I believe an error as the output should be the ideal response in this case (unlike the "Efficient instance" case).
Here's an example to clarify what I'm describing:
VM general config:
- Environment: Intel® optimized Base (with Intel® MKL)
- Other configs: default
Python3 Notebook cell code to be executed:
import numpy as np
np.random.rand(1000000000,10)
VM type / output:
- e2-highmem-2 (2 vCPUs, 16 GB memory) --> Kernel restart no notification/logging (IMG1 below).
- n1-highmem-2 (2 vCPUs, 13 GB memory) --> MemoryError traceback as cell output (IMG2 below).
- n2d-highmem-2 (2 vCPUs, 16 GB memory) --> MemoryError traceback as cell output (IMG2 below).
IMG1 (no notification though kernel already restarted)
IMG2 (Memory error traceback cell output)
