Files written to disk get deleted after a while

Question

I am running a notebook on google cloud datalab. It generates some intermediate output files. The files show up when running the notebook.

However after several hours when I open the notebook again only the files in the datalab git repository (notebook files mostly) are there and everything else is deleted. The notebook kernel also seem to get restarted.

Is there any reason why? and how can I avoid this?

score 2 · Accepted Answer · edited May 23 '17 at 11:44

2

Google Cloud Datalab runs on App Engine Managed VM Environment. These use ephemeral disks that do not preserve your data between restarts.

If intermediate output files need to be preserved for future use or compliance reasons, they should be persisted using Google Cloud Storage, or an other durable mechanism.

Storage inside the VM instances is only suitable for temporary, disposable data.

A related question explores the usage of Persistant Disks: Using persistent disks with google Datalab

edited May 23 '17 at 11:44

Community

1
1

answered Mar 06 '16 at 15:04

Karel Van Roey

100
9

So IIUC, if I install extra Python modules it won't persist and I need to re-install the modules every time. – Choghazanbil Mar 06 '16 at 21:45
I re-run `%%bash pip install x` when needed. Datalab is currently in beta, so I hope this will improve. – Karel Van Roey Mar 06 '16 at 22:16

Files written to disk get deleted after a while

1 Answers1