0

I've trained a model in the cloud and I saved it. Now, I'm trying to download it onto my local machine and also share it with other people. However, it seems that because the file is too large (700MB) it gets truncated every time I try to download it (ends up being 300MB in size rather than 700MB). I really need to download this model, and it takes days to re-train, is there anyway of getting a file this large from the Juypter Notebook Instance?

Anna
  • 47
  • 4
  • How are you running this server and training this model? Is it just running on a compute instance or a specific managed service? – Jake Nelson Nov 28 '21 at 23:47
  • It was just on a compute instance! I created a new notebook in the Vertex AI section – Anna Nov 28 '21 at 23:48

1 Answers1

1

In your pipeline you can define a ModelExportOP that will export your model to a Google Storage bucket that you have created and defined within the pipeline.

Once you have it in the pipeline you can grant yourself read privilege to the bucket and download it.

Jake Nelson
  • 1,748
  • 13
  • 22
  • 1
    Ok! I will give this a shot! Thanks so much :) – Anna Nov 29 '21 at 00:02
  • Ok, so it didn't work because I didn't train the model itself using Vertex AI. So there isn't a direct pipeline :/ do you have any other insight into this? I wrote everything in the Juypter Notebook Instance, and trained the models using the GPUs in those instances. (A basic example would just be a Keras model trained on your local machine). So I'm unclear about how to use ModelExportOp in this case. – Anna Nov 29 '21 at 15:33
  • 1
    Ah okay, if you used some library inside the notebook it might be dependant on the library you used, can you provide more information or a link to a public repository? Within Notebooks you can write values to files (as it's just Python) but the method for exporting the model itself would be dependant on how the model is saved in the Notebook and what libraries were used to create it. – Jake Nelson Nov 29 '21 at 23:54
  • 1
    Yes! I actually figured it out, similar to what you said, I did an API call from the notebook itself and then sent it directly from the notebook to a GCP bucket using python :) – Anna Dec 02 '21 at 17:52