0

I have a VM with 60GB and 45GB is used. It has multiple users.

I have deleted old users and now I have about 40GB used. I want to download all the data to my machine. How do I do that? The only way I know is with scp the command is something like this one: gcloud compute scp --recurse <$USER>@<INSTANCE>:/home/$USER ~/$LOCAL_DIR but it's very slow and might even take a couple of days. Is there a better and easier way to download the data?

DarkDead
  • 85
  • 8

1 Answers1

3

The bottleneck here seems to be that you're trying to download a considerable amount of data over SSH which is quite slow.

One thing you can try to speed up the process is break down the download into two parts:

  • Upload the content of your VM to Cloud Storage Bucket
  • Download the content from Cloud Storage to your machine

So in that case, from the VM you'll execute:

gsutil cp -R /home/$USER gs://BUCKET_NAME

And then from your local machine:

gsutil cp -R gs://BUCKET_NAME ~/$LOCAL_DIR

Gsutil uses parallel composite uploads to speed up uploading large files, in case you have any. And in the first step you'll be doing GCP <-> GCP communication which will be faster than downloading from SSH directly.

bhito
  • 2,083
  • 7
  • 13
  • I am getting `CommandException: No URLs matched: <$USER>@:/$HOME` – DarkDead Feb 09 '23 at 08:56
  • what was the exact command you tried to run? I have edited my answer because I realised I mistakenly copied the SSH part of the command, so `<$USER>@:` should be removed – bhito Feb 09 '23 at 09:21
  • I created a storage bucket named it apple-usa-backup and I tried the command `gsutils cp -R darkdead@apple-usa:/home gs://apple-usa-backup` I am getting `CommandException: No URLs matched: darkdead@apple-usa:/home` – DarkDead Feb 09 '23 at 09:24
  • okay yes, the command looks ok but the SSH part should be removed: `gsutils cp -R /home gs://apple-usa-backup`, as you're already inside the VM – bhito Feb 09 '23 at 09:26
  • I was not doing this from inside the VM though. I tried by going inside the VM and running `gsutil cp -R /home gs://apple-usa-backup` and I got the error: `Copying file://./darkdead/.bashrc [Content-Type=application/octet-stream]... AccessDeniedException: 403 Access denied.` I tried using sudo or as a root as well but that didn't work either. Getting same 403 error – DarkDead Feb 09 '23 at 09:28
  • 1
    Thank you! I managed to fix it. It was due to permissions errors. [This really helped](https://stackoverflow.com/questions/27275063/gsutil-copy-returning-accessdeniedexception-403-insufficient-permission-from) – DarkDead Feb 09 '23 at 10:43
  • 1
    Happy to see you making it work! Hopefully this helps making the process of retrieving this data faster – bhito Feb 09 '23 at 11:24
  • Honestly, it doesn't look like it does. So far it has only transferred about 700MB. But since I am running it in the VM I decided to install tmux and run the command inside that. I can just let it run in the background and not worry about it so that's great – DarkDead Feb 09 '23 at 12:09
  • 13.3GB done so far. Another couple of days before everything gets transferred. – DarkDead Feb 10 '23 at 04:20