How to speed up distcp when transferring data from Hadoop to Google Cloud Storage

Question

The google cloud provides connectors for working with Hadoop.(https://cloud.google.com/hadoop/google-cloud-storage-connector)

Using the connector, I receive data from hdfs to google cloud storage

ex)

hadoop discp hdfs://${path} gs://${path}

but data is too large(16TB) and receive speed is just 2mb/s

So, I try to change set up distcp ( map property, bandwith property ... )

However speed is same.

How to speed up distcp when transferring data from HDFS to Google Cloud Storage

do you know if google has any parameters like amazon `fs.s3a.fast.upload=true` — AM_Hawk, Mar 23 '17 at 13:22
Generally to achieve maximum throughput you need to set number of mappers to a number of files that you copy: `hadoop discp -m hdfs://${path} gs://${path}` — Igor Dvorzhak, Nov 08 '18 at 06:38

score 3 · Accepted Answer · answered Jun 13 '18 at 10:51

The official documentation states that the one of the best options of transferring data from on-premises clusters to GCP is using a VPN tunnel over the internet or even using multiple VPN tunnels for additional bandwidth.

Other options proposed are using direct peering between Google's edge points of presence (PoPs) and your network, or establishing a direct connection to Google's network with the help of a Cloud Interconnect service provider.

How to speed up distcp when transferring data from Hadoop to Google Cloud Storage

1 Answers1