How can I copy large files from one directory to other directory with less time in Data Lake through Databricks?

Question

I'm trying to copy 20GB files from one folder to another folder in Azure Data Lake and want to achieve it through Data Bricks. I have tried the below code but it is taking more then an hour. Can anyone suggest me how to achieve this with less then 20 minutes?

import shutil, os
shutil.copytree("/dbfs/mnt/storage1/ABC/", "/dbfs/mnt/storage1/copied/")

Possible duplicate of [Python: How to Copy Files Fast](https://stackoverflow.com/questions/22078621/python-how-to-copy-files-fast) — , Mar 05 '19 at 17:59

score 2 · Answer 1 · answered Mar 15 '19 at 10:34

2

Best option would be using dbutils.fs.

This would do it for you:

 dbutils.fs.cp ("/mnt/storage1/ABC/", "/mnt/storage1/copied/", recurse=True)

answered Mar 15 '19 at 10:34

Hauke Mallow

2,887
3
11
29

score 0 · Answer 2 · answered Mar 06 '19 at 16:00

0

Trying using the azure.datalake.store library, more details here: https://github.com/Azure/azure-data-lake-store-python

That should prevent databricks download and re-uploading the file.

answered Mar 06 '19 at 16:00

simon_dmorias

2,343
3
19
33

How can I copy large files from one directory to other directory with less time in Data Lake through Databricks?

2 Answers2