1

is there any simple way to copy data within Azure ADLS gen2 using Azure CLI, Rest API or Python?

Azure ADLS gen2 API documentation is very limited for now... https://learn.microsoft.com/en-us/rest/api/storageservices/data-lake-storage-gen2

fuggy_yama
  • 607
  • 6
  • 24

1 Answers1

0

According to my research, we can use Azure CLI or python to move a directory or move a file. For more details, please refer to the document.

For example

  1. Install the storage CLI extension. Please note that the CLI version should be larger than 2.0.67
az extension add -n storage-preview
  1. Script
# move directory
az storage blob directory move -c my-file-system -d my-new-directory -s my-directory --account-name mystorageaccount

# move a file
az storage blob move -c my-file-system -d my-file-new.txt -s my-file.txt --account-name mystorageaccount

Python

try:

       file_system_client = service_client.get_file_system_client(file_system="my-file-system")
       directory_client = file_system_client.get_directory_client("my-directory")

       new_dir_name = "my-directory-renamed"
       directory_client.rename_directory(rename_destination=directory_client.file_system_name + '/' + new_dir_name)

    except Exception as e:
     print(e)
Jim Xu
  • 21,610
  • 2
  • 19
  • 39
  • Thanks, what library do you use in your python example? – fuggy_yama Jan 21 '20 at 10:14
  • 1
    @fuggy_yama Regarding python sdk, please refer to https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-python – Jim Xu Jan 21 '20 at 12:14
  • @fuggy_yama Do you have any other concerns? – Jim Xu Jan 22 '20 at 00:52
  • @fuggy_yamaf since you have no other concerns, could you please [accept the answer](https://stackoverflow.com/help/someone-answers). It may help more people. – Jim Xu Jan 23 '20 at 02:04
  • 1
    I just realized that it is MOVE operation and I actuall need COPY files within ADLS gen2. – fuggy_yama Feb 19 '20 at 20:41
  • @JoshD As far as I knew, azure data lake gen2 python sdk does not provide copy function. But the gen2 is based on blob, we can use python blob sdk to implement it : https://stackoverflow.com/questions/32500935/python-how-to-move-or-copy-azure-blob-from-one-container-to-another – Jim Xu May 20 '20 at 03:33
  • @fuggy_yama did you find anything useful? – anidev711 Feb 02 '21 at 14:06
  • @anidev711 see https://stackoverflow.com/a/58940560/4496488 – fuggy_yama Feb 12 '21 at 10:38
  • This solution MOVES a directory via renaming it. The question is asking about how to COPY data from one directory to another – geominded Mar 14 '23 at 01:46