0

The directory I am trying to rename has around 5k parquet files.

Unfortunatelly, os and shutil libraries aren't helping with it

import os, shutil

os.rename('/dbfs/FileStore/AllInOneParquets/SdId=791221', '/dbfs/FileStore/AllInOneParquets/test1')
shutil.move('/dbfs/FileStore/AllInOneParquets/SdId=791221', '/dbfs/FileStore/AllInOneParquets/test2')

Both tries above failed. os.rename gave me the following exception: 

OSError: [Errno 7] Argument list too long

The shutil.move, which tries to use os.rename in the background, started to move the files instead of really renaming.

The 5k files are just a test and I am going for much more than it. Is there a way around?

Flavio Pegas
  • 388
  • 1
  • 9
  • 26
  • Why not create the new directory, move the files to the new directory, and then delete the original directory? – Matthew Kligerman Nov 08 '19 at 18:58
  • here [https://stackoverflow.com/questions/30650841/why-am-i-getting-errno-7-argument-list-too-long-and-oserror-errno-24-too-ma](https://stackoverflow.com/questions/30650841/why-am-i-getting-errno-7-argument-list-too-long-and-oserror-errno-24-too-ma) you can fina a very ssimilar error . Hope this helps – nicdelillo Nov 08 '19 at 19:01
  • @MatthewKligerman It costs too much time – Flavio Pegas Nov 08 '19 at 19:13

1 Answers1

0

Try google: python how to move files from one directory to another and you will found the answer :)

How to move a file in Python

and I thing you shod run the code on administrator mode maybe

  • That is not what I asked – Flavio Pegas Nov 12 '19 at 16:50
  • In fact has, because moving files you can specific another name and will change it. example from `d:\folder\folder2\*` to `d:\folder\folder3\*` –  Nov 13 '19 at 12:04
  • I know that, I tried that above. The problem is that the libs doesn't simply rename the folder when I have thousand of files inside it. Actually it moves the data, and that is the problem of the question. – Flavio Pegas Nov 13 '19 at 15:17
  • Ohh in this case you need to make a loop to read all files and to search for the old path and to replace it and inside of the file because some of them will have some lines what will specific to search on the old path... Will not be a easy task, but also not complicated –  Nov 13 '19 at 17:01