0

I'm trying to download files via wget, but instead of just using the server names, I'd also like to add a prefix to the server filename

i.e. foo-serverfilename.ext

I'm open to other command line based tools like curl etc if they would be better, so long as they can handle using a cookies.txt file for password protect site

I've been Googling for a solution for days, but can't seem to come up with anything

Asteroids With Wings
  • 17,071
  • 2
  • 21
  • 35
Newo
  • 9
  • 1
  • Can't you just rename the files? – Asteroids With Wings Dec 10 '20 at 15:04
  • Also, "append a prefix" makes no sense. "Append" means "put at the end". You want to _prepend_ some text. – Asteroids With Wings Dec 10 '20 at 15:05
  • https://unix.stackexchange.com/questions/519887/batch-download-a-series-of-files-and-add-prefix-to-them for some automation tips, though overkill for your "static prefix" case. – Asteroids With Wings Dec 10 '20 at 15:18
  • i don't want to rename them, entirely, as i would then lose the server name. i want to keep the server name with a prefix and this would be used for over 1,000 files, so renaming manually doesn't work, either. – Newo Dec 10 '20 at 15:39
  • Why would you lose the server name? Renaming doesn't need to involve losing the entirety of the original name. You can write a script in bash to add a prefix to all filenames in a directory in like three lines... – Asteroids With Wings Dec 10 '20 at 15:40
  • https://stackoverflow.com/q/208181/4386278 – Asteroids With Wings Dec 10 '20 at 15:41
  • So a bit more context... I'm downloading 1,400 files. The URL names and the downloaded filenames are nothing alike, due to redirects. As such, I have used OctoParse to extract the download URLs and the date of publication on the site. I want to basically download the files, keep the server names, but append the "publication" date from the website that I extracted, which is unique for each file... That's why renaming via the download process seems more reliable, to me, than anything else – Newo Dec 10 '20 at 15:47
  • Okay, so the scripting solutions in my last link are what you want. I don't see an appropriate filename transformation flag in either wget or curl (just look at the list of options!), probably because they're unnecessary. Download into an isolated, empty directory to avoid accidental renaming of unrelated files. – Asteroids With Wings Dec 10 '20 at 15:59

1 Answers1

0

Neither curl nor wget documentation lists an appropriate option for this (though curl has some more complex transformation flags for some situations).

That's probably because it's unnecessary: you just need a shell script to rename the files once you've got them:

for filename in *; do mv "$filename" "foo-$filename"; done;

I recommend that you download each batch into an isolated, empty directory to avoid accidental renaming of unrelated files; you can then move them wherever you like.

Asteroids With Wings
  • 17,071
  • 2
  • 21
  • 35
  • Sorta kinda closer, but I would still need a way to use the right date with the right file. Is there a way to batch download a list of files via WGET or CURL, but specify a unique path for each? Example Below (Dates are directory names) URL1 to 2020-01-01 URL2 to 2020-01-01 URL3 to 2019-10-12 URL4 to 2018-03-25 – Newo Dec 10 '20 at 16:32
  • This is the first time you've mentioned anything about a date. I don't really understand your followup question. Can you be more specific about the problem? – Asteroids With Wings Dec 10 '20 at 16:36
  • Apologies. The date was the "foo" I was trying to append and it varied file to file. Sorry about being unclear. That said, your responses have nudged me in the right direction and I am putting the finishing touches on a working script. I will be specifying path for each download, using the date as the folder name. Then, I will take a 2nd step and use a renaming tool to prepend the directory name to all files in the directory. Thanks again. I would have never come up with that solution without your help! – Newo Dec 10 '20 at 17:37