Download URLs from CSV into subdirectory given in first field

Question

So I want to export my products into my new website. I have an csv file with these data:

product id,image1,image2,image3,image4,image5
1,https://img.url/img1-1.png,https://img.url/img1-2.png,https://img.url/img1-3.png,https://img.url/img1-4.png,https://img.url/img1-5.png
2,https://img.url/img2-1.png,https://img.url/img2-2.png,https://img.url/img2-3.png,https://img.url/img2-4.png,https://img.url/img2-5.png

What I want to do is to make a script to read from that file, make directory named with product id, download images of the product and put them inside their own folder (folder 1 => image1-image5 of product id 1, folder 2 => image1-image5 of product id 2, and so on).

I can make a normal text file instead of using the excel format if it's easier to do. Thanks before.

Sorry I'm really new here. I haven't done the code yet because I'm clueless, but what I want to do is something like this:

for id in $product_id; do
  mkdir $id && cd $id && curl -o $img1 $img2 $img3 $img4 $img5 && cd ..
done

Answer 1

Here is a quick and dirty attempt which should hopefully at least give you an idea of how to handle this.

#!/bin/bash

tr ',' ' ' <products.csv |
while read -r prod urls; do
     mkdir -p "$prod"
     # Potential bug: urls mustn't contain shell metacharacters
     for url in $urls; do
         wget -P "$prod" "$url"
     done
done

You could equivalently do ( cd "$prod" && curl -O "$url" ) if you prefer curl; I generally do, though the availability of an option to set the output directory with wget is convenient.

If your CSV contains quotes around the fields or you need to handle URLs which contain shell metacharacters (irregular spaces, wildcards which happen to match files in the current directory, etc; but most prominently & which means to run a shell command in the background) perhaps try something like

while IFS=, read -r prod url1 url2 url3 url4 url5; do
    mkdir -p "$prod"
    wget -P "$prod" "$url1"
    wget -P "$prod" "$url2"
    : etc
done <products.csv

which (modulo the fixed quoting) is pretty close to your attempt.

Or perhaps switch to a less wacky input format, maybe generate it on the fly from the CSV with

awk -F , 'function trim (value) {
       # Trim leading and trailing double quotes
       sub(/^"/, "", value); sub(/"$/, "", value);
       return value; }
  { prod=trim($1);
    for(i=2; i<=NF; ++i) {
        # print space-separated prod, url
        print prod, trim($i) } }' products.csv |
while read -r prod url; do
    mkdir -p "$prod"
    wget -P "$prod" "$url"
done

which splits the CSV into repeated lines with the same product ID and one URL each, and any CSV quoting removed, then just loops over that instead. mkdir with the -p option helfully doesn't mind if the directory already exists.

Answer 2

If you followed the good advice that @Aaron gave you, this code can help you, as you seem to be new with bash I commented out the code for better comprehension.

#!/bin/bash

# your csv file
myFile=products.csv

# number of lines of file
nLines=$(wc -l $myFile | awk '{print $1}')

echo "Total Lines=$nLines"

# loop over the lines of file
for i in `seq 1 $nLines`;
    do
        # first column value
        id=$(sed -n $(($i+1))p $myFile | awk -F ";" '{print $1}')

        line=$(sed -n $(($i+1))p $myFile)

        #create the folder if not exist
        mkdir $id 2>/dev/null

        # number of images in the line
        nImgs=$(($(echo $line | awk -F ";" '{print NF-1}')-1))

        # go to id folder
        cd $id
        #loop inside the line values
        for j in `seq 2 $nImgs`;
            do
                # getting the image url to download it
                img=$(echo $line | cut -d ";" -f $j)
                echo "Downloading image $img**";echo
                # downloading the image
                wget $img
        done 
        # go back path
        cd ..
done