How could I send a curl output straight to an Azure Storage?

Question

I'm trying to send downloaded files using bash using a curl request:

    
for secao in $tipo_dou;
do
    download="curl -k --silent -fL -b cookies.iakim 'https://inlabs.in.gov.br/index.php?p=$ano-$mes-$dia&dl=$ano-$mes-$dia-$secao.zip' -H 'origem: 736372697074' -o


    " 
    echo $download > $ano-$mes-$dia-$secao.sh
    sh $ano-$mes-$dia-$secao.sh
    rm -rf $ano-$mes-$dia-$secao.sh
done

By doing that I was able to retrieve the data into my local machine.

The problem is: I must retrieve that data and send it towards an Azure Storage and for that I've been trying to make a pipe such this ex:

  filename=$ano-$mes-$dia&dl=$ano-$mes-$dia-$secao.zip
for secao in $tipo_dou;
do
    download="curl -k --silent -fL -b cookies.iakim 'https://inlabs.in.gov.br/index.php?p=$ano-$mes-$dia&dl=$ano-$mes-$dia-$secao.zip' -H 'origem: 736372697074' 
                 |
                curl -X PUT -T - -H x-ms-blob-type:BlockBlob 'https://server.blob.core.windows.net/ape/filename?st=2023-01-25T18:41:00Z&se=2025-12-01T02:41:00Z&spr=https&sv=2021-06-08&sr=c&sig=SIGNATUREID'


    " 
    echo $download > $ano-$mes-$dia-$secao.sh
    sh $ano-$mes-$dia-$secao.sh
    rm -rf $ano-$mes-$dia-$secao.sh
done

And so far I was only able to login into the system that I'm trying to download it but I got an error like this:

<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:6b17fae3-601e-0065-5483-31ec49000000
Time:2023-01-26T12:43:31.1912380Z</Message>

And to execute that script I'm using Node:

 const { exec } = require('child_process');
 const fs = require('fs');
 const zlib = require('zlib');
 
 const directoryFiles = fs.readdirSync('.');


 const download = exec('sh xml-downloader.sh', (error, stdout, stderr ) => {
     console.log('stdout: ', stdout);
        console.log('stderr: ', stderr);
        if (error !== null) {
            console.log('exec error: ', error);
        }
 })

Is there any other way that I could do that?

Appreciate any help!

Looks like an issue with your SAS URL (`https://server.blob.core.windows.net/ape/filename?st=2023-01-25T18:41:00Z&se=2025-12-01T02:41:00Z&spr=https&sv=2021-06-08&sr=c&sig=SIGNATUREID`). It is missing the permissions. There should be `sp` parameter in your SAS URL. — Gaurav Mantri, Jan 26 '23 at 13:31
The lack of quoting is jarring. Your code is very brittle and will not work if one of the variables contains whitespace or a shell wildcard character which happens to match some files. See [When to wrap quotes around a shell variable](https://stackoverflow.com/questions/10067266/when-to-wrap-quotes-around-a-shell-variable) — tripleee, Jan 26 '23 at 13:50
I've tried that tho. ``` >Signature did not match. String to sign used was racwdl ``` https://cncdevrseus2container01.blob.core.windows.net/ape?sp=racwdl&st=2023-01-25T18:41:00Z&se=2025-12-01T02:41:00Z&spr=https&sv=2021-06-08&sr=c&sig=SIGNATUREID — Fabio Faria, Jan 26 '23 at 13:54
@tripleee thanks for your reply. I read the exemple you gave and made few adjustments. Thanks a lot! — Fabio Faria, Jan 26 '23 at 14:10
@FabioFaria - Your 2nd URL is missing the blob name. It should be something like `https://cncdevrseus2container01.blob.core.windows.net/ape/filename?sp=racwdl&st=2023-01-25T18:41:00Z&se=2025-12-01T02:41:00Z&spr=https&sv=2021-06-08&sr=c&sig=SIGNATUREID`. — Gaurav Mantri, Jan 26 '23 at 14:27
@GauravMantri I checked that already but now I'm facing an issue because I'm sending a zip file and it's not allowed apparently. Since I'm getting a 501 For not supported encoding type like gzip. :/ Appreciate tho! — Fabio Faria, Jan 26 '23 at 15:21

score 1 · Answer 1 · answered Jan 26 '23 at 14:14

@FabioFaria as mentioned, you'll need to configure a SAS token to enable your app to access blob storage.

Alternatively, since you're already using node to run the application, you could create an app registration for your node application and use the blob client directly.

How could I send a curl output straight to an Azure Storage?

1 Answers1