Download file via Webservice and Push it to Azure Blob Storage via Node/Express

Question

I need to retrieve files from a vendor's Web service and push these into a unique blob container so users have a unique "workspace". Essentially, I would get files down from the vendor and users could edit these via the files in their own blob container so they don't cross each others working files. I have the unique blob containers working, but need to "download"/GET the files from my vendors API and push them into a blob container. I am able to successfully retrieve the files, which will be separate calls to get a PDF, text files, and images... but if I attempt to upload them to Azure Blob Storage, I get the following error in Node.js:

TypeError: Cannot read property 'length' of null

I am thinking that I need to encode the files as base64 on the client side to properly get the length, and have seen some examples of using a Canvas with toDataURL, but am unsure if this is the best method for essentially downloading and pushing directly to Azure Blob Storage, especially since I have files such as PDFs (not sure if PDFs can be base64 encoded).

Here is my AngularJS controller that calls the service (note that the actual endpoint may change depending on which files they call, so I am using a client side GET of files to control variables that a user may enter in a form):

$scope.getFiles = function () {

$.ajax({
url: 'http://vendorwebservice.net/ws/file1',
type: "GET",
success: function (result) {
console.log(result);
var filename = 'Texture_0.png';

$http.post('/postFile', { filename: filename, file: result }).success(function (data) {
console.log(data);
}, function (err) {
console.log(err);
});

alert("Files Retrieved!");
},
error: function (error) {
console.log("Failed to download image!");
}
})
}

Here is my backend/Node/Express code:

app.post('/postFile', function (req, res, next) {
    var filename = req.body.filename;
    var file = req.body.file;
    var base64Data;
    fileBuffer = decodeBase64Image(file);
    blobSvc.createBlockBlobFromText('blob5', filename, fileBuffer.data, { 'contentType': fileBuffer.type }, function (error, result, response) {
        if (!error) {
            console.log("Uploaded" + result);
        }
        else {
            console.log(error);
        }
    });
})

// Decode file for upload
function decodeBase64Image(dataString) {
    var matches = dataString.match(/^data:([A-Za-z-+\/]+);base64,(.+)$/),
        response = {};

    if (matches.length !== 3) {
        return new Error('Invalid input string');
    }

    response.type = matches[1];
    response.data = new Buffer(matches[2], 'base64');

    return response;
}

Update 1: Per Gary's suggestion I have tried the following, but messed up the code a bit since my vendors API does not have file URIs but rather endpoints that produce a file on a GET (aka, I am lost on how to pass the endpoint in versus Gary's example that makes sense). For example, my vendors endpoint of 'http://vendorapi.net/ws/texture_0' returns a file named "Texture_0.png".

Front end Angular Code:

 $scope.getFromVendor = function () {
            var filename = 'Texture_0.png';//jpg,txt...
            $http.post('/uploadapifiles', { filename: filename, url: 'http://vendorapi.net/ws/texture_0' }).success(function (data) {
                console.log(data);
            }, function (err) {
                console.log(err);
            });
        }

Server Side Download Processing (I believe this is the one that is the most messed up:

app.get(http://vendorapi.net/ws/texture_0', function (req, res, next) {
    res.download('http://vendorapi.net/ws/texture_0' + req.params.filename);
})

Server Side Upload Processing:

app.post('/uploadapifiles', function (req, res, next) {

    var filename = req.body.filename;
    var r = request(req.body.url).pipe(fs.createWriteStream('http://vendorapi.net/ws/texture_0' + filename))
    r.on('close', function () {
        blobsrv.createBlockBlobFromLocalFile('blob5', filename, 'http://vendorapi.net/ws/texture_0' + filename, function (error, result, response) {
            if (!error) {
                console.log("Uploaded" + result);
            }
            else {
                console.log(error);
            }
        });
    })
});

Answer 1

In your original idea, at first you get file content data in client side and then post the data to the Express web server.

If the file you get is in a large size, it will slow down your site because of the file data will be transferred twice via HTTP, and it may occur other problem.

Furthermore, in my test project, it is hardly to handle with file content data directly.

So I tried another idea as a workaround.

I just post the API of getting specific file to the server, pull the file save as a file on server directory and upload file to Storage on server side. Here is my code snippet:

Angular front end:

$scope.upload =function(){
    var filename = (new Date()).getTime()+'.pdf';//jpg,txt...
    $http.post('http://localhost:1337/uploadfile', { filename: filename, url: 'http://localhost:1337/output/123.pdf'}).success(function (data) {
        console.log(data);
    },function(err){
        console.log(err);
    });
  }

Back end:

I suspect the API which you get the file form would be like behind.

router.get('/output/:filename', function (req, res, next) {
    res.download('upload/'+req.params.filename);
})

The post request handler leverages package request, and there is no necessary to figure out file type or encoding type, createBlockBlobFromLocalFile will upload the file at the location you provide on blob storage, API reference:

router.post('/uploadfile', function (req, res, next) {
    var request = require('request');
    var filename = req.body.filename;
    var tmpFolder = 'upload/', //this folder is to save files download from vendor URL, and should be created in the root directory previously. 
        tmpFileSavedLocation = tmpFolder + filename; //this is the location of download files, e.g. 'upload/Texture_0.png'
    var r = request(req.body.url).pipe(fs.createWriteStream(tmpFileSavedLocation))//this syntax will download file from the URL and save in the location asyns
   r.on('close', function (){
        blobsrv.createBlockBlobFromLocalFile('vsdeploy', filename, tmpFileSavedLocation, function (error, result, response) {
            if (!error) {
                console.log("Uploaded" + result);
           }
            else {
                console.log(error);
            }
        });
    })

})