Send Binary Image from Serverless AWS Lambda Over FTP

Question

I'm building an AWS Lambda API, using the serverless stack, that will receive an image as a binary string then store that image on GitHub and send it over FTP to a static app server. I am NOT storing the image in S3 at all as I am required not to.

I have already figured out how to save the image to GitHub (by converting binary to base64), the issue is sending the binary image data over FTP to another server to hold statically. The static image server is pre-existing and I cannot use S3. I am using the ftp npm package to send the image. The image server indeed receives the image BUT NOT in the correct format and the image is just non-displayable junk data. I saw an example on how to do this on the client side by encoding the image in an Uint16Array and passing it to a new Blob() object, but sadly, NODE.JS DOES NOT SUPPORT BLOBS!

Using ftp npm module:

    async sendImage(image) {
        try {
            const credentials = await getFtpCredentials(this.platform);
            console.log('Sending files via FTP client');
            return new Promise((resolve, reject) => {
                let ftpClient = new ftp();
                ftpClient.on('ready', () => {
                    console.log(`Putting ${image.path} onto server`);
                    // Set transfer type to binary
                    ftpClient.binary(err => {
                        if(err)
                            reject(err);
                    });
                    // image.content is binary string data
                    ftpClient.put(image.content, image.path, (err) => {
                        if (err) {
                            reject(err);
                        }
                        console.log('Closing FTP connection');
                        ftpClient.end();
                        resolve(`FTP PUT for ${image.path} successful!`);
                    });
                });
                console.log('Connecting to FTP server');
                ftpClient.connect(credentials);
            });
        } catch(err) {
            console.log('Failed to load image onto FTP server');
            throw err;
        }
    }

This procedure sends the binary data to the server, but the data is un-readable by any browser or image viewer. Do I need to use another FTP package? Or am I just not encoding this right?? I've spent days googleing the answer to this seemingly common task and it's driving me up the wall! Can anyone instruct me on how to send binary image data from a node.js lambda function over FTP to another server so that the image is encoded properly and works when viewed? ANY help is very much appreciated!

Can you explain how the image object is created? Is it the same object that is being sent to the API? i.e. are you sending a string of 1s and 0s inside a JSON object and expecting it to act like binary data? — K Mo, Aug 22 '19 at 16:35
Hi @KMo, the image object is created by using the FileReader.readAsBinaryString() function on the client. Once the image data is read on client view, it is sent as a binary string back to the AWS API (URL encoded, then decoded back to UTF-8[?] string). — Anthony Gilliam, Aug 22 '19 at 17:17
Obviously your data is getting corrupted somewhere in the process, have you console logged the image data received in the event in Lambda to check it looks ok? — K Mo, Aug 22 '19 at 17:37
@KMo I think you're on to something. All signs point to the ftp module corrupting the data. You think it's an encryption issue? How can I stop this from happening. I placed an [un-corrupted before ftp image](https://www.dropbox.com/s/264w6u5yromvdpx/Snoopy_Original_Uncorrupted.png?dl=0) and an [after ftp corrupted image](https://www.dropbox.com/s/e8hbfg47cd17t1f/Snoopy_After_FTP_Corrupted.png?dl=0) to my dropbox. You can open them both up with regular notepad to view the differences. Notice there are a few characters missing in the corrupted image. How do I prevent this? Thanks! — Anthony Gilliam, Aug 22 '19 at 20:07
I submitted an [issue](https://github.com/mscdex/node-ftp/issues/253) to the ftp npm library. For now, I'll look for FTP modules that will not corrupt binary images on transfer... — Anthony Gilliam, Aug 22 '19 at 22:44

score 0 · Accepted Answer · answered Aug 28 '19 at 14:29

So, it turns out that the ftp node module that I was using was the issue all along. It will corrupt any binary image transferred over FTP. I submitted a ticket to the their GitHub repo, but they haven't made a commit in 4 years so I don't expect a timely fix.

To solve the problem, I used the Basic FTP package instead:

const ftp = require('basic-ftp');

async sendImage(image) {
    try {
        const { host, user, password } = await getFtpCredentials(this.platform);
        console.log('Received FTP credentials');
        console.log('Creating FTP client');
        const client = new ftp.Client();
        client.ftp.verbose = true;
        console.log('Logging in to remote server via FTP');
        await client.access({
            host, user, password, secure: false
        });
        console.log('Creating readable stream from binary data');
        const buffer = new Buffer(image.content, 'binary');
        const imgStream = new Readable();
        imgStream._read = () => {};
        imgStream.push(buffer);
        imgStream.push(null);
        console.log('Uploading readable stream to server via FTP');
        const result = await client.upload(imgStream, image.path);
        client.close();
        return result
    } catch(err) {
        console.log('Failed to load image onto FTP server');
        throw err;
    }
}

Note that this package requires data to be transferred via a Readable stream. Thanks to this SO answer, I learned how to convert a binary string to a Readable stream. All works fine.

Happy coding!

Send Binary Image from Serverless AWS Lambda Over FTP

1 Answers1