Post multiple binary files with NodeJs

Question

How can I POST multiple binary files to a server with Content-Type: "form-data", using only the http module?

For example, my keys would look like this:

{
    Image1: <binary-data>,
    Image2: <binary-data>
}

score 3 · Answer 1 · answered Jul 01 '17 at 05:54

TL:DR; See the full example code at the bottom of this answer.

I was trying to figure out how to POST multiple binary image files to a server in NodeJS using only core NodeJs modules (without using anything like npm install request, etc). After about 3 hours typing the wrong search terms into DuckDuckGo and finding no examples online, I was about to switch careers. But after banging my head against the desk for almost half the day, my sense of urgency was dulled and I managed to hobble together a working solution. This would not have been possible without the good people who wrote the answer to this Stackoverflow post, and this Github Gist. I used PostMan and Charles Proxy to analyze successful HTTP POSTS, as I dug around in the NodeJS docs.

There are a few things to understand to POST two binary images and a text field as multipart/form-data, relying only on core NodeJS modules:

1) Boundary Identifiers

What is the boundary in multipart/form-data?

The first part of the solution is to create a "boundary identifier", which is a string of dashes -, appended with a random number. You could probably use whatever you wish, foorbar and such.

------------0123456789

Then place that boundary between each blob of data; whether that is binary data or just text data. When you have listed all your data, you add the boundary identifier at the end, and append two dashes:

------------0123456789--

You also need to add the boundary to the headers so that server receiving the post understands which lines of your post data form the boundary between fields.

const headers = {
    // Inform the server what the boundary identifier looks like
    'Content-Type': `multipart/form-data; boundary=${partBoundary}`,
    'Content-Length': binaryPostData.length
}

2) Form Field Meta Descriptors

(This probably is not what they are called)

You will also need a way to write the meta data for each form-field you send, whether that form field contains a binary or a text object. Here are the descriptors for an image file, which contain the mime type:

Content-Disposition: form-data; name="Image1"; filename="image-1.jpg"
Content-Type: image/jpeg

And here is the descriptor for a text field:

Content-Disposition: form-data; name="comment"

The Post Data Output

So the entire post data that is sent to the server will look like this:

----------------------------9110957588266537
Content-Disposition: form-data; name="Image1"; filename="image-1.jpg"
Content-Type: image/jpeg

ÿØÿàJFIFHHÿáLExifMMi
ÿí8Photoshop 3.08BIM8BIM%ÔÙ²é    ìøB~ÿÀ... <<<truncated for sanity>>>
----------------------------9110957588266537
Content-Disposition: form-data; name="Image2"; filename="image-2.jpg"
Content-Type: image/jpeg

ÿØÿàJFIFHHÿáLExifMMi
ÿí8Photoshop 3.08BIM8BIM%ÔÙ²é    ìøB~ÿÀ... <<<truncated for sanity>>>
----------------------------9110957588266537
Content-Disposition: form-data; name="comment"

This is a comment.
----------------------------9110957588266537--

Once this post data is generated, it can be converted to binary and written to the HTTP POST request: request.write(binaryPostData).

Example Code

Here is the full example code that will allow you to POST binary file and text data without having to include other NodeJS libraries and packages in your code.

// This form data lists 2 binary image fields and text field
const form = [
    {
        name: 'Image1',
        type: 'file',
        value: 'image-1.jpg'
    },

    {
        name: 'Image2',
        type: 'file',
        value: 'image-2.jpg'
    },

    {
        name: 'comment',
        type: 'text',
        value: 'This is a comment.'
    }
]

// Types of binary files I may want to upload
const types = {
    '.json': 'application/json',
    '.jpg': 'image/jpeg'
}

const config = {
    host: 'ec2-192.168.0.1.compute-1.amazonaws.com',
    port: '80',
    path: '/api/route'
}

// Create an identifier to show where the boundary is between each
// part of the form-data in the POST
const makePartBoundary = () => {
    const randomNumbers = (Math.random() + '').split('.')[1]
    return '--------------------------' + randomNumbers
}

// Create meta for file part
const encodeFilePart = (boundary, type, name, filename) => {
    let returnPart = `--${boundary}\r\n`
    returnPart += `Content-Disposition: form-data; name="${name}"; filename="${filename}"\r\n`
    returnPart += `Content-Type: ${type}\r\n\r\n`
    return returnPart
}

// Create meta for field part
const encodeFieldPart = (boundary, name, value) => {
    let returnPart = `--${boundary}\r\n`
    returnPart += `Content-Disposition: form-data; name="${name}"\r\n\r\n`
    returnPart += value + '\r\n'
    return returnPart
}

const makePostData = {
    // Generate the post data for a file
    file: (item, partBoundary) => {
        let filePostData = ''

        // Generate the meta
        const filepath = path.join(__dirname, item.value)
        const extention = path.parse(filepath).ext
        const mimetype = types[extention]
        filePostData += encodeFilePart(partBoundary, mimetype, item.name, item.value)

        // Add the binary file data
        const fileData = fs.readFileSync(filepath, 'binary')
        filePostData += fileData
        filePostData += '\r\n'

        return filePostData
    },

    // Generate post data for the text field part of the form
    text: (item, partBoundary) => {
        let textPostData = ''
        textPostData += encodeFieldPart(partBoundary, item.name, item.value)
        return textPostData
    }
}

const post = () => new Promise((resolve, reject) => {
    let allPostData = ''

    // Create a boundary identifier (a random string w/ `--...` prefixed)
    const partBoundary = makePartBoundary()

    // Loop through form object generating post data according to type
    form.forEach(item => {
        if (Reflect.has(makePostData, item.type)) {
            const nextPostData = makePostData[item.type](item, partBoundary)
            allPostData += nextPostData
        }
    })

    // Create the `final-boundary` (the normal boundary + the `--`)
    allPostData += `--${partBoundary}--`

    // Convert the post data to binary (latin1)
    const binaryPostData = Buffer.from(allPostData, 'binary')

    // Generate the http request options
    const options = {
        host: config.host,
        port: config.port,
        path: config.path,
        method: 'POST',
        headers: {
            // Inform the server what the boundary identifier looks like
            'Content-Type': `multipart/form-data; boundary=${partBoundary}`,
            'Content-Length': binaryPostData.length
        }
    }

    // Initiate the HTTP request
    const req = http.request(options, res => {
        res.setEncoding('utf8')

        let body = ''

        // Accumulate the response data
        res.on('data', chunk => {
            body += chunk
        })

        // Resolve when done
        res.on('end', () => {
            resolve(body)
        })

        res.on('close', () => {
            resolve(body)
        })

        res.on('error', err => {
            reject(err)
        })
    })

    // Send the binary post data to the server
    req.write(binaryPostData)

    // Close the HTTP request object
    req.end()
})

// Post and log response
post().then(data => {
    console.log(data)
})
.catch(err => {
    console.error(err)
})

Post multiple binary files with NodeJs

1 Answers1

1) Boundary Identifiers

2) Form Field Meta Descriptors

The Post Data Output

Example Code