unable to add image binary into multipart/form-data

Question

I received an error: expected str instance, bytes found when i tried to add image binary into multipart/form-data.

The problem is i tried to append the imageData in binary format to strings. Is there a way to add binary image to multipart/form data?

I'm at my wits end, would appreciate some help for this.

imageData = request.FILES['filePath'].read()


content_type, request_body = encode_multipart_formdata([('include_target_data', targetMetaDataRet),
                                                     ('max_num_results', str(maxNoResultRet))],
                                                     [('image', imagePath, imageData)])

def encode_multipart_formdata(fields, files):

    BOUNDARY = '----------ThIs_Is_tHe_bouNdaRY_$'
    CRLF = '\r\n'
    lines = []
    for (key, value) in fields:
        lines.append('--' + BOUNDARY)
        lines.append('Content-Disposition: form-data; name="%s"' % key)
        lines.append('')
        lines.append(value)
    for (key, filename, value) in files:
        lines.append('--' + BOUNDARY)
        lines.append('Content-Disposition: form-data; name="%s"; filename="%s"' % (key, filename))
        lines.append('Content-Type: %s' % get_content_type(filename))
        lines.append('')
        lines.append(value)
    lines.append('--' + BOUNDARY + '--')
    lines.append('')
    body = CRLF.join(lines)
    content_type = 'multipart/form-data; boundary=%s' % BOUNDARY
    return content_type, body

Traceback:

  35.            response = get_response(request)

  128.           response = self.process_exception_by_middleware(e, request)

  126.           response = wrapped_callback(request, *callback_args, **callback_kwargs)

  166.           [('image', imagePath, imageData)])

  232.           body = CRLF.join(lines)

Exception Type: TypeError at /identify_shrine
Exception Value: sequence item 12: expected str instance, bytes found

as per @coltoneakins request I modified request body to bytes, but I seem to be getting a bad request error any idea why?

Code:

content_type = 'multipart/form-data; boundary=----------ThIs_Is_tHe_bouNdaRY_$'
request_body = '----------ThIs_Is_tHe_bouNdaRY_$' +  '\n'+'Content-Disposition: form-data; name="include_target_data"' + '\n' + '\n' + 'top'+ '\n' + '----------ThIs_Is_tHe_bouNdaRY_$' +'\n' + 'Content-Disposition: form-data; name="max_num_results"' + '\n' + '\n' + '1' + '\n' + '----------ThIs_Is_tHe_bouNdaRY_$' +'\n' + 'Content-Disposition: form-data; name="image"; filename="img_2.jpg"' + '\n' + 'Content-Type: image/jpeg' + '\n' + '\n'
request_body1 = request_body.encode('utf-8')
request_body2 = imageData
request_body3 = ('\n' + '\n' + '----------ThIs_Is_tHe_bouNdaRY_$').encode('utf-8')
request_body4 = request_body1 + request_body2 + request_body3

content_type_bare = 'multipart/form-data'

# Sign the request and get the Authorization header
# use client key
auth_header = authorization_header_for_request(CLIENT_ACCESS_KEY, CLIENT_SECRET_KEY, HTTP_METHOD, request_body4,
                                               content_type_bare,
                                               date, path)

request_headers = {
    'Accept': 'application/json',
    'Authorization': auth_header,
    'Content-Type': content_type,
    'Date': date
}
try:
    # Make the request over HTTPS on port 443
    connection = http.client.HTTPSConnection(CLOUD_RECO_API_ENDPOINT, 443)
    connection.request(HTTP_METHOD, path, request_body4, request_headers)

    response = connection.getresponse()
    response_body = response.read()
    reason = response.reason
    status = response.status

finally:
    connection.close()

Answer 1

You have a type issue in your code. You are getting a TypeError expected str instance, bytes found because you are attempting to join() a list that contains both str types and bytes types in Python.

Look at these lines in your code:

    for (key, filename, value) in files:
    lines.append('--' + BOUNDARY)
    lines.append('Content-Disposition: form-data; name="%s"; filename="%s"' % (key, filename))
    lines.append('Content-Type: %s' % get_content_type(filename))
    lines.append('')
    lines.append(value) # <---------- THIS IS BYTES, EVERYTHING ELSE IS STR
lines.append('--' + BOUNDARY + '--')
lines.append('')
body = CRLF.join(lines) # <---------- AHHH RED FLAG!!!

CRLF is type str. But, value (which is added on to your lines list) is bytes. This means you end up with lines containing both str and bytes types. When you are sending an image via the mulitpart/form-data request, the whole body of the request is bytes. So, you need to use join() with only bytes types.

This is what you are doing:

body = CRLF.join(lines)

which is really:

'\r\n, i am a str'.join(['i am also str', b'I am not a str, I am bytes']) # <-- NO

This is what you need to be doing:

b'I am bytes'.join([b'I am also bytes', b'Me too!'])

Also, just so that you are aware, the Requests library provides mechanisms for you to send files. See the files parameter in the Requests documentation or this StackOverflow answer:

https://stackoverflow.com/a/12385661/9347694

So, you may not need to reinvent the wheel here. Requests will multipart encode the file and construct the request for you.