Lua - Post multipart/form-data and a file via http.request

Question

I’m trying to use the REST APi for Paperless-ngx to upload documents to a http server, their instructions are as follows..

POSTing documents

The API provides a special endpoint for file uploads:

/api/documents/post_document/

POST a multipart form to this endpoint, where the form field document contains the document that you want to upload to paperless. The filename is sanitized and then used to store the document in a temporary directory, and the consumer will be instructed to consume the document from there.

The endpoint supports the following optional form fields:

title: Specify a title that the consumer should use for the document.

created: Specify a DateTime document was created (e.g. “2016-04-19” or “2016-04-19 06:15:00+02:00”).

correspondent: Specify the ID of a correspondent that the consumer should use for the document.

document_type: Similar to correspondent.

tags: Similar to correspondent. Specify this multiple times to have multiple tags added to the document.

The endpoint will immediately return “OK” if the document consumption process was started successfully. No additional status information about the consumption process itself is available, since that happens in a different process

While I’ve been able to achieve what I needed with curl (see below), I’d like to achieve the same result with Lua.

curl -H "Authorization: Basic Y2hyaXM62tgbsgjunotmeY2hyaXNob3N0aW5n" -F "title=Companies House File 10" -F "correspondent=12" -F "document=@/mnt/nas/10.pdf" http://192.168.102.134:8777/api/documents/post_document/

On the Lua side, I’ve tried various ways to get this to work, but all have been unsuccessful, at best it just times out and returns nil.

Update: I’ve progressed from a nil timeout, to a 400 table: 0x1593c00 HTTP/1.1 400 Bad Request {"document":["No file was submitted."]} error message

Please could someone help ..

local http = require("socket.http")
local ltn12 = require("ltn12")
local mime = require("mime")
local lfs = require("lfs")

local username = "username"
local password = "password"

local httpendpoint = 'http://192.168.102.134:8777/api/documents/post_document/'
local filepath = "/mnt/nas/10.pdf"
local file = io.open(filepath, "rb")
local contents = file:read( "*a" )

-- https://stackoverflow.com/questions/3508338/what-is-the-boundary-in-multipart-form-data

local boundary = "somerndstring"
local send = "--"..boundary..
            "\r\nContent-Disposition: form-data; "..
            "title='testdoc'; document="..filepath..
            --"\r\nContent-type: image/png"..
            "\r\n\r\n"..contents..
            "\r\n--"..boundary.."--\r\n";

-- Execute request (returns response body, response code, response header)

local resp = {}
local body, code, headers, status = http.request {
    url = httpendpoint,
    method = 'POST',
    headers = {
        -- ['Content-Length'] = lfs.attributes(filepath, 'size') + string.len(send),
        -- ["Content-Length"] = fileContent:len(), 
        -- ["Content-Length"] = string.len(fileContent), 
        ["Content-Length"] = lfs.attributes(filepath, 'size'),
        ['Content-Type'] = "multipart/form-data; boundary="..boundary,
        ["Authorization"] = "Basic " .. (mime.b64(username ..":" .. password)),
        --body = send
    },
    source = ltn12.source.file( io.open(filepath,"rb") ),
    sink = ltn12.sink.table(resp)
}

print(body, code, headers, status)
print(table.concat(resp))

if headers then 
    for k,v in pairs(headers) do 
        print(k,v) 
    end
end 

Answer 1

Seems that your Content-Length header value exceeds your actual content length you're trying to send.

That causes remote server to wait more data from you, which you don't provide. As a result, connection is being terminated by a timeout.

Check your code:

local size = lfs.attributes(filepath, 'size') + string.len(send)

send variable already contains your file contents, so you should not add your file content length twice by calling lfs.attributes.

Just try this:

local size = string.len(send)

You also do not use send variable anywhere in the actual request, which is another mistake.

Answer 2

Huge thanks to a person on GitHub who helped me with this, and also has their own module to do it - https://github.com/catwell/lua-multipart-post .

local http = require("socket.http")
local ltn12 = require("ltn12")
local lfs = require "lfs"
http.TIMEOUT = 5

local function upload_file ( url, filename )
    local fileHandle = io.open( filename,"rb")
    local fileContent = fileHandle:read( "*a" )
    fileHandle:close()

    local boundary = 'abcd'

    local header_b = 'Content-Disposition: form-data; name="document"; filename="' .. filename .. '"\r\nContent-Type: application/pdf'
    local header_c = 'Content-Disposition: form-data; name="title"\r\n\r\nCompanies House File'
    local header_d = 'Content-Disposition: form-data; name="correspondent"\r\n\r\n12'

    local MP_b = '--'..boundary..'\r\n'..header_b..'\r\n\r\n'..fileContent..'\r\n'
    local MP_c = '--'..boundary..'\r\n'..header_c..'\r\n'
    local MP_d = '--'..boundary..'\r\n'..header_d..'\r\n'

    local MPCombined = MP_b..MP_c..MP_d..'--'..boundary..'--\r\n'

    local   response_body = { }
    local   _, code = http.request {
            url = url ,
            method = "POST",
            headers = {    ["Content-Length"] =  MPCombined:len(),
                           ['Content-Type'] = 'multipart/form-data; boundary=' .. boundary
                         },
            source = ltn12.source.string(MPCombined) ,
            sink = ltn12.sink.table(response_body),
                }
     return code, table.concat(response_body)
end

 local rc,content = upload_file ('http://httpbin.org/post', '/mnt/nas/10.pdf' )
 print(rc,content)