I am trying to store images from a C# client in a Nginx server using the nginx upload module .
I have set up the Nginx as a Docker container and connected it over a network to an ASP.NET Core service in another Docker and an Echo service in a Docker to intercept requests to the Nginx and from the Nginx to the ASP.NET service.
This works with files sent via curl, but files from the C# client are not processed correctly and the Nginx sends an empty request via Proxy_Pass to the ASP.NET service, which then rejects it.
nginx config (mostly default):
load_module modules/ngx_http_upload_module.so;
error_log /var/log/nginx/error.log debug;
events {
}
http {
server {
listen 80;
# Allow file uploads max 1024M for example
client_max_body_size 1024M;
upload_buffer_size 10M;
# POST URL
location /Upload {
# Pass altered request body to this location
upload_pass @after_upload;
# upload_pass_args on;
# Store files to this directory
# upload_store /tmp/nginx_upload/;
upload_store /upload/;
# Allow uploaded files to be world readable
upload_store_access user:rw group:rw all:r;
# Set specified fields in request body
upload_set_form_field upload_file_name "$upload_file_name";
upload_set_form_field upload_content_type "$upload_content_type";
upload_set_form_field upload_tmp_path "$upload_tmp_path";
# Inform backend about hash and size of a file
upload_aggregate_form_field "$upload_field_name.md5" "$upload_file_md5";
upload_aggregate_form_field "$upload_field_name.size" "$upload_file_size";
upload_pass_form_field "^submit$|^description$";
upload_cleanup 400 404 499 500-505;
}
location @after_upload {
# proxy_pass http://commoditytracker;
proxy_pass http://echo:8080;
# add_header Content-Type "text/plain;charset=utf-8";
# return 200 "Upload succeeded";
}
root /upload;
location / {
root /upload;
}
location /upload {
alias /upload/;
}
}
}
Client code
building the Form Data (here with a test image):
Console.WriteLine("Hello, World!");
string path = @"C:\Users\User\Downloads\test.jpg";
byte[] bytes = new byte[8] { 0x40, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20 };
if (File.Exists(path))
{
bytes = File.ReadAllBytes(path);
}
var fileName = "test.jpg";
var contentType = "image/jpeg";
using (var client = new HttpClient())
{
using (var content = new MultipartFormDataContent("boundary1234"))
{
content.Add(new ByteArrayContent(bytes), "file", $"{fileName}");
List<KeyValuePair<string, string>> b = new List<KeyValuePair<string, string>>();
var requestParameters = new KeyValuePair<string, string>("basketId", Guid.Empty.ToString());
b.Add(requestParameters);
var addMe = new FormUrlEncodedContent(b);
content.Add(addMe);
var Message = await client.PostAsync($"http://xxxxxxxxxxxxxx:8081/upload", content);
Console.WriteLine(Message.StatusCode);
Console.WriteLine(await Message.Content.ReadAsStringAsync());
}
Console.WriteLine("-------------------------------------------");
using (var content = new MultipartFormDataContent("boundary1234"))
{
content.Add(new ByteArrayContent(bytes), "file", $"{fileName}");
List<KeyValuePair<string, string>> b = new List<KeyValuePair<string, string>>();
var requestParameters = new KeyValuePair<string, string>("basketId", Guid.Empty.ToString());
b.Add(requestParameters);
var addMe = new FormUrlEncodedContent(b);
content.Add(addMe);
var Message = await client.PostAsync($"https://xxxxxxxxxxxxxxx/Upload", content);
Console.WriteLine(Message.StatusCode);
Console.WriteLine(await Message.Content.ReadAsStringAsync());
// NavigationManager.NavigateTo("fetchdata");
}
}
Resulting requests
Original request from client received by echo service:
{
"path": "/upload",
"headers": {
"host": "xxxxxxxxxxxxxxxx:8081",
"content-type": "image/jpeg",
"content-length": "962"
},
"method": "POST",
"body": "--boundary1234\r\nContent-Disposition: form-data; name=file; filename=test.jpg; filename*=utf-8''test.jpg\r\n\r\n[...]\r\n--boundary1234\r\nContent-Type: application/x-www-form-urlencoded\r\nContent-Disposition: form-data\r\n\r\nbasketId=00000000-0000-0000-0000-000000000000\r\n--boundary1234--\r\n",
"fresh": false,
"hostname": "xxxxxxxxxxxxxxxx",
"ip": "xxxxxxxxxxxxxxxx",
"ips": [],
"protocol": "http",
"query": {},
"subdomains": [],
"xhr": false,
"os": {
"hostname": "xxxxxxxxxxxxxx"
},
"connection": {}
}
xxxxxxxxxxxxxxxx - - [16/Jul/2023:10:40:26 +0000] "POST /upload HTTP/1.1" 200 - "-" "-"
Proxy request after nginx processed the request received by echo service:
{
"path": "/Upload",
"headers": {
"host": "echo:8080",
"connection": "close",
"content-length": "108",
"x-real-ip": "10.0.0.2",
"x-forwarded-for": "10.0.0.2",
"x-forwarded-proto": "https",
"x-forwarded-ssl": "on",
"x-forwarded-port": "443",
"x-original-uri": "/Upload",
"content-type": "multipart/form-data; boundary=\"boundary1234\"",
"content-disposition": "attachment; filename=test.jpg"
},
"method": "POST",
"body": "--\"boundary1234\"\r\nContent-Disposition: form-data; name=\"<ngx_upload_module_dummy>\"\r\n\r\n\r\n--\"boundary1234\"--\r\n",
"fresh": false,
"hostname": "echo",
"ip": "10.0.0.2",
"ips": [
"10.0.0.2"
],
"protocol": "https",
"query": {},
"subdomains": [],
"xhr": false,
"os": {
"hostname": "xxxxxxxxxxxxxxxx"
},
"connection": {}
}
10.0.0.2 - - [16/Jul/2023:10:40:27 +0000] "POST /Upload HTTP/1.0" 200 - "-" "-"
Curl result for comparison:
Original request from client received by echo service:
PS C:\Users\User\Downloads> curl -X 'POST' 'http://xxxxxxxxxxxxx:8081/Upload?basketId=05aa7960-3f4a-42f4-8112-c22b5f96a9b5' -H 'accept: multipart/form-data' -F "upload_file_name=@$(pwd)/test.jpg"
{
"path": "/Upload",
"headers": {
"host": "xxxxxxxxxxxxx:8081",
"user-agent": "curl/8.0.1",
"accept": "multipart/form-data",
"content-length": "877",
"content-type": "multipart/form-data; boundary=------------------------46a959572bae9574"
},
"method": "POST",
"body": "--------------------------46a959572bae9574\r\nContent-Disposition: form-data; name=\"upload_file_name\"; filename=\"test.jpg\"\r\nContent-Type: image/jpeg\r\n\r\n[...]\r\n--------------------------46a959572bae9574--\r\n",
"fresh": false,
"hostname": "xxxxxxxxxxxxxxxxx",
"ip": "xxxxxxxxxxxxxxxxxxxxxxxxx",
"ips": [],
"protocol": "http",
"query": {
"basketId": "05aa7960-3f4a-42f4-8112-c22b5f96a9b5"
},
"subdomains": [],
"xhr": false,
"os": {
"hostname": "xxxxxxxxxxxxxxxxxxx"
},
"connection": {}
}
xxxxxxxxxxxxxxxxxxxxx - - [16/Jul/2023:11:10:05 +0000] "POST /Upload?basketId=05aa7960-3f4a-42f4-8112-c22b5f96a9b5 HTTP/1.1" 200 - "-" "curl/8.0.1"
Proxy request after nginx processed the request received by echo service:
PS C:\Users\User\Downloads> curl -X 'POST' 'https://xxxxxxxxxxxxxxx/Upload?basketId=05aa7960-3f4a-42f4-8112-c22b5f96a9b5' -H 'accept: multipart/form-data' -F "upload_file_name=@$(pwd)/test.jpg"
10.0.0.2 - - [16/Jul/2023:11:07:05 +0000] "POST /Upload HTTP/1.0" 200 - "-" "-"
{
"path": "/Upload",
"headers": {
"host": "echo:8080",
"connection": "close",
"content-length": "653",
"x-real-ip": "10.0.0.2",
"x-forwarded-for": "10.0.0.2",
"x-forwarded-proto": "https",
"x-forwarded-ssl": "on",
"x-forwarded-port": "443",
"x-original-uri": "/Upload?basketId=05aa7960-3f4a-42f4-8112-c22b5f96a9b5",
"user-agent": "curl/8.0.1",
"accept": "multipart/form-data",
"content-type": "multipart/form-data; boundary=------------------------069346ec29b44fc6"
},
"method": "POST",
"body": "--------------------------069346ec29b44fc6\r\nContent-Disposition: form-data; name=\"upload_file_name\"\r\n\r\ntest.jpg\r\n--------------------------069346ec29b44fc6\r\nContent-Disposition: form-data; name=\"upload_content_type\"\r\n\r\nimage/jpeg\r\n--------------------------069346ec29b44fc6\r\nContent-Disposition: form-data; name=\"upload_tmp_path\"\r\n\r\n/upload/0002424835\r\n--------------------------069346ec29b44fc6\r\nContent-Disposition: form-data; name=\"upload_file_name.md5\"\r\n\r\n77eb043e0f79321adf635dc9d9895b00\r\n--------------------------069346ec29b44fc6\r\nContent-Disposition: form-data; name=\"upload_file_name.size\"\r\n\r\n679\r\n--------------------------069346ec29b44fc6--\r\n",
"fresh": false,
"hostname": "echo",
"ip": "10.0.0.2",
"ips": [
"10.0.0.2"
],
"protocol": "https",
"query": {
"basketId": "05aa7960-3f4a-42f4-8112-c22b5f96a9b5"
},
"subdomains": [],
"xhr": false,
"os": {
"hostname": "xxxxxxxxxxxxxxxxxx"
},
"connection": {}
}
10.0.0.2 - - [16/Jul/2023:11:08:37 +0000] "POST /Upload?basketId=05aa7960-3f4a-42f4-8112-c22b5f96a9b5 HTTP/1.0" 200 - "-" "curl/8.0.1"
My guess is that the C# client is creating invalid MultipartFormData and therefore Nginx is not processing it correctly, but why? Everything C# does for the request is hidden in the MultipartFormDataContent.
Expected behavior:
- Client creates MultipartFormData with given image
- nginx saves image
- nginx sends metadata to ASP.Net Server
Tried:
- Successfully Post image as MultipartFormData using curl (Expected behavior)
- Changes in C# Client:
- Manually set content type (nginx rejects it)
- Escape file and file name (no difference)
- Set manual boundary (no difference)
- Manually define
Content-Disposition(no difference)
I tried the (somewhat hacky) solution from this post https://stackoverflow.com/a/29118333/22235181, where the Form Data is manually combined and it worked. I'm using this solution, but I'm still curious why MultipartFormDataContent and Nginx-upload don't play along.
