.net API error when attempting to accept model with large file

Question

I have an API method that receives a null model parameter when a large file is passed to it.
I created a test client to test this endpoint. Both the test client and the API have these same identical models and are using .NET 4.5:

public class FilingPostModel
    {
        public string Id { get; set; }
        public string TypeId { get; set; }
        public string FirstName { get; set; }
        public string MiddleName { get; set; }
        public string LastName { get; set; }
        public string Suffix { get; set; }
        public string Line1 { get; set; }
        public string Line2 { get; set; }
        public string City { get; set; }
        public string State { get; set; }
        public string PostalCode { get; set; }
        public string Country { get; set; }
        public string Email { get; set; }
        public string PhoneNumber { get; set; }
        public string Comment { get; set; }
        public string DateSubmitted { get; set; }
        public string Summary { get; set; }
        public List<FilePostModel> FileData { get; set; }
    }

    public class FilePostModel
    {
        public string FileId { get; set; }
        public string FileName { get; set; }
        public string ContentType { get; set; }
        public string FileContent { get; set; }
        public string DateSubmitted { get; set; }
        public string ClassificationId { get; set; }     
    }

The test client is submitting this model:

City: "j"
Comment: null
Country: "United States"
Email: "test@test.tst"
FileData: Count = 1
TypeId: "f94e264a-c8b1-44aa-862f-e6f0f7565e19"
FirstName: "fname"
Id: null
LastName: "lname"
Line1: "testdrive 1"
Line2: null
MiddleName: null
PhoneNumber: "3748923798"
PostalCode: "12345"
State: "Pennsylvania"
Suffix: null
Summary: null

The FileData component has one item:

FileContent: "xY+v6sC8RHQ19av2LpyFGu6si8isrn8YquwGRAalW/6Q..."
ClassificationId: null
ContentType: "text/plain"
FileName: "large.txt"

This is the test clients method used to create and send the API request

public async Task<ActionResult> PostNewFiling(FilingPostModel model)
{
    Dictionary<string, string> req = new Dictionary<string, string>
        {
            {"grant_type", "password"},
            {"username", "some user name"},
            {"password", "some password"},
        };
    FilingApiPostModel postModel = new FilingApiPostModel(model);
    using (HttpClient client = new HttpClient())
    {
        client.Timeout = TimeSpan.FromMinutes(15);
        client.BaseAddress = new Uri(baseUrl);
        var resp = await client.PostAsync("Token", new FormUrlEncodedContent(req));
        if (resp.IsSuccessStatusCode)
        {
            TokenModel token = JsonConvert.DeserializeObject<TokenModel>(await resp.Content.ReadAsStringAsync());
            if (!String.IsNullOrEmpty(token.access_token))
            {
                foreach (HttpPostedFileBase file in model.Files)
                {
                    if (file != null)
                    {                                    
                        FilePostModel fmodel = new FilePostModel();
                        fmodel.FileName = file.FileName;
                        fmodel.ContentType = file.ContentType;
                        byte[] fileData = new byte[file.ContentLength];
                        await file.InputStream.ReadAsync(fileData, 0, file.ContentLength);
                        fmodel.FileContent = Convert.ToBase64String(fileData);
                        fmodel.ClassificationId = model.Classification1;
                        postModel.FileData.Add(fmodel);
                    }
                }
                
                client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", token.access_token);
                var response = await client.PostAsJsonAsync("api/Filing/PostFiling", postModel);    
                var responseBody = await response.Content.ReadAsStringAsync();

                if (response.IsSuccessStatusCode)
                    return Json(new { responseBody });
                else
                    return Json(new { error = true, message = "Error Uploading", obj = responseBody });
            }
        }
        return Json(new { error = true, message = "Error Uploading" });
    }
}

Here is the API method to receive this client request:

public async Task<StatusModel> PostFiling(FilingPostModel model)

Here is the maxAllowedContentLength setting in web.config:

<system.webServer>
    <security>
      <requestFiltering>
        <requestLimits maxAllowedContentLength="4294967295" />
      </requestFiltering>
    </security>
  </system.webServer>

The API model is always null in this test scenario. I'm receiving two types of errors:

1. Newtonsoft.Json - Array dimensions exceeded supported range
2. Newtonsoft.Json - Unexpected character encountered while parsing value: x. Path 'FileData[0].Bytes', line 1, position 517

The file size of this test file is 560 MB. I used Dummy File Creator to create it. Here's a sample of what the content looks like:

ůêÀ¼Dt5õ«ö.œ…Ȭ®ªìD¥[þ6\hW åz·cʾYP¸‡>•;,–@Ó¶ÿm™­fø@ÃNÇIäÀ¿Y4~ëÆÃc¥EWÀ_÷õ9%«éÀG!WBÍ*G2P×æŸ7ú‚{ÓêRúÅîµMZSªšpt6ä”Òø˜H

I have also tried using "fsutil file createnew" to create a test file but receive similar error.

Everything works properly when testing with a 256 MB file.

Thank you for your help.

Answer 1

you can add two attributes to your action:

[RequestFormLimits(MultipartBodyLengthLimit = 209715200)]
[FromBody]

Answer 2

If you need to post JSON I would change your FileData object to use a Base64 encoded string in place of the byte array.

Change the Bytes property in FileData to a string and then when creating the request change fmodel.Bytes = fileData to fmodel.Bytes = Convert.ToBase64String(fileData).

After deserialising your JSON you can convert the base64 back to a byte[] using Convert.FromBase64String(String).

Not only will this prevent you exceeding the array limit in JSON but will also reduce the payload size considerably.

Answer 3

Every implementation has its own limit for example PHP has 2MB by default you can increase it using Post_Max_Size

ASP.NET Core enforces 30MB any file bigger than default limitation can cause errors.

Here's Upload Large Files In ASP.NET Core!

For IIS Express, you need to add web.config to your project and add the following markup into it:

<system.webServer>
  <security>
    <requestFiltering>
      <requestLimits maxAllowedContentLength="209715200" />
    </requestFiltering>
  </security>
</system.webServer>

Answer 4

If the issue is being caused by the total size of the array, rather than the number of elements in the array, the OutOfMemoryException might be avoided by targeting 64-bit compilation and by updating your app.config:

<configuration>
  <runtime>
    <gcAllowVeryLargeObjects enabled="true" />
  </runtime>
</configuration>

gcAllowVeryLargeObjects element