0

I am working on an asp.net core 3.1 application using mvc where I'm supposed to upload files of up to 200mb. Uploading up to 100mb works but anything above gives me a 404.

I'll say straight away that the code is not written by me and uses Kapitan to drive most of the configuration, which is completely new to me.

this is a mock of the form:

        @using (@Html.BeginForm("Create", "Model", FormMethod.Post, htmlAttributes: new { @class = "form", enctype = "multipart/form-data" }))
        { 
            <div class="form-group">
                @Html.Label("x File")
                @Html.TextBoxFor(x => x.Model.x, htmlAttributes: new
                {
                    type = "file",
                    accept = ".x",
                    required = "true"
                })
            </div>
        }

controller:

    [HttpPost]
    [RequestFormLimits(MultipartBodyLengthLimit = 568435456)]
    public async Task<IActionResult> Create(X x)
    {
    }

things I've tried: add web.config

<?xml version="1.0" encoding="utf-8"?>

        <configuration>
            <system.web>
                <httpRuntime maxRequestLength="307200" />
            </system.web>
            <system.webServer>
                <security>
                    <requestFiltering>
                        <requestLimits maxAllowedContentLength="524288000" />
                    </requestFiltering>
                </security>
            </system.webServer>
        </configuration>

startup:

        services.Configure<FormOptions>(options =>
        {
            options.MultipartBodyLengthLimit = 568435456;
        });

Changing the configuration of the yml file for the proxy-body-size removed the 413 I used to get from nginx but not the 404.

UPDATE

In incognito the response becomes:

502 Bad Gateway nginx/1.17.10

What could the causes be? I tried everything I could find online.

Thank you in advance.

Mirko Sangrigoli
  • 1
  • 4
  • Does it work on your local machine? (without involving IIS, nginx or other reverse proxies) – abdusco Jun 25 '21 at 21:52
  • Yes it goes past the controller (I can't test that it goes to the s3 storage as that's also configured in the kapitan) @abdusco – Mirko Sangrigoli Jun 25 '21 at 22:02
  • Getting 404 is very peculiar and not really a response any sane service would return. I'd try ruling out anything wrong with the app first by connecting it to my own S3 bucket, for example, then work my way up service by service. – abdusco Jun 25 '21 at 22:07
  • You could spin up a local MinIO instance really quick and test out this theory. https://docs.min.io/docs/minio-quickstart-guide.html – abdusco Jun 25 '21 at 22:08
  • I'll try that. What's interesting is that I can upload two files of 75mb each simultaneously but not one of over 100mb. – Mirko Sangrigoli Jun 25 '21 at 22:09
  • @abdusco I've unplugged all services from the controller. so pretty much now the form just post to a controller but the controller should only return to the index, without leveraging anything else. Still 404 – Mirko Sangrigoli Jun 26 '21 at 20:27
  • So the 404 error is coming from the app then? When you check the network connections on the browser console, do you see a redirect that's not supposed to be there? – abdusco Jun 26 '21 at 20:45
  • @abdusco no redirects that I can see, I'm afraid. this is what I get from the browser: Failed to load resource: the server responded with a status of 404 () chrome-error://chromewebdata/:1 VM9:7146 crbug/1173575, non-JS module files deprecated. – Mirko Sangrigoli Jun 26 '21 at 20:55
  • I meant the network activity, not the javascript console. https://developer.chrome.com/docs/devtools/network/#load – abdusco Jun 26 '21 at 22:14
  • @abdusco yes I know, I was just adding that I noticed those as well. No redirects in the network connectivity. Just comes out in red as (failed). Tried on both Chrome and Edge – Mirko Sangrigoli Jun 26 '21 at 22:30
  • Can you export an HAR of all network requests? https://auth0.com/docs/troubleshoot/tools/generate-and-analyze-har-files#google-chrome. You can use something like https://wormhole.app/ to share the file. Or just use the email in my profile – abdusco Jun 26 '21 at 22:34
  • Oh wait. I hope that network dump doesn't include the 100mb+ file you're trying to upload... – abdusco Jun 26 '21 at 22:43
  • Will do in the morning. I really appreciate all the support. Thank you – Mirko Sangrigoli Jun 26 '21 at 22:52
  • @abdusco I can't unfortunately share the file you requested as too much sensitive info is contained there but when using Incognito I get this response instead: 502 Bad Gateway nginx/1.17.10 – Mirko Sangrigoli Jun 27 '21 at 20:33
  • It's ok. HTTP 502 means nginx cannot talk to your app (or whatever service is running at that address). That means your app can not boot up and start listening to requests (or nginx is trying to proxy the requests to the wrong address). If the app can't start, that usually implies invalid configuration, but could be a lot of things. – abdusco Jun 27 '21 at 20:45
  • @Abdusco it only happens when 1 single file is above 100MB. The sum of the files (for example 2 75mb files) can instead be above it. Does that narrow it down? – Mirko Sangrigoli Jun 27 '21 at 20:49
  • Have you configured nginx to accept large requests? Maybe the default is too low. – abdusco Jun 27 '21 at 20:51
  • Also, how is nginx involved in the whole infrastructure? Is it running in front of IIS or vice versa? But it doesn't really matter. What matters is every reverse proxy from the edge to your app must accept responses large enough for your needs. – abdusco Jun 27 '21 at 20:52
  • Another question: when you say uploading 2 (x75MB) files simultaneously, do you mean in the same request? Like two sibling parts of a single `multipart/form-data` request? Or are they uploaded in two distinct requests? If the latter is the case, that doesn't tell us anything (because each individual request is below the phantom limit). If it's the former, then somewhere some proxy is holding you back. – abdusco Jun 27 '21 at 20:58
  • @abdusco The form accepts 4 files (of different extensions) if the sum of those is above 100Mb it still works. the max size was configured with this: nginx.ingress.kubernetes.io/proxy-body-size and set to 500m (which fixed the 513 I used to get). – Mirko Sangrigoli Jun 27 '21 at 21:01
  • If some actor is able to differentiate between a form submission of one 100MB and two 75MB files, then it's trying to parse the multipart request. That's usually your app, because it's noone else's responsibillity. But now the weird thing is you say your app works on local machine, so it must be parsing the request correctly. One last thing that came into my mind is how much RAM did you assign to your pod? Maybe the request is not being streamed and somehow buffered to RAM, and that causes the pod to malfunction? – abdusco Jun 27 '21 at 21:05
  • @abdusco that's a good question. I have no clue. I'm actually a backend developer with no K8s experience, but have been handed over some code to fix, hence the huge confusion...! I have no idea where to look for the RAM configurations. I'm so sorry – Mirko Sangrigoli Jun 27 '21 at 21:09
  • http://www.binaryintellect.net/articles/612cf2d1-5b3d-40eb-a5ff-924005955a62.aspx. It's the same 404 error. I think your configuration isn't taking effect. – abdusco Jun 27 '21 at 21:09
  • Try injecting `IOptions` in a controller and check the value of `MultipartBodyLengthLimit` inside the debugger. Have you tried configured Kestrel too? See https://learn.microsoft.com/en-us/aspnet/core/fundamentals/servers/kestrel/options?view=aspnetcore-5.0 for `MaxRequestBodySize` – abdusco Jun 27 '21 at 21:12
  • @abdusco 568435456 – Mirko Sangrigoli Jun 27 '21 at 21:17
  • How about Kestrel? I've edited my last comment, maybe you overlooked it – abdusco Jun 27 '21 at 21:17
  • ah sorry I only saw it now. It was one of my attempts yes. – Mirko Sangrigoli Jun 27 '21 at 21:20
  • Try increasing log level of `Microsoft.AspNetCore` to `Debug` (maybe even to `Trace`) to see Kestrel's logs. If a request is being dropped, it has to be logged. – abdusco Jun 27 '21 at 21:22
  • before I do that, if I add kestrel options, this happens: System.InvalidOperationException: 'Application is running inside IIS process but is not configured to use IIS server.' at the "Main" – Mirko Sangrigoli Jun 27 '21 at 21:30
  • Forget Kestrel, because IIS is playing the server role. But increasing the log level is still a good idea. – abdusco Jun 27 '21 at 21:54
  • Might be a silly question but, I can't see the logs once I deploy so, how can I access them after increasing them? – Mirko Sangrigoli Jun 27 '21 at 21:58
  • boundary=----WebKitFormBoundary(some random letters) 164677592 Could this be the issue? – Mirko Sangrigoli Jun 27 '21 at 22:07
  • `---boundary\nContent-Disposition... \r\n\r\n ` That's [multipart boundary](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/POST#example) which marks the start of a new field in a multipart form request. If that's from the browser, it doesn't mean anything, every multipart request has those. But if it's from the logs, the error / bug shouldn't be too far from that message. – abdusco Jun 28 '21 at 06:55
  • Regarding how to view the logs, search for `kubernetes tail logs` – abdusco Jun 28 '21 at 06:56
  • @abdusco finally a big step forward. The issue is an Out of memory exception. I am compressing the large file with gzip on upload and that throws an error (only on deployment, not on my machine). – Mirko Sangrigoli Jun 28 '21 at 12:51
  • That good news! I won't say I told you so :). You should be able to do streaming compression to relieve memory pressure: https://learn.microsoft.com/en-us/dotnet/api/system.io.compression.gzipstream?view=net-5.0 – abdusco Jun 28 '21 at 13:20
  • @abdusco that's exactly the problem ahahah. The compression IS the issue that causes the OOM exception. I don't understand why it happens on the deployed version though, in my machine it works fine. Thank you for the tip, they were invaluable. – Mirko Sangrigoli Jun 28 '21 at 13:39
  • You're welcome. I doubt zipping is the cause for OOM, it is "Streaming", after all. Some method downstream could be buffering the zip stream before passing it to something else. You need to profile your app (locally) and see which procedure causes large memory allocations. https://learn.microsoft.com/en-us/visualstudio/profiling/memory-usage?view=vs-2019 – abdusco Jun 28 '21 at 14:04
  • @abdusco it goes up to 825,800.61KB when compressing, which might be the explanation – Mirko Sangrigoli Jun 28 '21 at 14:38
  • That's concerning. You might be doing something wrong: https://stackoverflow.com/a/49991245/5298150. – abdusco Jun 28 '21 at 14:51
  • You can also selectively zip files: you can't really zip videos, images and archives, because they're already compressed. Also lowering compression level might help. You don't get much out of high levels, anyway. – abdusco Jun 28 '21 at 14:52
  • I don't see an issue with what I did as it seems to fit with the instructions on the link. I'm checking the previous dev code to find possible memory leaks. What range should the heap size stay within? – Mirko Sangrigoli Jun 28 '21 at 15:25

1 Answers1

0

For people who may come to face this issue, the problem was caused by OOM as somewhere in the code there were poorly managed streams. The 404 response seems to have been caused by the Pod failing before the response could be completed (as the memory limit was hit). Many thanks to @Abdusco for his invaluable help debugging the issue.

Mirko Sangrigoli
  • 1
  • 4