what is delay between HttpPost being sent to server and server responding

Question

I'm an uploading a zipfile from a Java desktop application to an Httpserver (running Tomcat 7), Im using Apache httpClient 4.5.3 and I display a progress bar showing progress using this wrapper solution https://github.com/x2on/gradle-hockeyapp-plugin/blob/master/src/main/groovy/de/felixschulze/gradle/util/ProgressHttpEntityWrapper.groovy

So in my code Im updating progressbar every time the callback gets called

HttpEntity reqEntity = MultipartEntityBuilder.create()
        .addPart("email", comment)
        .addPart("bin", binaryFile)
        .build();

ProgressHttpEntityWrapper.ProgressCallback progressCallback = new ProgressHttpEntityWrapper.ProgressCallback() {

    @Override
    public void progress(final float progress) {
        SwingUtilities.invokeLater(
                new Runnable()
                {
                    public void run()
                    {
                        MainWindow.logger.severe("progress:"+progress);
                        Counters.getUploadSupport().set((int)progress);
                        SongKong.refreshProgress(CreateAndSendSupportFilesCounters.UPLOAD_SUPPORT_FILES);
                    }
                }
        );
    }
};

httpPost.setEntity(new ProgressHttpEntityWrapper(reqEntity, progressCallback));
HttpResponse response = httpclient.execute(httpPost);
HttpEntity resEntity = response.getEntity();
MainWindow.logger.severe("HttpResponse:"+response.getStatusLine());

This reports files uploaded as a percentage, but there is a sizeable delay between it reporting 100% creation and actually receiving http status from server.

07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.19408
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.40069
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.6073
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.81391
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.99768
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.99778
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.99789
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.999794
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.9999
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:100.0
07/07/2017 14.24.11:BST:CreateSupportFile:sendAsHttpPost:SEVERE: HttpResponse:HTTP/1.1 200 OK
07/07/2017 14.24.11:BST:CreateSupportFile:sendAsHttpPost:SEVERE: Unknown Request

Note is not due to my tomcat code doing much since I haven't yet implemented the tomcat code for this function so it just defaults to the "Unknown Request" code.

protected void doPost(javax.servlet.http.HttpServletRequest request, 

    javax.servlet.http.HttpServletResponse response)
                throws javax.servlet.ServletException, java.io.IOException
        {
            String createMacUpdateLicense   = request.getParameter(RequestParameter.CREATEMACUPDATELICENSE.getName());
            if(createMacUpdateLicense!=null)
            {
                createMacUpdateLicense(response, createMacUpdateLicense);
            }
            else
            {
                response.setCharacterEncoding("UTF-8");
                response.setContentType("text/plain; charset=UTF-8; charset=UTF-8");
                response.getWriter().println("Unknown Request");
                response.getWriter().close();
            }
        }

How can I more accurately report to the user when it will complete

Update I have now fully implemented the serverside, this has increased the discrepancy

    @Override
    protected void doPost(javax.servlet.http.HttpServletRequest request, javax.servlet.http.HttpServletResponse response)
            throws javax.servlet.ServletException, java.io.IOException
    {
        String uploadSupportFiles   = request.getParameter(RequestParameter.UPLOADSUPPORTFILES.getName());
        if(uploadSupportFiles!=null)
        {
            uploadSupportFiles(request, response, uploadSupportFiles);
        }
        else
        {
            response.setCharacterEncoding("UTF-8");
            response.setContentType("text/plain; charset=UTF-8; charset=UTF-8");
            response.getWriter().println("Unknown Request");
            response.getWriter().close();
        }
    }

private void uploadSupportFiles(HttpServletRequest request, HttpServletResponse response, String email) throws IOException
    {
        Part filePart;
        response.setCharacterEncoding("UTF-8");
        response.setContentType("text/plain; charset=UTF-8; charset=UTF-8");

        try
        {
            filePart = request.getPart("bin");
            String fileName = getSubmittedFileName(filePart);
            response.getWriter().println(email+":File:" + fileName);

            //Okay now save the zip file somewhere and email notification
            File uploads = new File("/home/jthink/songkongsupport");
            File supportFile = new File(uploads, email+".zip");

            int count =0;
            while(supportFile.exists())
            {
                supportFile = new File(uploads, email+"("+count+").zip");
                count++;
            }
            InputStream input;
            input = filePart.getInputStream();
            Files.copy(input, supportFile.toPath());

            Email.sendAlert("SongKongSupportUploaded:" + supportFile.getName(),  "SongKongSupportUploaded:" + supportFile.getName());
            response.getWriter().close();
        }
        catch(ServletException se)
        {
            response.getWriter().println(email+":"+se.getMessage());
            response.getWriter().close();
        }


    }

Answer 1

Assuming your server-side code just writes the uploaded file somewhere and responds something like "DONE" at the end, here is a rough timeline of what happens:

Bytes written to socket OutputStream
============================|
<--> Buffering              |
    Bytes sent by TCP stack |
    ============================
    <------> Network latency|
            Bytes received by Tomcat
            ============================
                            |           (Tomcat waits for all data to finish uploading
                            |            before handing it out as "parts" for your code)  
                            |            File written to local file on server
                            |            =====
                            |                
                            |                  Response "DONE" written by servlet to socket output
                            |                  ==
                            |                  <---> Network latency 
                            |                       == Response "DONE" received by client
                            |                         |
                            |                         |
  "100%" for entity wrapper ^             Actual 100% ^
                             Discrepancy
                             <----------------------->
                             "Twilight Zone" : part of discrepancy you cannot do much about.
                             (progress feedback impossible without using much lower level APIs)
                             <--------------------->

The scales are of course completely arbitrary, but it shows that there are several factors that can participate into the discrepancy.

Your server writes the file after receiving all bytes, but it does not make a big difference here.

So, the factors:

• (client side) Buffering (possibly at several levels) between the Java I/O layer and the OS network stack
• Network latency
• (server-side) Buffering (possibly at several levels) between the OS network stack and the Java I/O layer
• Time to write (or finish writing) zip file on disk
• Time to print response (negligible)
• Network latency
• (client side) Time to read response (negligible)

So you could take that discrepancy into account and adjust the "upload complete" step to 90% of the total progress, and jump from 90 to 100 when you get the final response. From 0% to 90% the user would see "Uploading", with a nice progress bar moving, then you show "Processing...", perhaps with a throbber, and when done, jump to 100%.

That's what many other tools do. Even when I download a file with my browser, there is a small lag towards the end, the download seems stuck at "almost" 100% for a second (or more on my old computer) before the file is actually usable.

If the "twilight zone" time is much higher than the upload time as perceived by your progress wrapper, you might have a problem, and your question would thus be "where does this delay come from?" (for now I don't know). In that case, please provide complete timings (& make sure client & server machines have their clocks synchronized).

---

If you reaaaally need a more accurate/smooth progress report towards the end, you will need a much more involved setup. You will probably need to use more low-level APIs on the server side (e.g. not using @MultipartConfig etc), in order to have your server do something like writing to disk as data is received (which makes error handling much more difficult), print a dot to output and flush, for every 1% of the file that is written to disk (or any other kind of progress you want, provided it's actual progress on server-side). Your client side would then have the ability to read that response progressively, and get accurate progress report. You can avoid threading on client side, it's fine to do this sequentially:

• POST data, report progress but scaled to 90% (ie if wrapper says 50%, you report 45%)
• when done, start reading output from server, and report 91%, 95%, whatever, up until 100%.

Even with that I'm not sure it's possible to display progress info for all the steps (especially between 100% sent and first byte the server can possibly send), so maybe even that extremely complex setup would be useless (it could very well stall at 90% for a moment, then go 91/92/...99/100 in an instant).

So really at this point it's probably not worth it. If you really have a 17s step between last byte sent by client, and response received, something else is off. Initially I was assuming it was for humongous files, but since then you said your files were up to 50MB, so you might have something else to look at.

Answer 2

some of the server-side code might change depending on how the chunk data is represented, but the concept is roughly the same. Let's say you are uploading a 10MB file and you have your chunk size set to 1MB. You will send 10 requests to the server with 1MB of data each. The client is actually responsible for breaking all of this up. That is what you will do in Javascript. Then, each request is sent up via HttpRequest along with some other data about the file, chunk number and number of chunks. Again, I use the plupload plugin which handles this for me so some of the Request data may differ between implementations.

The method I am showing you is part of a Webservice which outputs JSON data back to the client. Your javascript can then parse the JSON and look for an error or success message and act appropriately. Depending on your implementation, the data you send back might be different. The javascript will ultimately handle the progress bar or percentage or whatever, increasing it as it gets successful chunk uploads. My implementation for my project lets plupload deal with all that, but maybe that article I gave you will give you more control over the client-side.

protected void Upload()
{
    HttpPostedFile file = Request.Files[0];
    String relativeFilePath = "uploads/";
    try
    {
        if(file == null)
            throw new Exception("Invalid Request.");
        //plupload uses "chunk" to indicate which chunk number is being sent
        int chunk = (int)Request.Form["chunk"];
        //plupload uses "chunks" to indicate how many total chunks are being sent
        int chunks = (int)Request.Form["chunks"];
        //plupload uses "name" to indicate the original filename for the file being uploaded
        String filename = Request.Form["name"];
        relativeFilePath += filename;


        //Create a File Stream to manage the uploaded chunk using the original filename
        //Note that if chunk == 0, we are using FileMode.Create because it is the first chunk
        //otherwise, we use FileMode.Append to add to the byte array that was previously saved
        using (FileStream fs = new FileStream(Server.MapPath(relativeFilePath), chunk == 0 ? FileMode.Create : FileMode.Append))
        {
           //create the byte array based on the data uploaded and save it to the FileStream
           var buffer = new byte[file.InputStream.Length];
           file.InputStream.Read(buffer, 0, buffer.Length);
           fs.Write(buffer, 0, buffer.Length);
        }

        if((chunks == 0) || ((chunks > 0)&&(chunk == (chunks - 1))))
        {
          //This is final cleanup.  Either there is only 1 chunk because the file size
          //is less than the chunk size or there are multiple chunks and this is the final one
          //At this point the file is already saved and complete, but maybe the path is only
          //temporary and you want to move it to a final location
          //in my code I rename the file to a GUID so that there is never a duplicate file name
          //but that is based on my application's needs
          Response.Write("{\"success\":\"File Upload Complete.\"}");
        }
        else
          Response.Write("{\"success\":\"Chunk "+chunk+" of "+chunks+" uploaded.\"}");
    }
    catch(Exception ex)
    {
        //write a JSON object to the page and HtmlEncode any quotation marks/HTML tags
        Response.Write("{\"error\":\""+HttpContext.Current.Server.HtmlEncode(ex.Message)+"\"});
    }
}