Unexpected failure of ajax operation

Question

We use ajax in a number of places on our website. One instance sometimes acts irregularly as if the ajax call failed. It is just in one particular location--all the others work as expected.

When the failure occurs, the user gets an error message that the operation failed.

We cannot reproduce the failure. It always just works for us. Most users do not experience the issue. The few that do sometimes report that they tried it again later and it worked.

We can simulate the failure by renaming the ajax script. Then when the operation is attempted, the ajax call fails and we get the same error message. Yet we don't have any reason to believe that the error is caused by inability to access the ajax script, although it could be. If that were the case, then the same issue should occur with our other ajax scripts.

We have come up with a workaround. If the user goes into incognito mode before attempting the operation, the problem is avoided.

The incognito mode workaround suggests that it might be related to a cookie. Accordingly, we have attempted to interfere with cookies in some way to try to recreate the problem. Nothing has succeeded.

To assist with troubleshooting, we have added additional diagnostic information to the error message, but so far nothing has come out of this other than information similar to "Error". Here is some code that shows how we are trying to get more information:

error: function (jqXHR, textStatus, errorThrown)
{
    // alert('read-unread error');
    ajaxError('Read-Unread', textStatus, errorThrown);  // TODO comment out to suppress the error reporting
},

For completeness, I have included the full javascript function below.

To set up a simple test environment to demonstrate the issue will not work very well, as it would simplify so much from the real environment as to destroy its test value. We think the only reasonable test would be to simulate a use case as the users are doing. Unfortunately that is complex, as it involves by creating a profile on our system, attempting to use the system to communicate with other members, having some response communication from the other members and then try to reply to such a communication. That re-creates the the use case, and the place where the failure occurs. Again, unfortunately, we suspect that it would still never trigger the failure for us--the toy system would also "just work".

It would be one thing it it happened to every user every time, but it does not. Only some users some of the time, and only for one ajax usage out of many on our site. Once it starts happening, at least for a while it is repeatable for that user.

My questions:

1. How could we create a stripped down version of the environment so we could make it worthwhile for the stackoverflow community to take a look at it, short of simulating use of our website?

2. Does the "incognito mode" workaround provide any insight to understand what is going wrong?

3. Apart from diagnosing code that in most cases works correctly, it would be helpful to get insight on other ajax failure modes. As a made up example, such as "certain systems are too slow and the ajax times out".

[The javascript function. Note: for some reason I cannot get the first few lines of the function into the code markup.]

function toggleReadUnread(usertypeid, opp_user_label_short, principalid, repid, contactuid, value) {
    var new_response = (value) ? 'read' : 'unread';
    var prefix = '#' + contactuid;

    $('#ur-read-unread-text').html(new_response);
    $('#read-unread-dialog').dialog
    ({
        show: "fade",
        title: "Change to " + new_response + "?",
        modal: true,
        resizeable: false,
        width: getDialogWidth(400, .9),
        maxWidth: 400,
        fluid: true,
        buttons:
        {
            Ok: function ()
            {
                $(this).dialog('close');
                $.ajax(
                {
                    // async: false 
                    type: 'POST',
                    url: '/ajax/set-var.php',
                    // url: '/ajax/tr-toggle-status.php',
                    data:
                    {
                        'which_function': 'read-unread',
                        'principalid': principalid,
                        'repid': repid,
                        'usertypeid': usertypeid,
                        'value': value
                    },
                    error: function (jqXHR, textStatus, errorThrown)
                    {
                        // alert(jqXHR.responseText);
                        ajaxError('Read-Unread', textStatus, errorThrown);
                    },
                    success: function (data, textStatus, jqXHR)
                    {
                        if (data)
                        {
                            var connector = '\', \'';
                            if (value)
                            {
                                $(prefix + '-unread').remove();
                                $('<i id="' + contactuid + '-read" class="fa fa-envelope-open tooltip" onclick="toggleReadUnread(\'' + usertypeid + connector + opp_user_label_short + connector + principalid + connector + repid + connector + contactuid + '\'' + ', false);"><span id="' + contactuid + '-read-unread-tt" class="tooltiptext">You have no unread messages from this ' + opp_user_label_short + '</span>').insertAfter(prefix + '-read-unread-placeholder');
                            }
                            else
                            {
                                $(prefix + '-read').remove();
                                $('<i id="' + contactuid + '-unread" class="fa fa-envelope tooltip" onclick="toggleReadUnread(\'' + usertypeid + connector + opp_user_label_short + connector + principalid + connector + repid + connector + contactuid + '\'' + ', true);"><span id="' + contactuid + '-read-unread-tt" class="tooltiptext">You have a new message from this ' + opp_user_label_short + '</span>').insertAfter(prefix + '-read-unread-placeholder');
                                $('.tc-' + contactuid).hide();
                            }
                            jConfirm('Success', 'Messages marked "' + new_response + '"');
                        }
                    }
                })
            },
            Cancel: function () {
                $(this).dialog('close');
            }
        }
    })
 }

EDIT 1:

1. We are still having this issue with users. We still cannot reproduce it internally.

2. Going incognito prevents the problem from occurring. While this should be a great clue, it has not helped us so far. It seems it points to session and caching issues. We have ruled out stale cache issues.

3. It is intermittent, and only affects a small number of users. And later even that user will no longer have the problem. Again seems like cache-related.

4. We added diagnostic info and sometimes is says "Forbidden" in the ajax errorThrown. Checking on that implies it has to do with cross domain operations. There is no cross domain activity going on; all requests are to our own server.

5. We have asked some users about their extensions. In one case, they had no extensions installed; in another case, they only had the Grammarly extension for Chrome. We tested with that extension and still could not reproduce the issue.

6. When the error occurs, a dialog is displayed. After clicking "OK" on that dialog, the rest of the operation should just proceed as there is nothing that dialog blocks. However, users report they cannot complete the operation at that point when not in incognito mode.

EDIT 2:

I started looking into the https-access.log. While most accesses to the ajax file return 200, there are cases where 302 is returned. The implication is that "file is moved". But the file has not moved and shortly thereafter normal access resumes.

Here is an extract from the access log, showing a series of 200 responses, with some 302s interspersed, and then resuming 200.

108.221.39.97 - - [13/Jan/2023:10:07:52 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
108.221.39.97 - - [13/Jan/2023:10:08:34 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
108.221.39.97 - - [13/Jan/2023:10:09:56 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
73.176.158.231 - - [13/Jan/2023:10:10:21 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
74.73.226.238 - - [13/Jan/2023:10:15:07 -0600] "POST /ajax/set-var.php HTTP/2.0" 302 - "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
184.178.239.162 - - [13/Jan/2023:10:15:18 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?reply=tr-28560-80354" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36 Edg/108.0.1462.76"
74.73.226.238 - - [13/Jan/2023:10:15:20 -0600] "POST /ajax/set-var.php HTTP/2.0" 302 - "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
24.144.188.195 - - [13/Jan/2023:10:15:22 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
74.73.226.238 - - [13/Jan/2023:10:15:28 -0600] "POST /ajax/set-var.php HTTP/2.0" 302 - "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
74.73.226.238 - - [13/Jan/2023:10:15:34 -0600] "POST /ajax/set-var.php HTTP/2.0" 302 - "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
74.73.226.238 - - [13/Jan/2023:10:15:37 -0600] "POST /ajax/set-var.php HTTP/2.0" 302 - "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
74.73.226.238 - - [13/Jan/2023:10:15:49 -0600] "POST /ajax/set-var.php HTTP/2.0" 302 - "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
72.38.12.208 - - [13/Jan/2023:10:15:54 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?reply=tr-43850-7444" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36 Edg/108.0.1462.76"
74.73.226.238 - - [13/Jan/2023:10:16:07 -0600] "POST /ajax/set-var.php HTTP/2.0" 302 - "https://www.rephunter.net/track-relationships.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
74.73.226.238 - - [13/Jan/2023:10:16:19 -0600] "POST /ajax/set-var.php HTTP/2.0" 302 - "https://www.rephunter.net/track-relationships.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
108.221.39.97 - - [13/Jan/2023:10:17:23 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
108.221.39.97 - - [13/Jan/2023:10:17:28 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
72.38.12.208 - - [13/Jan/2023:10:17:49 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?reply=tr-43850-7444" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36 Edg/108.0.1462.76"
108.221.39.97 - - [13/Jan/2023:10:17:58 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
24.45.242.118 - - [13/Jan/2023:10:20:41 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
24.45.242.118 - - [13/Jan/2023:10:21:10 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"

EDIT 3:

We have a candidate solution inspired by the comment from @ibrahim to the answer from Lajos Arpad below. In short, adding retry functionality to the ajax call as shown at https://forum.framework7.io/t/app-request-what-is-the-best-way-to-retry-an-ajax-request-using-app-request/4759, which appears to be based on What's the best way to retry an AJAX request on failure using jQuery?.

After we confirm that this fix works, which will take some time of "no reported errors", I would mark it as the "answer", should @ibramim post his comment about retries as an answer.

Answer 1

Symptom

You have an issue with an AJAX script that generally works, but for some (rare) users sometimes the file is briefly not found.

Work-around

Using incognito mode seems to work this around.

Are we sure about the work-around?

There is some slight chance that the work-around coincides but does not cause the resuming of the AJAX. It is possible therefore that very rarely the file is unavailable (ex. there is an intermittent script rebuilding/minimizer that temporarily removes the file in order to move the new version in its place) for a very brief amount of time and users are experiencing the problem you have mentioned, yet, by the time they try to reach it via incognito mode, the file is already restored, in short: it is possible that the incognito mode is only coinciding with the restoration of the file, but not causing this.

Disproving this could be tried by trying out the incognito work-around when the issue happens, see that it works and then try in non-incognito mode and see it still not working (after a page reload, of course). In the latter part of the answer I will presume that incognito is causing the issue to disappear for the user and not only coincides with the disappearance of the problem.

Possible causes

Cookies

Some cookies are problematic. For example, the cookie of the user expired and cannot validly be used, yet, the page has started to load when the cookie was still valid, yet, the server throws a file moved error for the request to the file because it is programmed to serve this request only for logged in users. Also, it's possible that /ajax/set-var.php on its own tries to load some file that surely exists for existent session, but the current session has been destroyed and the file to be reached no longer exists.

Cache

Client-side caches may have an exact filename or path whereas the file has been renamed or moved. It is also possible that the file was requested during a build process, when the file was being written and that incomplete file has been cached into the browser. In your case this does not seem to be the problem, since you have a php file that's being moved according to the error message.

Server-side may have cached some information for you that has become deprecated/invalid since then. For instance, a static page generated for the user's session to avoid repeating the same long request logic.

sessionStorage / localStorage

It's quite possible that some values are stored in sessionStorage and are lost when the session times out, yet the page is still loaded, or some values specific to a session are stored in localStorage and no longer valid. In both cases the request is invalid and on the server-side there is some error being thrown, resulting file not found.

What could file not found mean?

It means that some file is not found at the expected location. So, in order to fix it:

• check the error logs as well around the time when the error happens and see whether some exception/error/notice is being stored
• study all jobs that are writing critically important files, like cron jobs, build processes, minifiers, etc.
• study /ajax/set-var.php, particularly the section that handles this request
• study carefully https://www.rephunter.net/track-relationships.php

Answer 2

There could be a ton of things going wrong in backend, so with that in mind, my first approach would be to simulate 'fresh request', like in cognito as much as possible:

1- Make a function to generate a random hexstring, you will use this to append it as a query string to your request so it cannot be cached:

function genHex(n){
    return Array.from({length:n}).reduce((ac,d) => ac += (Math.random() * 1e6 | 0).toString(16),"")
}

genHex(2)//'7ec05646dc'

2- Add the request headers to invalidate cache:

            $.ajax({
                    asynch: false,//what the hell is this 'asynch'??
                    type: 'POST',
                    url: '/ajax/set-var.php?' + genHex(2),
                    headers:{
                      "Cache-control": "no-cache, no-store, must-revalidate",
                      "Pragma": "no-cache",
                      "Expires": "0"
                    },

The idea is that, if your server disregards the cache control headers for whatever reason, at least we have the random query parameter to simulate similar behavior. Try it out and let me know, otherwise I will delete.

3- Wrap everything in a function that self executes itself on error with a set limit.

function reAjax(limit = 2, firstCall = true, ...rest){
    if(!limit){return}
    $.ajax({
        asynch: false,//what the hell is this 'asynch'??
        type: 'POST',
        url: `/ajax/set-var.php${firstCall ? "" : `?${genHex(2)}`}`,
        headers:{
          "Cache-control": "no-cache, no-store, must-revalidate",
          "Pragma": "no-cache",
          "Expires": "0"
        },
        error: (...rest) => reAjax(--limit, false, ...rest)  
    })
}

The function will call itself without query parameters (for cache busting) the first time and it will keep calling on error with cache busting until limit is hit. rest is the parameters passed on jquery error here

PS: One more thing to add, if your backend is controlled by a third party virtual machine, sometimes they put rate-limiting on how many POST requests one can make. Essentially they act as a reverse proxy between you and the client. So make sure these limitations are NOT the case