Logout seems to not invalidate session in Azure Static Web Apps

Question

I created a static Web site using Azure Static Web Apps (under the Free plan for now -- not sure if that's relevant to the issue at hand.) I can access the Web site through an ***.azurestaticapps.net URL.

I then tried to test Azure Active Directory authentication per Microsoft's instruction from this page: https://learn.microsoft.com/en-us/azure/static-web-apps/authentication-authorization

My problem is that logging out does not seems to work as I would expect on a normal Web site. I could log out, but when I tried to login again (hoping to use a different Azure AD account,) the Microsoft login screen flashes by and I got logged right back in with the user that I just logged out from. Switching to a different tab doesn't help.

Steps to repro:

1. Navigate to /.auth/login/aad and login using a Microsoft Account. Say NO to 'Stay signed in?' prompt.
2. Navigate to /.auth/me to see the basic information on the logged in account to prove that I'm in the logged in state.
3. Navigate to /.auth/logout to logout. Immediately, navigate to /.auth/me again to confirm that my static Web App regards me as 'logged out.' I'd see this:

{"clientPrincipal": null}

4. Navigate to /.auth/login/aad again. Microsoft login page flashes by, and I am logged right back in with the previously logged out user.

Things that sort of worked

Any of the two actions below alone seems to make the browser forget my logged-in state:

1. Close the whole browser and relaunch it. I'd get asked to pick the previous user (and then enter the password) or choose a new user. This sort of works but reminds me of some Web sites 15-20 years ago that said 'For security reasons, please don't forget to close your entire browser after logging out from this one tab.'
2. Open a new tab in the same browser, and navigate to hotmail.com. That tab will enjoy my logged-in state from the Static Web Apps tab. I'd see my mails right away. Then I log out from the hotmail.com tab, switch back to ***.azurestaticapps.net tab and see that I am still logged into my Static Web App. Good! Then if I log out from my Static Web App and try to log back in, it has forgotten my logged-in state this time. In other words, logging out from the hotmail.com tab is somehow more powerful.

I also tested /.auth/login/google too and the same problem arises! So the issue seems to be on the Azure Static Web Apps side, not how idPs handle their logout process.

Am I missing anything obvious?