Why does HttpContext.User.Identity.Name contain the wrong name? - asp.net-mvc

Question

I have a .NET 5 MVC application that periodically checks access based on the HttpContext.User.Identity.Name value. The app uses Azure AD. Sometimes as I switch between users (Signing out and Signing in), the HttpContext.User.Identity.Name value will contain the previous user's name and the current user will have the access of the previous user.
Even more strange is the the login shown is correct.
This is just the boiler plate code generated by visual studio:
#using System.Security.Principal
<ul class="navbar-nav">
#if (User.Identity.IsAuthenticated)
{
<li class="nav-item">
<span class="navbar-text text-light">Hello #User.Identity.Name!</span>
</li>
<li class="nav-item">
<a class="nav-link text-light" asp-area="MicrosoftIdentity" asp-controller="Account" asp-action="SignOut">Sign out</a>
</li>
}
else
{
<li class="nav-item">
<a class="nav-link text-light" asp-area="MicrosoftIdentity" asp-controller="Account" asp-action="SignIn">Sign in</a>
</li>
}
</ul>
So #User.Identity.Name on the .cshtml is always correct, but HttpContext.User.Identity.Name on the controller is the previous (cached?) user.
Any clues to why the previous user gets cached?
Here is a sample of how I call it on the controller:
ViewData["User"] = HttpContext.User;
ViewData["Menu"] = await _menuService.GetReportMenuAsync(MenuService.ReportMenuItemOption.Genesis01, User.Identity.Name);
var userAccess = await _userAccessService.GetUserEntityAccessAsync(User.Identity.Name);
var model = new SomeModel
{
SearchDate = DateTime.Now.AddDays(-7),
Banks = userAccess.Banks,
ATMIds = userAccess.ATMs
};
return View(model);
This is in my StartUp:
services.AddControllersWithViews(options =>
{
var policy = new AuthorizationPolicyBuilder()
.RequireAuthenticatedUser()
.Build();
options.Filters.Add(new AuthorizeFilter(policy));
}).AddMicrosoftIdentityUI();