Custom response headers not send with 404 status code, ASP.NET MVC - asp.net-mvc

Question

I have an API controller (.NET Framework 4.8 and IIS server) that returns a file or 404 if the file is not found. I added some custom headers to the response just to specify what was the problem (I'm than using this information on FE app to show some specifying messages to user).
However, when I call the API, I don't get the custom headers when returning 404. I tried changing the status code to 200, 204 and it works.
The controller method looks like this:
[HttpPost]
public ActionResult FileDownload(DataModel data)
{
try
{
return DownloadFile(data);
}
catch
{
}
Response.Headers.Add("x-custom-header", "custom header content");
return new HttpStatusCodeResult(404);
}
The FE method for fetching the file:
fetch(props.href)
.then((response) => {
let customHeader= response.headers.get("x-custom-header");
console.log(customHeader);
if (response.status !== 200) {
throw response.status;
}
return response.blob();
})
.then((blob: Blob) => {
let fileUrl: string = window.URL.createObjectURL(blob);
let a: HTMLAnchorElement = document.createElement("a");
a.href = fileUrl;
a.download = fileName;
document.body.appendChild(a);
a.click();
document.body.removeChild(a);
setLinkText(linkTextSuccess);
})
.catch((error) => {
console.log(error);
setLinkText(linkTextError);
});
Why is it that with 404 status code, the headers are not sent? Could I somehow enforce that?
Thank you for your time!