iOS: What is IOFence and why is it freezing our app? - ios

Question

We have an AR project (built in Unity) that's running 24/7 on a number of iPads Pros (12.9, gen 3) in a museum. It's been live for 2.5 years now. We've always had the occasional freeze (on iOS 13 and below), but lately they have gotten exponentially worse (on iOS 14.4.2 and 14.6).
The problem manifests as follows, at completely random times: our app keeps running in the background (we can hear sound effects and see database activity continuing), but it stops getting redrawn, causing the screen to freeze on whatever frame was drawn last.
We're getting a gpuevent .ips log at freeze time (see below), which has information about an IOSurface being blocked by IOFence. There's very little info online about these logs though, so we're sort of stuck with the analysis. From what little I can find, it seems like we have some resource, possibly a texture buffer, that gets accessed by two processes simultaneously, causing one of those processes (the one responsible for drawing the screen) to get killed by the OS. Does that sound plausible? And if so, how do we find out where in our code the issue is triggered?
Can anyone here interpret this log better than we can, or does anyone have suggestions as to our next steps?
{"bug_type":"284","timestamp":"2021-07-10 10:51:30.00 +0200","os_version":"iPhone OS 14.6 (18F72)","incident_id":"C3ABD477-2A61-4D43-A3B1-25D7CCD706E6"}
{
"process_name" : "hats",
"registers" : {
},
"timestamp" : 1625907090,
"analysis" : {
"iofence_list" : {
"iofence_num_iosurfaces" : 1,
"iofence_iosurfaces" : [
{
"iofence_current_queue" : [
{
"iofence_acceleratorid" : 1,
"iofence_backtrace" : [
-68148358856,
-68148357636,
-68144533400,
-68144532380,
-68144496896,
-68144598792,
-68144425788,
-68144434312
],
"iofence_direction" : 1
}
],
"iosurface_id" : 243,
"iofence_waiting_queue" : [
{
"iofence_acceleratorid" : 2,
"iofence_backtrace" : [
-68148358856,
-68148357636,
-68146420632,
-68154559340,
-68146423972,
-68146427004,
-68146495056,
-68154290536
],
"iofence_direction" : 2
},
{
"iofence_acceleratorid" : 1,
"iofence_backtrace" : [
-68148358856,
-68148357636,
-68144533400,
-68144532380,
-68144496896,
-68144598792,
-68144425788,
-68144434312
],
"iofence_direction" : 1
}
]
}
]
},
"fw_ta_substate" : {
"slot0" : 0,
"slot1" : 0
},
"fw_power_state" : 0,
"fw_power_boost_controller" : 0,
"guilty_dm" : 1,
"fw_power_controller_in_charge" : 0,
"fw_cl_state" : {
"slot0" : 0
},
"fw_perf_state_lo" : 1,
"fw_ta_state" : {
"slot0" : 0,
"slot1" : 0
},
"signature" : 625,
"fw_power_substate" : 4,
"command_buffer_trace_id" : 490782964,
"fw_perf_state_select" : 0,
"restart_reason" : 7,
"fw_3d_state" : {
"slot0" : 0,
"slot1" : 0,
"slot2" : 0
},
"fw_gpc_perf_state" : 0,
"fw_perf_state_hi" : 1,
"fw_power_limit_controller" : 7,
"restart_reason_desc" : "blocked by IOFence"
}
}