Why would output values from neural network increase / 'explode'? - machine-learning

Question

I'm training a neural network as part of reinforcement learning using PPO, and when the model is first initialised the outputs look 'normal' like this:
[-0.00611657 -0.01238834 0.00284606 0.02600986 -0.00587347 -0.00182077
-0.00975011]
But gradually the outputs increase and after a few hours of training look like this:
[-3119.771 1075.3921 513.0797 -3039.5288 -2695.9114 388.93564
-3112.1418 ]
The rewards from the environment is always between -5 to +5 and they're normalized:
rewards = (rewards - rewards.mean()) / torch.max(rewards.std(), torch.tensor(1e-5))
I'm thinking that points to an error somewhere? Any ideas of what could cause outputs to 'explode' like this?
Model architecture is as follows:
Sequential(
(0): Linear(in_features=174, out_features=500, bias=True)
(1): ReLU()
(2): Linear(in_features=500, out_features=500, bias=True)
(3): ReLU()
(4): Linear(in_features=500, out_features=500, bias=True)
(5): ReLU()
(6): Linear(in_features=500, out_features=500, bias=True)
(7): ReLU()
(8): Linear(in_features=500, out_features=500, bias=True)
(9): ReLU()
(10): Linear(in_features=500, out_features=500, bias=True)
(11): ReLU()
(12): Linear(in_features=500, out_features=7, bias=True)
)