How to prevent my model from outputting zero-vectors while training using one-hot encoded vectors? - machine-learning

Question

I have been training a model for a study on one-shot learning.
It has 19280 examples in the training dataset (basically the popular Omniglot dataset), and a 300-length vector for each data sample.
The model consists of the following architecture-
----------------------------------------------------------------
Layer (type) Output Shape Param #
================================================================
Conv2d-1 [-1, 32, 50, 50] 1,600
BatchNorm2d-2 [-1, 32, 50, 50] 64
ReLU-3 [-1, 32, 50, 50] 0
Conv2d-4 [-1, 32, 50, 50] 9,248
BatchNorm2d-5 [-1, 32, 50, 50] 64
ReLU-6 [-1, 32, 50, 50] 0
Conv2d-7 [-1, 32, 50, 50] 9,248
BatchNorm2d-8 [-1, 32, 50, 50] 64
ReLU-9 [-1, 32, 50, 50] 0
Conv2d-10 [-1, 64, 24, 24] 18,496
BatchNorm2d-11 [-1, 64, 24, 24] 128
ReLU-12 [-1, 64, 24, 24] 0
Conv2d-13 [-1, 64, 24, 24] 36,928
BatchNorm2d-14 [-1, 64, 24, 24] 128
ReLU-15 [-1, 64, 24, 24] 0
Conv2d-16 [-1, 256, 11, 11] 147,712
BatchNorm2d-17 [-1, 256, 11, 11] 512
ReLU-18 [-1, 256, 11, 11] 0
Conv2d-19 [-1, 512, 5, 5] 1,180,160
BatchNorm2d-20 [-1, 512, 5, 5] 1,024
ReLU-21 [-1, 512, 5, 5] 0
Conv2d-22 [-1, 1024, 2, 2] 4,719,616
BatchNorm2d-23 [-1, 1024, 2, 2] 2,048
ReLU-24 [-1, 1024, 2, 2] 0
Linear-25 [-1, 300] 1,229,100
================================================================
Total params: 7,356,140
Trainable params: 7,356,140
Non-trainable params: 0
----------------------------------------------------------------
Basically the input is a 105x105 image, single channel (grayscale). I have applied sigmoid on the output.
While I train the model, I use simple Mean-squared error. I use the Adam optimizer with the learning rate set to $10^{-5}$ , to help tuning.
The model gets struck at the same constant loss after 2 or 3 epochs.
On further investigating, I found out that the model generalizes and makes outputs a zero vector in each case. I am assuming it is stuck on the local minima, but how do I go about training my model successfully?
Also, the model architecture has been chosen randomly (not literally, but no particular logic behind the dimensionality retained at the end of each layer), so please advise me if you think there is any present irregularity that would better training after rectification, in terms of layers.
I would love to hear some tips :) .