RNN Language Model in PyTorch predicting the same three words repeatedly - machine-learning

Question

I am attempting to create a word-level language model using an RNN in PyTorch. Whenever I am training the loss stays about the same for the whole training set and when I try to sample a new sentence the same three words are predicted in the same order. For example in my most recent attempt the RNN predicted 'the' then 'same' then 'of' and that sequence just kept repeating. I have tried changing the how I've set up the RNN including using LSTM's, GRU's, and different embeddings but so far nothing has worked.
The way that I am training the RNN is by taking a sentence of 50 words, and selecting a progressively larger part of the sentence with the next word being the target. At the end of the sentence, I have an EOS tag. I am using text from Republic by Plato as my training set and embedding it using a pytorch embedding layer. I am then feeding it into an LSTM and then a linear layer to get the right shape. I am not sure if the problem is in the RNN, the data, the training or what so any help would be greatly appreciated.
If anyone has any experience in nlp or in language modeling I would greatly appreciate any help you could offer for this fixing this problem. My end goal is simply to just be able to generate a sentence. Thanks in advance!
Here is my RNN
class LanguageModel(nn.Module):
"""
Class that defines the reccurent neural network.
Methods
-------
forward(input, h, c)
Forward propogation through the RNN.
initHidden()
Initializes the hidden and cell states.
"""
def __init__(self, vocabSize, seqLen = 51, embeddingDim = 30, hiddenSize = 32, numLayers = 1, bid = False):
"""
Initializes the class
Parameters
----------
seqLen : int, optional
The length of the input sequence.
embeddingDim : int, optional
The dimension that the embedding dimension for the encoder should be.
vocabSize : int
The length of the vocab dictionary.
hiddenSize : int, optional
The size that the hidden state should be.
numLayers : int, optional
The number of LSTM Layers.
bid : bool, optional
Whether the RNN should be bidirctional or not.
"""
super(LanguageModel, self).__init__()
self.hiddenSize = hiddenSize
self.numLayers = numLayers
# Set value of numDirections based on whether or not the RNN is bidirectional.
if bid == True:
self.numDirections = 2
else:
self.numDirections = 1
self.encoder = nn.Embedding(vocabSize, embeddingDim)
self.LSTM = nn.LSTM(input_size = embeddingDim, hidden_size = hiddenSize, num_layers = numLayers, bidirectional = bid)
self.decoder = nn.Linear(seqLen * self.numDirections * hiddenSize, vocabSize)
def forward(self, input, h, c):
"""
Forward propogates through the RNN
Parameters
----------
input : torch.Tensor
Input to RNN. Should be formatter using makeInput() and padSeq().
h : torch.Tensor
Hidden state.
c : torch.Tensor
Cell state.
Returns
-------
torch.Tensor
Log probabilities for the predicted word from the RNN.
"""
emb = self.encoder(input)
emb.unsqueeze_(1) # Add in the batch dimension so the shape is right for the LSTM
out, (h, c) = self.LSTM(emb, (h, c))
out = out.view(1, -1) # Reshaping to fit into the loss function.
out = self.decoder(out)
logProbs = F.log_softmax(out, dim = 1)
return logProbs
def initHidden(self):
"""
Initializes the hidden and cell states.
Returns
-------
torch.Tensor
Tensor containing the initial hidden state.
torch.Tensor
Tensor containing the intial cell state.
"""
h = torch.zeros(self.numLayers * self.numDirections, 1, self.hiddenSize)
c = torch.zeros(self.numLayers * self.numDirections, 1, self.hiddenSize)
return h, c
here is how I create my input and targets
def makeInput(sentence):
"""
Prepares a sentence for input to the RNN.
Parameters
----------
sentence : list
The sentence to be converted into input. Should be of form: [str]
Returns
-------
torch.Tensor
Tensor of the indices for each word in the input sentence.
"""
sen = sentence[0].split() # Split the list into individual words
sen.insert(0, 'START')
input = [word2Idx[word] for word in sen] # Iterate over the words in sentence and convert to indices
return torch.tensor(input)
def makeTarget(sentence):
"""
Prepares a sentence to be a target.
Parameters
----------
sentence : str
The sentence to be made into a target. Should be of form: [str]
Returns
-------
torch.Tensor
Tensor of the indices for the target phrase including the <EOS> tag.
"""
sen = sentence[0].split() # Split the list into individual words
sen.append('EOS')
target = [word2Idx[word] for word in sen]
target = torch.tensor(target, dtype = torch.long)
return target.unsqueeze_(-1) # Removing dimension for loss function
def padSeq(seq, refSeq):
"""
Pads a sequence to be the same shape as another sequence.
Parameters
----------
seq : torch.Tensor
The sequence to pad.
refSeq : torch.Tensor
The reference sequence. seq will be padded to be the same shape as refSeq.
Returns
-------
torch.Tensor
Tensor containing the padded sequence.
"""
padded = pad_sequence([refSeq, seq])
tmp = torch.t(padded) # Transpose the padded sequence for easier indexing on return
return tmp[1] # Return only the padded seq not both sequences
and here is my training loop
def train():
"""
Trains the model.
"""
start = time.time()
for i, data in enumerate(trainLoader):
inputTensor = makeInput(data)
targetTensor = makeTarget(data)
targetTensor = targetTensor.to(device)
h, c = model.initHidden()
h = h.to(device)
c = c.to(device)
optimizer.zero_grad()
loss = 0
for x in range(inputTensor.size(0)): # Iterate over all of the words in the input sentence
""" Preparing input for the rnn """
input = inputTensor[: x + 1] # We only want part of the input so the RNN can learn on predicting the next words
input = padSeq(input, inputTensor)
input = input.to(device)
out = model(input, h, c)
l = criterion(out, targetTensor[x])
loss += l
loss.backward()
optimizer.step()
if i % 250 == 0: # Print updates to the models loss every 10 iters.
print('[{}] Epoch: {} -> {}'.format(timeSince(start), i, loss / inputTensor.size(0)))