Neural Network not recognizing basic input pattern - machine-learning

Question

I've built a recurrent neural network to predict time series data. I seemed to be getting reasonable results, it would start at 33% accuracy for three classes as expected and would get better up to a point, also as expected. I wanted to test the network just to make sure it's actually working so I created a basic input / output as follows:
in1 in2 in3 out
1 2 3 0
5 6 7 1
9 10 11 2
1 2 3 0
5 6 7 1
9 10 11 2
I copied this pattern through a million lines in a csv. I would have assumed the neural network could easily identify this pattern, as it's always the same. I've tried a leaning rate of .1, .01, .001, .0001, but it always stays at about 33% (34% for the .0001 lr). I'll post my code below, should the neural net easily be able to identify this, is there something very wrong with my set up?
import os
import pandas as pd
from sklearn import preprocessing
from collections import deque
import random
import numpy as np
import time
import random
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, LSTM, CuDNNLSTM, BatchNormalization
from tensorflow.keras.models import load_model
df = pd.DataFrame()
df = pd.read_csv('/home/drew/Desktop/numbers.csv')
times = sorted(df.index.values)
last_5pct = times[-int(0.1*len(times))]
validation_df = df[(df.index >= last_5pct)]
main_df = df[(df.index < last_5pct)]
epochs = 10
batch_size = 64
train_x = main_df[['in1','in2','in3']]
validation_x = validation_df[['in1','in2','in3']]
train_y = main_df[['out']]
validation_y = validation_df[['out']]
train_x = train_x.values
validation_x = validation_x.values
train_x = train_x.reshape(train_x.shape[0],1,3)
validation_x = validation_x.reshape(validation_x.shape[0],1,3)
model = Sequential()
model.add(LSTM(128, input_shape=(train_x.shape[1:]), return_sequences=True))
model.add(Dropout(0.2))
model.add(BatchNormalization())
model.add(LSTM(128, input_shape=(train_x.shape[1:]), return_sequences=True))
model.add(Dropout(0.2))
model.add(BatchNormalization())
model.add(Dense(32, activation="relu"))
model.add(Dropout(0.2))
model.add(Dense(3, activation="softmax"))
opt = tf.keras.optimizers.Adam(lr=0.0001, decay=1e-6)
model.compile(loss='sparse_categorical_crossentropy',
optimizer=opt,
metrics=['accuracy'])
model.fit(train_x, train_y,
batch_size=batch_size,
epochs=epochs,
validation_data=(validation_x, validation_y))
result:
Train on 943718 samples, validate on 104857 samples
Epoch 1/10
943718/943718 [===] - 125s 132us/sample - loss: 0.0040 - acc: 0.3436 - val_loss: 2.3842e-07 - val_acc: 0.3439
Epoch 2/10
943718/943718 [===] - 111s 118us/sample - loss: 2.1557e-06 - acc: 0.3437 - val_loss: 2.3842e-07 - val_acc: 0.3435
...
Train on 943718 samples, validate on 104857 samples
Epoch 1/10
943718/943718 [==============================] - 125s 132us/sample - loss: 0.0040 - acc: 0.3436 - val_loss: 2.3842e-07 - val_acc: 0.3439
Epoch 2/10
943718/943718 [==============================] - 111s 118us/sample - loss: 2.1557e-06 - acc: 0.3437 - val_loss: 2.3842e-07 - val_acc: 0.3435
Epoch 6/10
719104/943718 [=====================>........] - ETA: 25s - loss: 2.4936e-07 - acc: 0.3436

Answer 1

It seems that you are not predicting time series result, so there is no need to return sequence after the second LSTM layer.
If you change the second LSTM to:
model.add(LSTM(128, input_shape=(train_x.shape[1:])))
Then, you should get high accuracy.