I am training a model in tensorflow and I am doing checkpoints for my model. I the Checkpoints directory, I have four files namely,
checkpoint
model.cpkt-0.data-00000-of-00001
model.cpkt-0.index
model.cpkt-0.meta
Now I want to extract the weights values for each layer in my graph, how can I do that?
I tried this:
import tensorflow as tf
sess = tf.InteractiveSession()
saver = tf.train.import_meta_graph('model.cpkt-0.meta')
w = saver.restore(sess, 'model.cpkt-0.data-00000-of-00001')
But I am getting the following error:
Unable to open table file ./model.cpkt-0.data-00000-of-00001: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
You are restoring in a wrong way
saver.restore(sess, 'model.cpkt-0')
# get the graph
g = tf.get_default_graph()
w1 = g.get_tensor_by_name('some_variable_name as per your definition in the model')
Related
I'm new in Keras. I want save model with best weights like as:
model1.compile(loss="mean_squared_error", optimizer="RMSprop")
model1.summary()
mcp_save = ModelCheckpoint('best_model.h5', save_best_only=True, monitor='val_accuracy', mode='auto', verbose=2)
callbacks_list = [mcp_save]
epochs = 5000
batch_size = 50
# fit the model
history = model1.fit(x_train, y_train,
batch_size=batch_size,
epochs=epochs,
callbacks=callbacks_list,
validation_data=(x_test, y_test),
verbose=2)
I couldn't come across warning or error message on Pycharm 2019 Community edition. But I am not able to see 'best_model.h5' on project file folder or somwhere else on my computer after trainig process finished?? Would you give me advices please?? What are my faults??
Your code looks fine to me. I use this callback often. All I can suggest is that you use a full path to designate where to save the model rather than a relative path.
I am new in Keras and I learned fitting and evaluating the model.
After evaluating the model one can see the actual predictions made by model.
I am wondering Is it also possible to see the predictions during fitting in Keras? Till now I cant find any code doing this.
Since this question doesn't specify "epochs", and since using callbacks may represent extra computation, I don't think it's exactly a duplication.
With tensorflow, you can use a custom training loop with eager execution turned on. A simple tutorial for creating a custom training loop: https://www.tensorflow.org/tutorials/eager/custom_training_walkthrough
Basically you will:
#transform your data in to a Dataset:
dataset = tf.data.Dataset.from_tensor_slices(
(x_train, y_train)).shuffle(some_buffer).batch(batchSize)
#the above is buggy in some versions regarding shuffling, you may need to shuffle
#again between each epoch
#create an optimizer
optimizer = tf.keras.optimizers.Adam()
#create an epoch loop:
for e in range(epochs):
#create a batch loop
for i, (x, y_true) in enumerate(dataset):
#create a tape to record actions
with tf.GradientTape() as tape:
#take the model's predictions
y_pred = model(x)
#calculate loss
loss = tf.keras.losses.binary_crossentropy(y_true, y_pred)
#calculate gradients
gradients = tape.gradient(loss, model.trainable_weights)
#apply gradients
optimizer.apply_gradients(zip(gradients, model.trainable_weights)
You can use the y_pred var for doing anything, including getting its numpy_pred = y_pred.numpy() value.
The tutorial gives some more details about metrics and validation loop.
I have trained a constitutional net using transfer learning from ResNet50 in keras as given below.
base_model = applications.ResNet50(weights='imagenet', include_top=False, input_shape=(333, 333, 3))
## set model architechture
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(1024, activation='relu')(x)
x = Dense(256, activation='relu')(x)
predictions = Dense(y_train.shape[1], activation='softmax')(x)
model = Model(input=base_model.input, output=predictions)
model.compile(loss='categorical_crossentropy', optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),
metrics=['accuracy'])
model.summary()
After training the model as given below I want to save the model.
history = model.fit_generator(
train_datagen.flow(x_train, y_train, batch_size=batch_size),
steps_per_epoch=600,
epochs=epochs,
callbacks=callbacks_list
)
I can't use save_model() function from models of keras as model is of type Model here. I used save() function to save the model. But later when i loaded the model and validated the model it behaved like a untrained model. I think the weights were not saved. What was wrong.? How to save this model properly.?
As per Keras official docs,
If you only need to save the architecture of a model you can use
model_json = model.to_json()
with open("model_arch.json", "w") as json_file:
json_file.write(model_json)
To save weights
model.save_weights("my_model_weights.h5")
You can later load the json file and use
from keras.models import model_from_json
model = model_from_json(json_string)
And similarly, for weights you can use
model.load_weights('my_model_weights.h5')
I am using the same approach and this works perfectly well.
I don't know what happens with my models, but I've never been able to use save_model() and load_model(), there is always an error associated. But these functions exist.
What I usually do is to save and load weights (it's enough for using the model, but may cause a little problem for further training, as the "optimizer" state was not saved, but it was never a big problem, soon a new optimizer finds its way)
model.save_weights(fileName)
model.load_weights(fileName)
Another option us using numpy for saving - this one never failed:
np.save(fileName,model.get_weights())
model.set_weights(np.load(fileName))
For this to work, just create your model again (keep the code you use to create it) and set its weights.
I have a large (1 TB) set of data split over about 3,000 CSV files. My plan is to convert it to one large LMDB file so it can be read quickly for training a neural network. However, I have not been able to find any documentation on how to load an LMDB file into TensorFlow. Does anyone know how to do this? I know TensorFlow can read CSV files, but I believe that would be too slow.
According to this there are several ways to read data in TensorFlow.
The simplest one is to feed your data through placeholders. When using placeholders - the responsibility for shuffling and batching is on you.
If you want to delegate shuffling and batching to the framework then you need to create an input pipeline. The problem is this - how do you inject lmdb data into the symbolic input pipeline. A possible solution is to use the tf.py_func operation. Here is an example:
def create_input_pipeline(lmdb_env, keys, num_epochs=10, batch_size=64):
key_producer = tf.train.string_input_producer(keys,
num_epochs=num_epochs,
shuffle=True)
single_key = key_producer.dequeue()
def get_bytes_from_lmdb(key):
with lmdb_env.begin() as txn:
lmdb_val = txn.get(key)
example = get_example_from_val(lmdb_val) # A single example (numpy array)
label = get_label_from_val(lmdb_val) # The label, could be a scalar
return example, label
single_example, single_label = tf.py_func(get_bytes_from_lmdb,
[single_key], [tf.float32, tf.float32])
# if you know the shapes of the tensors you can set them here:
# single_example.set_shape([224,224,3])
batch_examples, batch_labels = tf.train.batch([single_example, single_label],
batch_size)
return batch_examples, batch_labels
The tf.py_func op inserts a call to regular python code inside of the TensorFlow graph, we need to specify the inputs and the number and types of the outputs. The tf.train.string_input_producer creates a shuffled queue with the given keys. The tf.train.batch op create another queue that contains batches of data. When training, each evaluation of batch_examples or batch_labels will dequeue another batch from that queue.
Because we created queues we need to take care and run the QueueRunner objects before we start training. This is done like this (from the TensorFlow doc):
# Create the graph, etc.
init_op = tf.initialize_all_variables()
# Create a session for running operations in the Graph.
sess = tf.Session()
# Initialize the variables (like the epoch counter).
sess.run(init_op)
# Start input enqueue threads.
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(sess=sess, coord=coord)
try:
while not coord.should_stop():
# Run training steps or whatever
sess.run(train_op)
except tf.errors.OutOfRangeError:
print('Done training -- epoch limit reached')
finally:
# When done, ask the threads to stop.
coord.request_stop()
# Wait for threads to finish.
coord.join(threads)
sess.close()
I am struggling to use Random Forest in Python with Scikit learn. My problem is that I use it for text classification (in 3 classes - positive/negative/neutral) and the features that I extract are mainly words/unigrams, so I need to convert these to numerical features. I found a way to do it with DictVectorizer's fit_transform:
from sklearn.preprocessing import LabelEncoder
from sklearn.metrics import classification_report
from sklearn.feature_extraction import DictVectorizer
vec = DictVectorizer(sparse=False)
rf = RandomForestClassifier(n_estimators = 100)
trainFeatures1 = vec.fit_transform(trainFeatures)
# Fit the training data to the training output and create the decision trees
rf = rf.fit(trainFeatures1.toarray(), LabelEncoder().fit_transform(trainLabels))
testFeatures1 = vec.fit_transform(testFeatures)
# Take the same decision trees and run on the test data
Output = rf.score(testFeatures1.toarray(), LabelEncoder().fit_transform(testLabels))
print "accuracy: " + str(Output)
My problem is that the fit_transform method is working on the train dataset, which contains around 8000 instances, but when I try to convert my test set to numerical features too, which is around 80000 instances, I get a memory error saying that:
testFeatures1 = vec.fit_transform(testFeatures)
File "C:\Python27\lib\site-packages\sklearn\feature_extraction\dict_vectorizer.py", line 143, in fit_transform
return self.transform(X)
File "C:\Python27\lib\site-packages\sklearn\feature_extraction\dict_vectorizer.py", line 251, in transform
Xa = np.zeros((len(X), len(vocab)), dtype=dtype)
MemoryError
What could possibly cause this and is there any workaround? Many thanks!
You are not supposed to do fit_transform on your test data, but only transform. Otherwise, you will get different vectorization than the one used during training.
For the memory issue, I recommend TfIdfVectorizer, which has numerous options of reducing the dimensionality (by removing rare unigrams etc.).
UPDATE
If the only problem is fitting test data, simply split it to small chunks. Instead of something like
x=vect.transform(test)
eval(x)
you can do
K=10
for i in range(K):
size=len(test)/K
x=vect.transform(test[ i*size : (i+1)*size ])
eval(x)
and record results/stats and analyze them afterwards.
in particular
predictions = []
K=10
for i in range(K):
size=len(test)/K
x=vect.transform(test[ i*size : (i+1)*size ])
predictions += rf.predict(x) # assuming it retuns a list of labels, otherwise - convert it to list
print accuracy_score( predictions, true_labels )