I have URL for weight file (.h5) of a CNN model. I want to load that weight directly to Python file and compile a keras model. How can this be done.?
Is there a direct approach or should I download the weight file and load it from disk.?
There is a function to download weights from URL and stock it in ~/.keras/
It's the function get_file
from keras.utils.data_utils import get_file
Example :
from keras.utils.data_utils import get_file
weights_path = get_file(
'the name under the model will be saved',
'YOUR URL')
model.load_weights(weights_path)
It returns the path where the weights have been saved.
Then you can use load_weight.
But in all the case, the weights will be saved on your computer.
Related
Hallo can someone tell me in what format my input data has to be. Now I have it in csv format with the first column being the target variable but I always get a Algorithm Error which I think is due to wrong input data format.
trainpath = sess.upload_data(
path='revenue_train.csv', bucket=bucket,
key_prefix='production')
testpath = sess.upload_data(
path='revenue_test.csv', bucket=bucket,
key_prefix='production')
# launch training job, with asynchronous call
sklearn_estimator.fit({'train':trainpath, 'test': testpath}, wait=False)
when you use a custom Docker or framework estimator (like you do) you can use any file format (csv, pdf, mp4, whatever you have in S3). The Sklearn container and estimator are agnostic of the file format ; it is the role of your user-provided Python code in the estimator to know how to read those files.
Is it possible to upload a pre-trained machine learning model (from my local computer, for which I generated a model.pkl) on databricks, and serve it?
Or is it impossible on Databricks ?
I managed it this way.
with mlflow.start_run(run_name="Your_MlFlow_RunName") as mlflow_run:
model=load_model(<filepath_mod>)
model.load_weights(<filepath_weights>)
model.summary()
mlflow.keras.log_model(model,"YourModelName",keras_module=tf.keras,registered_model_name="YourModelName")
But I did upload a zip file, not pickle to also have access to best weights .
import zipfile
with zipfile.ZipFile("/dbfs/FileStore/shared_uploads/<xxx#yyy>/TF_model/MyModel.zip", 'r') as zip_ref:
zip_ref.extractall("/dbfs/FileStore/shared_uploads/<xxx#yyy>/TF_model/")
Also note first time I did databraicks changed some filenames while unzipping, eg. MyModelWeights.data_0000 to MyModelWeights_data_0000 so had to write a fix for that too. Other times it did not happen though, so no idea why
We are trying to use text classification example from the TensorFlow Examples (tensorflow/examples/learn/text_classification.py) . It works well with db_pedia data.
Now we are trying to save/restore the model using Saver but we are not getting where to use Saver APIs as code in text_classification.py doesn’t use Session at all and Saver API need session to save/restore.
This example uses tf.estimator.Estimator, which has a special method
export_savedmodel for saving.
In addition, you can specify model_dir in constructor:
Directory to save model parameters, graph and etc. This can also be
used to load checkpoints from the directory into a estimator to
continue training a previously saved model. If None, the model_dir in
config will be used if set. If both are set, they must be same. If
both are None, a temporary directory will be used.
I am using a custom image set to train a neural network using Tensorflow API. After successful training process I get these checkpoint files containing values of different training var. I now want to get an inference model from these checkpoint files, I found this script which does that, which I can then use to generate deepdream images as explained in this tutorial. The problem is when I load my model using:
import tensorflow as tf
model_fn = 'export'
graph = tf.Graph()
sess = tf.InteractiveSession(graph=graph)
with tf.gfile.FastGFile(model_fn, 'rb') as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
t_input = tf.placeholder(np.float32, name='input')
imagenet_mean = 117.0
t_preprocessed = tf.expand_dims(t_input-imagenet_mean, 0)
tf.import_graph_def(graph_def, {'input':t_preprocessed})
I get this error:
graph_def.ParseFromString(f.read())
self.MergeFromString(serialized)
raise message_mod.DecodeError('Unexpected end-group tag.')
google.protobuf.message.DecodeError: Unexpected end-group tag.
The script expect a protocol buffer file, I am not sure the script I am using to generate inference models is giving me proto buffer files or not.
Can someone please suggest what am I doing wrong, or is there a better way to achieve this. I simply want to convert checkpoint files generated by tensor to proto buffer.
Thanks
The link to the script you ran is broken, but in any case the recommended thing is not to try to generate an inference model from a checkpoint, but rather to embed code at the end of your training program that will emit a "SavedModel" export (which is not the same thing as a checkpoint).
Please see [1], and in particular the heading "Building a Saved Model". Note that a Saved Model constitutes multiple files, one of which is indeed a protocol buffer (which directly answers your question I hope); the others are variable files and (optional) asset files.
[1] https://www.tensorflow.org/programmers_guide/saved_model
I am training a huge data file for libsvm and the resulting training file is too large. Is there any way to save the libsvm libraries model file in binary format?
If you are using Matlab:
Download svm_savemodel.c and svm_model_matlab.c (this is already included in libsvm, you can try to use the original one, but if it doesn't work, try this link) to your libsvm dir. Compile the Mex file (mex svm_savemodel.c), then it should work:
%save model model
fid = fopen('model.bin','w');
model = fwrite(fid, model, 'int16');
%load('model.mat');
fid = fopen('model.bin','rb');
model = fread(fid, model, 'int16');
svm_savemodel(model,'model.model');
If you are using C++:
There is a function that saves a model to a file:
int svm_save_model(const char *model_file_name, const struct svm_model *model);
More details are included in the github.