Hallo can someone tell me in what format my input data has to be. Now I have it in csv format with the first column being the target variable but I always get a Algorithm Error which I think is due to wrong input data format.
trainpath = sess.upload_data(
path='revenue_train.csv', bucket=bucket,
key_prefix='production')
testpath = sess.upload_data(
path='revenue_test.csv', bucket=bucket,
key_prefix='production')
# launch training job, with asynchronous call
sklearn_estimator.fit({'train':trainpath, 'test': testpath}, wait=False)
when you use a custom Docker or framework estimator (like you do) you can use any file format (csv, pdf, mp4, whatever you have in S3). The Sklearn container and estimator are agnostic of the file format ; it is the role of your user-provided Python code in the estimator to know how to read those files.
Related
I have built an Microsoft Azure ML Studio workspace predictive web service, and have a scernario where I need to be able to run the service with different training datasets.
I know I can setup multiple web services via Azure ML, each with a different training set attached, but I am trying to find a way to do it all within the same workspace and passing a Web Input Parameter as the input value to choose which training set to use.
I have found this article, which describes almost my scenario. However, this article relies on the training dataset that is being pulled from the Load Trained Data module, as having a static endpoint (or blob storage location). I don't see any way to dynamically (or conditionally) change this location based on a Web Input Parameter.
Basically, does Azure ML support a "conditional training data" loading?
Or, might there be a way to combine training datasets, then filter based on the passed Web Input Parameter?
This probably isn't exactly what you need, but hopefully, it helps you out.
To combine data sets, you can use the Join Data module.
To filter, that may be accomplished by executing a Python script. Here's an example.
Using the Adult Census Income Binary Classification dataset, on the age column, there's a minimum age of 17.
If I wanted to filter the data set by age, connect it to an Execute Python Script module and here's the filtering code with the pandas query method.
# The script MUST contain a function named azureml_main
# which is the entry point for this module.
import pandas as pd
def azureml_main(dataframe1 = None, dataframe2 = None):
# Return value must be of a sequence of pandas.DataFrame
return dataframe1.query("age >= 25")
And looking at that output it filters out the data set where the minimum age is now 25.
Sure, you can do that. What you would want is to use an Execute R Script or SQL Transformation module to determine, based on your input data, what model to use. Something like this:
Notice, your input data is cleaned/updated/feature engineered, then it's passed to two different SQL transforms which will tell it to go to one of two paths.
Each path has it's own training data.
Note: I am not exactly sure what your use case is, but if it were me, I would instead train two different models using the two different training data, then try to just use the models in my web service, not actually train on the web service as that would likely be quite slow.
We are trying to use text classification example from the TensorFlow Examples (tensorflow/examples/learn/text_classification.py) . It works well with db_pedia data.
Now we are trying to save/restore the model using Saver but we are not getting where to use Saver APIs as code in text_classification.py doesn’t use Session at all and Saver API need session to save/restore.
This example uses tf.estimator.Estimator, which has a special method
export_savedmodel for saving.
In addition, you can specify model_dir in constructor:
Directory to save model parameters, graph and etc. This can also be
used to load checkpoints from the directory into a estimator to
continue training a previously saved model. If None, the model_dir in
config will be used if set. If both are set, they must be same. If
both are None, a temporary directory will be used.
I've followed the TensorFlow for Poets tutorial and replaced the stock flower_photos with a few classes of my own. Now I've got my labels.txt file and my graph.pb saved on my local machine.
Is there a way for me to deploy this pre-trained model to Google Cloud Platform? I've been reading the docs and all I can find are instructions on how to create, train, and deploy models from within their ML Engine. But I don't want to spend money training my model on Google's servers when I only need them to host my model so I can call it for predictions.
Anyone else run into the same problem?
Deploying a locally trained model is a supported use case; the instructions are essentially the same regardless of where you trained it:
To deploy a model version you'll need:
A TensorFlow SavedModel saved on Google Cloud Storage. You can get a
model by:
Following the Cloud ML Engine training steps to train in the
cloud.
Training elsewhere and exporting to a SavedModel.
Unfortunately, TensorFlow for Poets does not show how to export a SavedModel (I've filed a feature request to address that). In the meantime, you can write a "converter" script like the following (you could alternatively do this at the end of training instead of saving out graph.pb and reading it back in):
input_graph = 'graph.pb'
saved_model_dir = 'my_model'
with tf.Graph() as graph:
# Read in the export graph
with tf.gfile.FastGFile(input_graph, 'rb') as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
tf.import_graph_def(graph_def, name='')
# CloudML Engine and early versions of TensorFlow Serving do
# not currently support graphs without variables. Add a
# prosthetic variable.
dummy_var = tf.Variable(0)
# Define SavedModel Signature (inputs and outputs)
in_image = graph.get_tensor_by_name('DecodeJpeg/contents:0')
inputs = {'image_bytes':
tf.saved_model.utils.build_tensor_info(in_image)}
out_classes = graph.get_tensor_by_name('final_result:0')
outputs = {'prediction': tf.saved_model.utils.build_tensor_info(out_classes)}
signature = tf.saved_model.signature_def_utils.build_signature_def(
inputs=inputs,
outputs=outputs,
method_name='tensorflow/serving/predict'
)
# Save out the SavedModel.
b = saved_model_builder.SavedModelBuilder(saved_model_dir)
b.add_meta_graph_and_variables(sess,
[tf.saved_model.tag_constants.SERVING],
signature_def_map={'predict_images': signature})
b.save()
(Untested code based on this codelab and this SO post).
If you want the output to use string labels instead of integer indices, make the following change:
# Loads label file, strips off carriage return
label_lines = [line.rstrip() for line
in tf.gfile.GFile("retrained_labels.txt")]
out_classes = graph.get_tensor_by_name('final_result:0')
out_labels = tf.gather(label_lines, ot_classes)
outputs = {'prediction': tf.saved_model.utils.build_tensor_info(out_labels)}
Partial answer only, unfortunately, but I have been able to accomplish this...but with some ongoing issues that I have not yet resolved. I ported over the trained pb and txt files to my server, installed Tensorflow, and am calling the trained model via HTTP request. It works perfectly...on the first run. Then fails every other time.
tensorflow deployment on openshift, errors with gunicorn and mod_wsgi
Surprised there are not more people out there trying to go after this general issue.
I am using a custom image set to train a neural network using Tensorflow API. After successful training process I get these checkpoint files containing values of different training var. I now want to get an inference model from these checkpoint files, I found this script which does that, which I can then use to generate deepdream images as explained in this tutorial. The problem is when I load my model using:
import tensorflow as tf
model_fn = 'export'
graph = tf.Graph()
sess = tf.InteractiveSession(graph=graph)
with tf.gfile.FastGFile(model_fn, 'rb') as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
t_input = tf.placeholder(np.float32, name='input')
imagenet_mean = 117.0
t_preprocessed = tf.expand_dims(t_input-imagenet_mean, 0)
tf.import_graph_def(graph_def, {'input':t_preprocessed})
I get this error:
graph_def.ParseFromString(f.read())
self.MergeFromString(serialized)
raise message_mod.DecodeError('Unexpected end-group tag.')
google.protobuf.message.DecodeError: Unexpected end-group tag.
The script expect a protocol buffer file, I am not sure the script I am using to generate inference models is giving me proto buffer files or not.
Can someone please suggest what am I doing wrong, or is there a better way to achieve this. I simply want to convert checkpoint files generated by tensor to proto buffer.
Thanks
The link to the script you ran is broken, but in any case the recommended thing is not to try to generate an inference model from a checkpoint, but rather to embed code at the end of your training program that will emit a "SavedModel" export (which is not the same thing as a checkpoint).
Please see [1], and in particular the heading "Building a Saved Model". Note that a Saved Model constitutes multiple files, one of which is indeed a protocol buffer (which directly answers your question I hope); the others are variable files and (optional) asset files.
[1] https://www.tensorflow.org/programmers_guide/saved_model
I run caffe using an image_data_layer and don't want to create an LMDB or LevelDB for the data, But The compute_image_mean tool only works with LMDB/LevelDB databases.
Is there a simple solution for creating a mean file from a list of files (the same format that image_data_layer is using)?
You may notice that recent models (e.g., googlenet) do not use a mean file the same size as the input image, but rather a 3-vector representing a mean value per image channel. These values are quite "immune" to the specific dataset used (as long as it is large enough and contains "natural images").
So, as long as you are working with natural images you may use the same values as e.g., GoogLenet is using: B=104, G=117, R=123.
The simplest solution is to create a LMDB or LevelDB database of the image set.
The complicated solution is to write a tool similar to compute_image_mean, which takes image inputs and do the transformations and find the mean!