I'm trying to train a YOLO model with the PASCAL VOC dataset using Tensorflow on Google Cloud ML Engine but my training job keeps hanging at "Restoring parameters from /root.../yolo_tiny.ckpt".
Is there a reason for this? I've been waiting for the past 4-5 hours.
Related
When doing reinforcement learning, I have to start training from the beginning each time. It costs lots of time. Is there any solution on starting training from the previous training results? Thanks.
If you are doing reinforcement learning based on episodes, you can save the networks you have trained to a file every X episodes. When you are running the script, you can check if that file exists, and load it instead of starting with an empty network.
How you can do this depends on which programming language you're using.
If you are using Python, you can save your data, state table and neural network weights using the Pickle or JSON modules.
I have not GPU support so it often happens that my model takes hours to train. Can I train my model in batches , for example if I want to have 100 epochs for my model,but due to power cut my training stops(at 50th epoch) but when I retrain my model I want to train it from where it was left (from 50th epoch).
It would be much appreciated if anyone can explain it by some example. https://timbu.com user
The weights are already updated, retraining the model with the updated weights without reintializing them will carry where it left.
You can work with online notebooks like google's colab or microsoft azure notebook if you have ressources problems. They offer a good working environnement, colab for exemple has gpu and tpu enabled and 16go ram limit.
I have worked through the README file of the Movielens CloudML sample: https://github.com/GoogleCloudPlatform/cloudml-samples/tree/master/movielens
After the pre-processing step I'm left with a bunch of TFRecord files for training evaluation and prediction. I can take the prediction records and run the prediction successfully.
To use this recommender in production, how do I get the prediction record for a specific user? My hope was something along the lines of deploying a streaming Dataflow pipeline that produces the prediction records. How would I go about that?
When I use tensorboard to show the data, I found that there is more than one curve. I think this is related to the configuration. So could someone tell me what each curve represents?
This is not related in any way with the Cloud ML Engine. You can find
all the configurable parameters for the Engine in the docs for its REST API (training input, training output, prediction input, prediction output, model resource, version resource).
These curves from your tensorboard is something you configured in your tensorflow code, probably the training cost for several different runs, set as a summary scalar with the name "train_cost".
I'm using a bunch of images to train my Tensorflow Image recognition project using this tutorial https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/index.html#4
Actually I need a lot of Cpu to train my model and it takes a lot of time on my laptop.
I have registered a Google ML account and started this tutorial:
https://cloud.google.com/ml/docs/quickstarts/training
Everything is set up and running but this is for mnist sample code. There is no image_retraining sample code like the retrain.py from tensorflow.
Looking for some examples on how to to run the Tensorflow Image Recognition script retrain in Google ML.