Tensorflow with GPU on Google Cloud - machine-learning

Question

I have a model on Google Machine Learning using tensorflow, and it's ok.
Now I want to do some predicts using the GPU.
I saw this link, but it tells about trainning with GPU not prediction. There's nothing about GPU in prediction session.
Someone Know if its possible to do prediction using google machine learning engine with GPU? Or if I use the trainning with GPU, my Prediction automatically run with GPU?
I'm using the follow commandline:
gcloud ml-engine predict --model ${MODEL_NAME} --json-instances request.json
This command works, but It's using CPU.
Additional information: My model is published in us-east1 zone, and my scale is automatically.

Answer 1

You cannot choose to use the GPU for prediction in ml-engine. It's unclear whether they are using GPUs by default -- I would link to documentation but there is none available.
I am sure, however, that they are not using TPUs. Currently, Google is only using TPUs for internal services; although they have created a TPU cloud exclusively for researchers to experiment with: https://yourstory.com/2017/05/google-cloud-tpus-machine-learning-models-training/
If you want more control over how your prediction is run, for the same price you can configure a Google Compute Engine with a high-powered Tesla K80 GPU. Your Tensorflow model will work there, too, and it is straightforward to set up.
My suggestion would be to make benchmark predictions using your GCE instance and then compare them to the ml-engine. If ml-engine is faster than GCE, then Google is probably using GPUs for prediction. Surely, their goal is to provide GPUs and TPUs as an ml-engine in the future, but demand is overloading the HPC cloud these days.

Answer 2

Online predictions on GCP ML Engine use Single core CPUs by default which have high latency. If it suits your requirement, you can use the Quad core CPU which serve predictions faster. For using that, you must specify the type of CPU for predictions which creating a version of your model on ML Engine. Link to the documentation : https://cloud.google.com/ml-engine/docs/tensorflow/online-predict.

Answer 3

We support GPU now. Documentation here!
Example:
gcloud beta ai-platform versions create version_name \
--model model_name \
--origin gs://model-directory-uri \
--runtime-version 2.1 \
--python-version 3.7 \
--framework tensorflow \
--machine-type n1-standard-4 \
--accelerator count=1,type=nvidia-tesla-t4 \
--config config.yaml
If you use one of the Compute Engine (N1) machine types for your model version, you can optionally add GPUs to accelerate each prediction node.
NVIDIA Tesla K80
NVIDIA Tesla P4
NVIDIA Tesla P100
NVIDIA Tesla T4
NVIDIA Tesla V100

Answer 4

This site has some information:
https://cloud.google.com/ml-engine/docs/how-tos/getting-started-training-prediction
However, it is a completely different way to train and predict. They provide the means for training and predictiong on their service infrastructure. You just build the model with your Tensorflow program and then use their hardware with their cloud SDK. So it shouldn't bother you whether it runs on CPU or GPU.