I have just started coding with TensorFlow and I have classified Images.
Is there any possibility of making a prediction based on testing data?
How can I predict missing value based on the model?
Question
Does Tensorflow have the capability to read training data and test data from two separate files?
Answer
Yes! Here is an example of processing Iris data, an intro machine learning data set for Tensorflow.
If you look at the code you will see the following lines
IRIS_TRAINING = "iris_training.csv"
IRIS_TEST = "iris_test.csv"
The data is clearly separated into training and test files. You don't have to separate your data into different files in Tensorflow, but it certainly supports it and this tutorial link shows how to do it.
Related
I have a large set of Training data which consists of various texts. They should be the input for my neural network. I have no output, or I don't know what to put as output.
Anyway, after the learning phase I want the neural network to create new texts based on the training data.
I read about this like „I made a bot watch 1000 hours of xy and asked it to write a new xy“.
Now my question is, what kind of machine learning is this? I am not looking for instructions on how to write it, but just a hint on how to find some keywords or tutorials. My Google searches so far were useless.
Your problem can usually be solved by an Encoder-Decoder architecture. This architecture would learn a set of latent vectors from your input, then try to output in whatever form you want. This architecture can be built with RNN, LSTM or CNN. Nowadays, attention-based models like transformers are more common among the big names. If you want to do text generation, you can start by reading about Generative Adversarial Networks (GANs).
I want to classify image documents(like Passport, Driving Licence etc) using Machine Learning.
Does anybody has any link or documents where I can get idea to do this task.
What I am thinking is of first converting the document to text format and then fro Text file extract the information.But this I can do with one file at a time.
I want to know how can I perform this in millions of document.
You don't need to convert documents to text, you can do this with images directly.
To do image classification you can build basic CNNs with Keras library.
https://towardsdatascience.com/building-a-convolutional-neural-network-cnn-in-keras-329fbbadc5f5
This basic CNN will be enough for you to train an image classifier. But you want to get state of the art accuracy, I recommend get a pretrained resnet50 and train it to build an image classifier. Other than accuracy, there is another major advantage of using pre trained network, you'll need less data to train a robust image classifier.
https://engmrk.com/kerasapplication-pre-trained-model/?utm_campaign=News&utm_medium=Community&utm_source=DataCamp.com
The only thing that you'll need to change is number of output classes from 1000 to the number of classes you want.
I currently pre-process some data on my laptop before passing it to ML Engine for training. Is it possible to apply a custom pre-processing function to my data and then train, all within ML Engine?
So instead of these steps:
Pre-process data on laptop.
Send pre-processed data to ML engine for training.
I would do:
Define pre-processing function for ML Engine
Send raw data to ML Engine, where it will:
a) pre-process my data by applying the function I've specified and
b) train on that data
Is this possible and, if so, how would I do it? I don't see anything in the docs.
Thanks!
You can use some of the samples code here:
Pre-processing is done using DataFlow and then training in ML Engine using the output generated during the pre-process phase.
For preprocessing TensorFlow models, consider TensorFlow Transform (Getting Started Guide).
You may be interested in the chicago_taxi example, which includes a script for integrating the preprocessing with classification on Cloud ML Engine.
Some users might see this as opinion-based-question but if you look closely, I am trying to explore use of Caffe as a purely testing platform as opposed to currently popular use as training platform.
Background:
I have installed all dependencies using Jetpack 2.0 on Nvidia TK1.
I have installed caffe and its dependencies successfully.
The MNIST example is working fine.
Task:
I have been given a convnet with all standard layers. (Not an opensource model)
The network weights and bias values etc are available after training. The training has not been done via caffe. (Pretrained Network)
The weights and bias are all in the form of MATLAB matrices. (Actually in a .txt file but I can easily write code to get them to be matrices)
I CANNOT do training of this network with caffe and must used the given weights and bias values ONLY for classification.
I have my own dataset in the form of 32x32 pixel images.
Issue:
In all tutorials, details are given on how to deploy and train a network, and then use the generated .proto and .caffemodel files to validate and classify. Is it possible to implement this network on caffe and directly use my weights/bias and training set to classify images? What are the available options here? I am a caffe-virgin so be kind. Thank you for the help!
The only issue here is:
How to initialize caffe net from text file weights?
I assume you have a 'deploy.prototxt' describing the net's architecture (layer types, connectivity, filter sizes etc.). The only issue remaining is how to set the internal weights of caffe.Net to pre-defined values saved as text files.
You can get access to caffe.Net internals, see net surgery tutorial on how this can be done in python.
Once you are able to set the weights according to your text file, you can net.save(...) the new weights into a binary caffemodel file to be used from now on. You do not have to train the net if you already have trained weights, and you can use it for generating predictions ("test").
If I have a video dataset of a specific action , how could I use them to train a classifier that could be used later to classify this action.
The question is very generic. In general, there is no foul proof way of training a classifier that will work for everything. It highly depends on the data you are working with.
Here is the 'generic' pipeline:
extract features from the video
label your features (positive for the action you are looking for; negative otherwise)
split your data into 2 (or 3) sets. One for training, one for testing and the other optionally for validation
train a classifier on the labeled examples (e.g. SVM, Neural Network, Nearest Neighbor ...)
validate the results on the validation data, if that is appropriate for the algorithm
test on data you haven't used for training.
You can start with some machine learning tools here http://www.cs.waikato.ac.nz/ml/weka/
Make sure you never touch the test data for any other purposes than testing
Good luck
Almost 10 years later, here's an updated answer.
Set up a camera and collect raw video data
Save it somewhere in form of single frames. Do this yourself locally or using a cloud bucket or use a service like Sieve API. Helpful repo linked here.
Export from Sieve or cloud bucket to get data labeled. Do this yourself or using some service like Scale Rapid.
Split your dataset into train, test, and validation.
Train a classifier on the labeled samples. Use transfer learning over some existing model and fine-tune just the last few layers.
Run your model over the test set after each training epoch and save the one with the best test set performance.
Evaluate your model at the end using the validation set.
There are many repos that can help you get started: https://github.com/weiaicunzai/awesome-image-classification
The two things that can help you ensure best results include 1. high quality labeled data and 2. a diverse, curated dataset. That's what Sieve can help with!