I have a data set which form is matlab file.
The data set contains 600,000 samples and every sample is a matrix of 7-by-256.
My data is not image but signal.
I want to use CNN of caffe to train the data.
So how can I convert it to LMDB as my input of CNN.
I'm badly need the solution!
Converting data in matlab directly to lmdb might be a little tricky.
Why don't you try exporting your data to hdf5 binary files (supported both by matlab and caffe)?
Here is an answer describing how this can be done.
Related
I am trying to process data to train a model.
I have a dataset processed and saved in a HDF5 file (original HDF file) to separate into two unoverlapping HDF files at the ratio 90:10.
I would like to separate data stored in that HDF file into two other HDF i.e. one HDF for training purpose which contains 90% of dataset in original HDF file and another HDF for validation purpose which contains 10% of dataset in original HDF file.
If you have any ideas to do it, please guide me.
Thank you so much in advance.
You don't have to separate the data into separate files for training and testing. (In fact, to properly train your model, you would have to do this multiple times -- randomly dividing the data into different training and testing sets each time.)
One option is to randomize the input when you read the data. You can do this by creating 2 lists of indices (or datasets). One list is the training data, and the other is the test data. Then, iterate over the lists to load the desired data.
Alternately (and probably simpler), you can use the h5imagegenerator from PyPi. Link to the package description here: pypi.org/project/h5imagegenerator/#description
If you search SO, you will find more answers on this topic:
Keras: load images batch wise for large dataset
How to split dataset into K-fold without loading the whole dataset
at once?
Reading large dataset from HDF5 file into x_train and use it in
keras model
Hope that helps. If you still want to know how to copy data from 1 file to another, take a look at this answer. It shows multiple ways to do that: How can I combine multiple .h5 file? You probably want to use Method 2a. It copies data as-is.
At the moment I'm trying to build an Autoencoder for detecting anomalies in time series data.
My approach is based on this tutorial: https://keras.io/examples/timeseries/timeseries_anomaly_detection/
But as often, my data is more complex then this simple tutorial.
I have two different time series, from two sensors and some metadata, like from which machine the time series was recorded.
with a normal MLP network you could have one network for the time series and one for the metadata and merge them in higher layers. But how can you use this data as an input to an Autoencoder?
Do you have any ideas, links to tutorials or papers I didn't found?
in this tutorial you can see a LSTM-VAE where the input time series is somehow concatenated with categorical data: https://github.com/cerlymarco/MEDIUM_NoteBook/tree/master/VAE_TimeSeries
There is an article explayining the code (but not on detail). There you can find the following explanation of the model:
"The encoder consists of an LSTM cell. It receives as input 3D sequences resulting from the concatenation of the raw traffic data and the embeddings of categorical features. As in every encoder in a VAE architecture, it produces a 2D output that is used to approximate the mean and the variance of the latent distribution. The decoder samples from the 2D latent distribution upsampling to form 3D sequences. The generated sequences are then concatenated back with the original categorical embeddings which are passed through an LSTM cell to reconstruct the original traffic sequences."
But sadly I don't understand exactly how they concatenate the input datas. If you understand it it would be nice if you could explain it =)
I think I understood it. you have to take a look at the input of the .fit() funktion. It is not one array, but there are seperate arrays for seperate categorical datas. additionaly there is the original input (in this case a time series). Because he has so many arrays in the input, he needs to have a corresponding number of input layers. So there is one Input layer for the Timeseries, another for the same time series (It's an autoencoder so x_train works like y_train) and a list of input layers, directly stacked with the embedding layers for the categorical data. after he has all the data in the corresponding Input layers he can concatenate them as you said.
by the way, he's using the same list for the decoder to give him additional information. I tried it out and it turns out that it was helpfull to add a dropout layer (high dropout e.g. 0.6) between the additional inputs and the decoder. If you do so, the decoder has to learn from the latent z and not only from the additional data!
hope I could help you =)
I need to build a classifier of skin lesions and have image dataset along with metadata which has description consisting of the classification.
any help as to how to match the data and images and use both in training my convolutional neural network
Seems like you are asking how to incorporate non-numeric data in your metadata in the Neural Network you are trying to train. Hopefully, you will find this document on tf.feature_columns helpful. It will help you incorporate categorical, ordinal, etc. as numeric columns.
To train the model with both kind of information, one way would be:
Apply CNN to your images and get a dense representation
Simultaneously, build another NN to pass your metadata through
Concatenate the two resultant Tensors from both NNs and then pass it all through a softmax
Hopefully this should help.
Now the above mentioned tf.feature_columns is no longer recommended for new code. Instead of this use Keras preprocessing layers.
I am learning to create a learning model using TensorFlow.
I have successfully run the MNIST tutorial, now would like to test the model with my own images. They are same-size image (224x224) and classified into folders.
Now I would like to use those images as input for my model as in the MNIST example. I tried to open the MNIST data-set but it's unreadable. I guess it has been converted into some binary types. Through the example, I think the MNIST dataset somehow has a structure like this:
mnist
test
images
labels
train
images
labels
How can I make a dataset look like the MNIST data from my own images files?
Thank you very much!
MNIST is not stored in image format. From the mnist web-site (http://yann.lecun.com/exdb/mnist/) you could see that it has specific format which is already close to the tensor or numpy array, which could be used in tensorflow with minimal adjustments. It is a kind of a matrix with numbers.
What you need to work with usual images (.jpg for instance) is to use any python lib for image processing to convert into the np.array. For example PIL will work, like here:
PIL and numpy
Another option is to use a built-in functions from tensorflow to convert your images straight to tensors supported by tensofrlow, check this out:
https://www.tensorflow.org/versions/r0.9/api_docs/python/image.html
I'm trying to make some modifications on Caffe in order to accept my 16 bit data.
I succeded into creating a lmdb dataset filled with 16 bit unsigned, unencoded 256x256 image instead of usual caffe 8 bit unsigned data, saved as "string" as would be the usual 8 bit lmdb that can be created with image_convert or DIGITS utilities.
I've changed the io.py functions array_to_datum and datum_to_array in order to create this lmdb at 16bit "string" data.
Now, if I use this lmdb on caffe (just 4 classes), the networks runs but doesn't converge. I strongly suspect that is not reading my data properly.
Problem is, the io.py functions array_to_blobproto and blobproto_to_array doesn't seems to make any distinction between internal data contents, and I cannot find the code where I should modify for dealing with 16bits.
Could anyone give me an hint to where to work on?
Edit:
Messing around in the code, I think that one of the possibility should be to create a new data layer or a new image data layer if I would like to work directly on the png without going throught lmdb. But trying to modify that C++ code is not a trivial task for me, esp. I cannot easily follow the data flow inside the code. I see that new layer can be written in python. Do you think that a new input data layer could work nicely or would slow down the cnn performance?
I don't know much about converting and adapting caffe/lmdb interface, but it seems like a very risky path to take if you are not 100% certain with what you are doing.
For instance, you changed the io functions in the python interface, but I don't think caffe is using this interface when running from command line (e.g., $CAFFE_ROOT/build/tools/caffe train ...). Have you looked into the cpp io functions in io.cpp file?
I would strongly suggest an alternative path: use hdf5 binary inputs instead of lmdb.
You can convert the 16bit images you have to float32 and store them in hdf5 files, then input them to caffe via "HDF5Data" layer.
Read more about hdf5 and caffe in this thread.