I run caffe using an image_data_layer and don't want to create an LMDB or LevelDB for the data, But The compute_image_mean tool only works with LMDB/LevelDB databases.
Is there a simple solution for creating a mean file from a list of files (the same format that image_data_layer is using)?
You may notice that recent models (e.g., googlenet) do not use a mean file the same size as the input image, but rather a 3-vector representing a mean value per image channel. These values are quite "immune" to the specific dataset used (as long as it is large enough and contains "natural images").
So, as long as you are working with natural images you may use the same values as e.g., GoogLenet is using: B=104, G=117, R=123.
The simplest solution is to create a LMDB or LevelDB database of the image set.
The complicated solution is to write a tool similar to compute_image_mean, which takes image inputs and do the transformations and find the mean!
Related
This is a bit of an abstract question.
I have a group of 28x28 px images from certain people, and I would like to label that data with each person who wrote it. How would I go about labeling it for training and testing? This is my first neural network, and I'm having difficulty finding any tutorials that suit my particular need. It feels like most Data, like MNIST/EMNIST, are already labeled.
Some more info is that I'm using Python 3, and Keras with Tensorflow backend.
I am assuming that you know who wrote each image. Then this is a matter of associating that information (the class label) with each image. There are several ways of doing this. Two common approaches are:
Folder structure
Make a folder for each class (person), and put the images inside.
Folder contents:
john/01.png
john/02.png
jane/03.png
susan/...
CSV file
In this case the images can be all in one folder, and then a dedicate Comma-Separated-Values file is used to contain
Folder contents:
dataset.csv
images/01.png
images/02.png
images/03.png
images/....
dataset.csv contents:
filename,person
images/01.png,john
images/02.png,john
images/03.png,jane
...
The CSV approach is nice if you have additional data about each file that you want to store. For instance metadata that could be relevant such as who recorded the file, when was it recorded, with what kind of equipment, what locations etc.
Combinations of the two are also possible, of course.
I'm building the neural network and should test in on modified CIFAR-10. I have used keras.datasets.cifar10.load_data() for retrieving the dataset and then parsed it in a dict using pickle.load(datafile, encoding='bytes'). After some modifications I've written the images in keras-like format using pickle.dump().
I noticed that the resulting file after pickle.dump() is 53 bytes bigger then the source file. Even if I don't make any modifications and use the dump() right away after load() the resulting file has extra 53 bytes. Looks like the structure of the resulting file isn't violated because I'm able to restore images, labels, filenames from it and they are correct. But if I'm learning and testing the neural network (even the simplest NN from the example!) I'm getting very bad score (~0.5).
Please help me to figure out how the loading-dumping affects the NN's result if the structure in general doesn't change?
How I can load and dump to leave the structure and the size of files without changes? How to avoid the inconsistency of load-dump operation?
P.S. Looks like the dump() writes some header to the file and doesn't write it if the header already exists (i've tried to apply load-dump twice but the size was changes only at first applying). But how I can avoid writing this header?
Please help me to figure out how the loading-dumping affects the NN's
result if the structure in general doesn't change?
If the structure does not change, it has no effect to the network. You can simple dump with human readable outputs and compare the files. First read about protocols here.
How I can load and dump to leave the structure and the size of files
without changes? How to avoid the inconsistency of load-dump
operation?
there is no real inconsistancy, its probably just using another protocol or attaching headers. You should not deal with these internals. If you want to make your model human readable or put it under version control, you can use json or another human readbale protocol. (e.g. simplejson)
Apologies for tagging this just ImageJ - it's a problem regarding MicroManager, a microscopy plugin for it and I thought this would be best.
I'd recently taken images for an important experiment using MicroManager (a recent version, though I cannot recall the exact number). The IT services at my institution have recently been having some networking problems and my saved preferences for the software had been erased. I'd got half way through my experiment when I realised that I'd saved my images as separate image files (three greyscale TIFFs plus metadata text files) instead of OME-TIFF iamge stacks.
All of my ImageJ macros for image processing rely on having a multiple channel image stack, so this is a bit of a problem. Is there any easy way in MicroManager (or ImageJ) to bulk convert these single channel greyscale images into the OME-TIFF image stack after the images have already been taken?
Cheers.
You can start with a macro like this one:
// Convert your images to a stack
run("Images to Stack", "name=Stack title=[] use");
// The stack will default the images to time points. Convert to channels
run("Stack to Hyperstack...", "order=xyczt(default) channels=3 slices=1 frames=1 display=Color");
// Export as OME-TIFF
run("Bio-Formats Exporter");
This is designed to reconstruct one dataset at a time (open 3 images, run the macro and export the OME-TIFF).
If you don't want any dialogs to show you can pass an output directory to the Bio-Formats exporter:
run("Bio-Formats Exporter", "save=/path/to/image.ome.tif export compression=Uncompressed");
For the output file name you can get the original image name in the macro with getTitle()
There is also a template example on iterating over all the files in a directory, if you want to completely automate the macro. However this may take some tweaking since you want to operate on your images 3 at a time.
Hope that helps!
New user of hadoop and mapreduce, i would like to create a mapreduce job to do some measure on images. this why i would like to know if i can passe an image as input to mapreduce?if yes? any kind of example
thanks
No.. you cannot pass an image directly to a MapReduce job as it uses specific types of datatypes optimized for network serialization. I am not an image processing expert but I would recommend to have a look at HIPI framework. It allows image processing on top of MapReduce framework in a convenient manner.
Or if you really want to do it the native Hadoop way, you could do this by first converting the image file into a Hadoop Sequence file and then using the SequenceFileInputFormat to process the file.
Yes, you can totally do this.
With the limited information provided, I can only give you a very general answer.
Either way, you'll need to:
1) You will need to write a custom InputFormat that instead of taking chunks of files in HDFS locations (like TextInputFormat and SequenceFileInputFormat do), it actually passes to each map task the Image's HDFS path name. Reading the image from that won't be too hard.
If you plan to have a Reduce phase in which Images are passed around through the framework, you'll need to:
2) You will need to make an "ImageWritable" class that implements Writable (or WritableComparable if you're keying on the image). In your write() method, you'll need to serialize your image to a byte array. When you do this, what I would do is first write to the output an int/long which is the size of the array you're going to write. Lastly, you'll want to write the array as bytes.
In your read() method, you'll read an int/long first (which will describe the payload of the image), create an byte array of this size, and then read the bytes fully into your byte array up to the length of your int/long that you captured.
I'm not entirely sure what you're doing, but that's how I'd go about it.
I'm using cv::PCA class for a face recognition project. I convert photos of faces to one row vectors, concatenate them to one big array and feed to pca, to acquire a new space in which I can try to use distance for recognition. Problem is, that calculating the pca from scratch each time I start the program is really time consuming (almost five minutes). I figured out that I need to save the calculated pca to hard drive, and load it when I start the program again. And here is the problem. As I can see, all cv::Mat objects in cv::PCA are of type CV_32F. When i try to save it as a normal picture, its converted to 8 bit image, and there is some data lost. When i use XML/YAML persistence, the generated file is really big, and data is also lost (I have saved it, loaded to another structure and ran cerr<<sum(pca_orginal.mean==pca_loaded.mean)[0]<<endl to check how big is the difference). Right now I'm trying to use std::ofstream::write with std::ofstream::binary flag, and istream::read, but there are some type issues (out.write(_pca.mean.data,_pca.mean.rows*_pca.mean.cols*4/*CV_32F->4*CV_8U*/\); generates error: no matching function for call to ‘std::basic_ofstream<char, std::char_traits<char> >::write(uchar*&, int). I've also heard about openexr library and it's file format, but I would rather avoid using additional libraries. I'm using OpenCV 2.3.1 and OpenCV 2.2.
edit:
I'm sorry for the confusion. I misread cv::Mat operator== description, and thought that it works the opposite way that it does, so sum(pca_orginal.mean==pca_loaded.mean)[0] giving 0 is the worse possible result, not the best. It means that XML/YML works fine apart from generating huge files. Also, after using c-style casting I was able to make the binary streams work, but the files generated are also big (over 150MB).
In the C interface, there are functions cvSave and cvLoad for saving arbitrary matrices. There are probably C++ interface counterparts, too.