Caffe HDF5 Output Layer error - looking for working example - hdf5

Question

All - I'm having some trouble getting HDF5 output files to work in Caffe. Has anyone successfully used the HDF5 output file layer in Caffe? If so could you provide an example or help me debug my definition. I'm not able to find any public examples/tutorials using the HD5F output feature, so I'm afraid it might not be very robust yet. Thanks in advance
Here is my prototxt:
layer {
type: "HDF5Output"
name: "hdf5output"
bottom: "Ytest" #
bottom: "ip2" #
hdf5_output_param {
file_name: "./datah5/output.h5"
}
include { phase: TEST }
}
The Caffe error snip is copied below. The output file 'outfile.h5' exists and has some data in it. HDF5 input seems to work ok
I0803 20:30:36.776832 27929 solver.cpp:338] Iteration 0, Testing net (#0)
I0803 20:30:36.785679 27929 hdf5_output_layer.cpp:32] Saving HDF5 file ./datah5/output.h5
I0803 20:30:36.785854 27929 hdf5_output_layer.cpp:37] Successfully saved 100 rows
I0803 20:30:36.792243 27929 hdf5_output_layer.cpp:32] Saving HDF5 file ./datah5/output.h5
HDF5-DIAG: Error detected in HDF5 (1.8.11) thread 70366426137120:
#000: ../../../src/H5D.c line 170 in H5Dcreate2(): unable to create dataset
major: Dataset
minor: Unable to initialize object
#001: ../../../src/H5Dint.c line 439 in H5D__create_named(): unable to create and link to dataset
major: Dataset
minor: Unable to initialize object
#002: ../../../src/H5L.c line 1638 in H5L_link_object(): unable to create new link to object
major: Links
minor: Unable to initialize object
#003: ../../../src/H5L.c line 1882 in H5L_create_real(): can't insert link
major: Symbol table
minor: Unable to insert object
#004: ../../../src/H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal failed
major: Symbol table
minor: Object not found
#005: ../../../src/H5Gtraverse.c line 641 in H5G_traverse_real(): traversal operator failed
major: Symbol table
minor: Callback failed
#006: ../../../src/H5L.c line 1674 in H5L_link_cb(): name already exists
major: Symbol table
minor: Object already exists
F0803 20:30:36.792457 27929 hdf5.cpp:101] Check failed: status >= 0 (-1 vs. 0) Failed to make float da
taset data
*** Check failure stack trace: ***
# 0x3fff835520f0 (unknown)

Answer 1

Your.prototxt if perfectly fine. The reason you are getting this error is because you are running the network for multiple iterations. In the first iteration, the HDF5Output layer will create a file with the name ./datah5/output.h5. But in the next iteration, it will again try to create a file with the same name and fail. It is failing because the file already exists.
To deal with this you can do two things.
Run only a single batch at a time. Take the output and rename/move it before running another batch.
Edit the caffe code to use the iteration count in the output file name. Refer this - https://groups.google.com/forum/#!topic/caffe-users/zkGKk5UbInI