Caffe, joining outputs from 2 models - machine-learning

Question

I have 2 different models, let's say NM1 and NM2.
So, what I'm looking is something that works like in the example below.
Let's say that we have a picture of a dog.
NM1 predicts that it's a cat on the picture with a probability 0.52 and that it's a dog with a probability 0.48.
NM2 predicts that it's a dog with a probability 0.6 and that it's a cat with a probability 0.4.
NM1 - will predict wrong
NM2 - will predict correctly
NM1 + NM2 - connection will predict correctly (because 0.48 + 0.6 > 0.52 + 0.4)
So, each model ends with InnerProducts (after Softmax) which give me 2 vectors of probabilities.
Next step, I have those 2 vectors and I want to add them. Here I use Eltwise layer.
layer {
name: "eltwise-sum"
type: "Eltwise"
bottom: "fc8"
bottom: "fc8N"
top: "out"
eltwise_param { operation: SUM }
}
Before joining NM1 had accuracy ~70% and NM2 ~10%.
After joining accuracy can't reach even 1%.
Thus, my conclusion is that I understand something wrong and I'd be grateful if someone could explain to me where I'm wrong.
PS. I did turn off shuffle when creating lmdb.
UPDATE
layer {
name: "eltwise-sum"
type: "Eltwise"
bottom: "fc8L"
bottom: "fc8NL"
top: "out"
eltwise_param {
operation: SUM
coeff: 0.5
coeff: 0.5
}
}
#accur for PI alone
layer {
name: "accuracyPINorm"
type: "Accuracy"
bottom: "fc8L"
bottom: "label"
top: "accuracyPiNorm"
include {
phase: TEST
}
}
#accur for norm images alone
layer {
name: "accuracyIMGNorm"
type: "Accuracy"
bottom: "fc8NL"
bottom: "labelN"
top: "accuracyIMGNorm"
include {
phase: TEST
}
}
#accur for them together
layer {
name: "accuracy"
type: "Accuracy"
bottom: "out"
bottom: "label"
top: "accuracy"
include {
phase: TEST
}
}

Answer 1

If you want to add (element-wise) the probabilities, you need to add after the "Softmax" layer, and not after the "InnerProduct" layer. You should have something like
layer {
type: "InnerProduct"
name: "fc8"
top: "fc8"
# ...
}
layer {
type: "Softmax"
name: "prob_nm1"
top: "prob_nm1"
bottom: "fc8"
}
layer {
type: "InnerProduct"
name: "fc8N"
top: "fc8N"
# ...
}
layer {
type: "Softmax"
name: "prob_nm2"
top: "prob_nm2"
bottom: "fc8N"
}
# Joining the probabilites
layer {
type: "Eltwise"
name: "prob_sum"
bottom: "prob_nm1"
bottom: "prob_nm2"
top: "prob_sum"
eltwise_param {
operation: SUM
coeff: 0.5
coeff: 0.5
}
}