Weka ThresholdSelector and CostSensitiveClassifier on stream learning - stream

Question

Are Weka's ThresholdSelector and/or CostSensitiveClassifier compatible with stream learning (updatable classifiers) ? My goal is to use them with weka.classifiers.meta.MOA to focus learning on a specific class and minimize FN on some imbalanced data.
Thanks a lot!

Answer 1

Following my post on Weka Pentaho forum the answer is that nor ThresholdSelector nor CostSensitiveClassifier supports updateable classifiers. So streaming learning with those meta classifiers is not currently possible.
Therefore, I proposed a draft code to create an updateable version of those classifiers. Any comment/suggestion would be highly welcome.
weka.classifiers.meta.CostSensitiveClassifier code update to create an updateable version (this one "seems" the most straightforward)
/*
weka.classifiers.meta.CostSensitiveClassifier: draft code update and questions to make it compatible with updateable classifiers
*/
import weka.classifiers.UpdateableClassifier;
....
implements ... UpdateableClassifier;
...
protected boolean classifierAlreadyUpdated = False;
public void updateClassifier(Instance instance) throws Exception {
if (!instance.classIsMissing()) {
if (m_Classifier == null)
throw new Exception("No base classifier has been set!");
// not sure on how to properly check if m_CostMatrix has already been fully intialized here or from elsewhere (ie. external call to buildClassifier)
if (m_CostMatrix is null || (m_CostMatrix.size() == 1 && !classifierAlreadyUpdated)) {
buildClassifier(new Instances[] {instance}); // re-use intialization process from buildClassifier
classifierAlreadyUpdated = True;
}
else {
double factor = 1.0;
int classValIndex = (int) instance.classValue();
Object element = (classValIndex == 0) ? m_CostMatrix.getCell(classValIndex, 1) : m_CostMatrix.getCell(classValIndex, 0);
if (element instanceof Double) {
factor = ((Double) element).doubleValue();
} else {
factor = ((AttributeExpression) element).evaluateExpression(instance);
}
double weightOfInstance = instance.weight() * factor;
if (!m_MinimizeExpectedCost) {
((UpdateableClassifier)m_Classifier).updateClassifier(instance.setWeight(weightOfInstance));
} else {
((UpdateableClassifier)m_Classifier).updateClassifier(instance);
}
}
}
}
weka.classifiers.meta.ThresholdSelector code update to create an updateable version (waiting for your comments/suggestions):
/*
weka.classifiers.meta.ThresholdSelector draft code update and questions to make it compatible with updateable classifiers
I've got the big picture but I would need some help on findThreshold and the evaluation mode
findThreshold:
double low, high, maxValue and Instance maxInst => should become protected class properties in order
to keep them updated across build&all updates and could be resetted when calling buildClassifier
Evaluation mode and getPredictions: should I create a new Evaluation mode ?
EVAL_TRAINING_SET does not seem a good option as it would skip the updateClassifier
I could then modify toString and add the code below to getPredictions ?
case EVAL_STREAM:
return eu.getTrainTestPredictions(m_Classifier, instances, instances);
For updateClassifier, please find below a draft code
*/
import weka.classifiers.UpdateableClassifier;
....
implements ... UpdateableClassifier;
...
protected boolean classifierAlreadyUpdated = False;
public void updateClassifier(Instance instance) throws Exception {
if (!instance.classIsMissing()) {
if (m_Classifier == null)
throw new Exception("No base classifier has been set!");
// Don't know how to properly check if m_CostMatrix has already been fully intialized here or from elsewhere
if (!classifierAlreadyUpdated)) {
buildClassifier(new Instances[] {instance}); // re-use intialization process from buildClassifier
classifierAlreadyUpdated = True;
}
else {
// If data contains only one instance of positive data
// optimize on training data
if (stats.distinctCount != 2) {
System.err.println("Couldn't find examples of both classes. No adjustment.");
m_Classifier.updateClassifier(instance);
}
else {
// m_DesignatedClass: already initialized via buildClassifier (called if needed during first update)
if (m_manualThreshold) {
m_Classifier.updateClassifier(instance);
return;
}
if (stats.nominalCounts[m_DesignatedClass] == 1) {
System.err.println("Only 1 positive found: optimizing on training data");
findThreshold(getPredictions(new Instances[] {instance}, EVAL_TRAINING_SET, 0));
} else {
int numFolds = Math.min(m_NumXValFolds, stats.nominalCounts[m_DesignatedClass]);
findThreshold(getPredictions(new Instances[] {instance}, m_EvalMode, numFolds));
if (m_EvalMode != EVAL_TRAINING_SET) {
m_Classifier.updateClassifier(instance);
}
}
}
}
}
Thanks