Difference between train(), run() and fit() functions in Spark - machine-learning

Question

There are several options of using logistic regression with Apache Spark (Version 1.5.2) in Java:
spark.ml:
1) LogisticRegression lr = new LogisticRegression();
a) lr.train(dataFrame);
b) lr.fit(dataFrame);
spark.mllib:
2) LogisticRegressionWithSGD lr = new LogisticRegressionWithSGD();
a) lr.train(rdd);
b) lr.run(rdd);
3) LogisticRegressionWithLBFGS lr = new LogisticRegressionWithLBFGS();
a) lr.train(rdd);
b) lr.run(rdd);
I was wondering what is the difference between a) and b), except the GeneralizedLinearAlgorithm output from run() function instead of LogisticRegressionModel from the other? I couldn't find any hint in the Java or Scala documentation. Thanks in advance for the help.

Answer 1

Spark does contain two libraries that can be used for machine learning: ML and MLLib. Could you specify which version of Spark you're using please?
MLLib. It was the first machine learning library of Spark. It actually has a very shallow structure and uses RDD to be run on. This is kind of anarchical in MLLib and so you have to look at the code to know which one to use. I'm not sure which language or version you're using, but for Spark 1.6.0 on scala, there's a singleton:
object LogisticRegressionWithSGD {
def train(input: RDD[LabeledPoint], ...) = new LogisticRegressionWithSGD(...).run(input,...)
}
which means that train is to be called as a static method on the object LogisticRegressionWithSGD, but if you have an instance of LogisticRegressionWithSGD there's only a run method:
LogisticRegressionWithSGD.train(rdd, parameters)
// OR
val lr = new LogisticRegressionWithSGD()
lr.run(rdd)
anyway, if you have another version, you sould definitly perfer the super version, i.e. run.
ML. It's the newest library that's based on the use of DataFrame, which is basically an RDD[Row] (a Row is just a sequence of untyped objects) with a schema (ie an object that contains information about the colum names, types, metadata...). I definitly advise you to use this as it enables optimizations! In this case, you should use the fit method which is the method that all the estimators need to implement.
Explanation: The ML library uses the notion of Pipeline (kind of the same as in sci-kit learn). A pipeline instance is basically an array of stages (of type PipelineStage), each one of them being either an Estimator or a Transformer (there are some other types, e.g. Evaluator but I won't get into them here as they are being rare). A Transformer is simply an algorithm that transforms your data, so its main method is transform(DataFrame) and it outputs another DataFrame. An Estimator is a an algorithm that produces a Model (a subtype of Transformer). It's basically any block that needs to fit on data, so it has a function fit(DataFrame) that outputs a Transformer. For instance if you want to multiply all your data by $2$, you only need a transformer that implements a transform method that takes your input and multiply it by $2$. If you need to compute the mean and substract it, you need an estimator that fits on the data to compute the mean and outputs a transformer the substracts the mean learned. So any time you use ML, use the fit and transform methods. It allows you to do something like:
val trainingSet = // training DataFrame
val testSet = // test DataFrame
val lr = new LogisticRegession().setInputCol(...).setOutputCol(...) // + setParams()
val stage = // another stage, i.e. something that implements PipelineStage
val stages = Array(lr, stage)
val pipeline: Pipeline = new Pipeline().setStages(stages)
val model: PipelineModel = pipeline.fit(trainingSet)
val result: DataFrame = model.transform(testSet)
Now if you really want to know why train exists, it's a function inherited by Predictor which itself extends Estimator. Indeed there are a tones of possible Estimators - you could compute the mean, IDF,... When you implement a predictor such as the logistic regression you have an abstract class Predictor that extends Estimator and allows you some shortcuts (e.g. it has a label column, feature column and prediction column). In particular the piece of code already overrides fit to change the schema of your dataframe accordingly to those label/features/prediction and you simply need to implement your own train:
override def fit(dataset: DataFrame): M = {
// This handles a few items such as schema validation.
// Developers only need to implement train().
transformSchema(dataset.schema, logging = true)
copyValues(train(dataset).setParent(this))
}
protected def train(dataset: DataFrame): M
as you see the train method should be protected/private so not used by an external user.