This openml link seems to be a benchmark for mnist.
There are 13133 runs in there.
What does a run represent? If I submit 2 versions, say vgg16 and svm. Do they count for another 2?
A run represents the result of a flow on a specific task.
So yes it would be two
Related
I am student working with SPSS (statistics) for the first time. I used 1,000 rows of test data to run k-means cluster tool and obtained the results. I now want to take those results and run against a test set (another 1,000) to see how my model did.
I am not sure how to do this; any help is greatly appreciated!
Thanks
For clustering model (or any unsupervised model), there really is no right or wrong result. As such, there is no target variable that you can compare the cluster model result (the cluster allocation) to and the idea of splitting the data set into a training and a testing partition does not apply to these types of models.
The best you can do is to review the output of the model and explore the cluster allocations and determine whether these appear to be useful for the intended purpose.
So I've recently started using Weka and there are several test options when building a tree with, for example, J48. The following are the options, including my undestanding of them:
Use training set - I know just that it's highly optimistic and not necessarily useful. Even Weka's documentation at 2.1.5 isn't being all too specific.
Supplied test set - Pretty self-explanatory, you supply it a test set.
Cross-Validation - I understood it by reading this short example.
Percentage Split - I assume it means partitioning the data set into two sets of a certain percentage, one set for training and one for testing.
What I want to know is what exactly is the training set (first option) and what it does. Where does it get this training set from and what data does it test on exactly? And also if you could correct my understanding of the rest, if it's wrong.
The first option simply means "use all data loaded to run this algorithm". You choose this
to try things out,
to have a first look at the results-section in the output,
to check the performance/run duration,
to check if Weka's output matches the implementation of the same algorithm of a different software, say R or Matlab.
...
Option one is:
test set = training set
The resulting scores will be prone to overfitting of course, and that is why its "highly optimistic and not necessarily useful".
(Since my original question is probably not getting an answer because of too specific about one package, I will ask another general.)
According to the RNN Model, we have an input and output for every step. Let's say a model trained with data of 6 time steps. Of course if I use test data of 6 time steps, I will get outputs, and I have succeed in that. But theoretically, if I only have data of first 3 time steps, I should get an output from the 3rd output node too (without re-train a model with first 3 time steps). But I found at least "keras" package can't do this.
Is there any packages that support such prediction? Better in python language and better to have LSTM layer.
As far as I understand your problem you have two options, depending on your goal.
You can pad sequences in the end with zeros so they are the correct dimension but in the output you just use the first n steps according to your test data dimension.
You can use a stateful implementation
I have written two LSTM RNN codes in python that do sequence-prediction. I have a simple sequence (say a noisy-sinewave) and I am training my networks to "predict" future values along the sinewave. My first code just predicts the single next value (so there is only 1 output neuron), while the second code I wrote predicts the 5-next values (i.e. 5 output neurons). To get the prediction 5-steps in advance for the first code I need to call the predict function several times (utilising the previous predict's output).
Both cases seem to work quite well, but what I'm really trying to work out is which of these two network architectures is best for this problem. There is practically nothing in the literature comparing these output models.
I think using output as an input is not a good idea for this problem. Your output will always have some error and it might increase with each step ( Steady state error ).
I am facing a classification problem, so I thought I could use libSVM and in fact everything works just fine.
Now I would like to introduce some 'tolerance' and see if my system can guess the correct label of the data (which I know a priori) in N guesses (or attempts). What I mean is this: is it possible to have libSVM output not only the label it guesses but also the second-best, third-best, ... ?
EDIT - SOLVED
Actually I 'discovered' that I can use the option -b 1 to ask libSVM to output the probabilities. Then I can just sort them to obtain the N most likely labels.
You mean the underlying probabilities of class classification. This is not produced by support vector machine algorithms. You can however, run an auxiliary model as described in Platt (1999).