Does anyone know any function for plotting the obtained measures in Caffe? I would like to plot train loss, test loss, and accuracy, train moving average and etc. in one plot. Is there any function except Caffe built-in function that is available online?
Edited:
First, I ran parse_log.py file (the following command):
$python /path/to/caffe/tools/extra/parse_log.py /logfile_path/logfile.log /output_dir
Two files are created based on the log file (lofile.log.train and logfile.log.test). After that,I ran plot_training_log.py file. It has options like:
0: Test accuracy vs. Iters
1: Test accuracy vs. Seconds
2: Test loss vs. Iters
3: Test loss vs. Seconds
4: Train learning rate vs. Iters
5: Train learning rate vs. Seconds
6: Train loss vs. Iters
7: Train loss vs. Seconds
Whenever, I chose option 3, it is showing the following graph:
and by choosing option 0 :
However, whenever, I want to plot train-loss figure, it is giving error:
$python /path/to/caffe/tools/extra/plot_training_log.py.example 6 /output_dir/train_loss_cnn1.png ./logfile.log
Traceback (most recent call last):
File "/home/ss/caffe-master/tools/extra/plot_training_log.py.example", line 191, in <module>
plot_chart(chart_type, path_to_png, path_to_logs)
File "/home/ss/caffe-master/tools/extra/plot_training_log.py.example", line 117, in plot_chart
data = load_data(data_file, x, y)
File "/home/ss/caffe-master/tools/extra/plot_training_log.py.example", line 88, in load_data
data[1].append(float(fields[field_idx1].strip()))
ValueError: invalid literal for float(): 0.522037s/50
My question can be folded into three parts:
Are the plots correct? Is the network behaving well?
From which point this error stem from? I have the following columns in logfile.log.train (#Iters|Seconds |TrainingLoss |LearningRate).
How can I show all chart types in one plot? I tried to include them by comma, like 0,2,3,6, however, it is showing error.
Many thanks in advance.
Take a look at parse_log.py found in $CAFFE_ROOT/tools/extra.
This python utility helps parsing and distilling information from caffe running log.
start training your model by executing the command below:
/home/ubuntu/caffe/build/tools/caffe train --solver /home/ubuntu/yourpath/solver.prototxt 2>&1 | tee /home/ubuntu/yourpath/model_train.log
The training logs will be stored under yourpath/model_train.log.
I haven't looked at caffe's built in plot scripts, but I use the script from here. This only plots your train/test loss, but you can add moving average calculation.
Consider also installing DIGITS, that provides a real-time plot showing all that kind of stuff.
I am trying to use Vowpal Wabbit to do a binary classification, i.e. given feature values vw will classify it either 1 or 0. This is how I have the training data formatted.
1 'name | feature1:0 feature2:1 feature3:48 feature4:4881 ...
-1 'name2 | feature1:1 feature2:0 feature3:5 feature4:2565 ...
etc
I have about 30,000 1 data points, and about 3,000 0 data points. I have 100 1 and 100 0 data points that I'm using to test on, after I create the model. These test data points are classified by default as 1. Here is how I format the prediction set:
1 'name | feature1:0 feature2:1 feature3:48 feature4:4881 ...
From my understanding of the VW documentation, I need to use either the logistic or hinge loss_function for binary classifications. This is how I've been creating the model:
vw -d ../training_set.txt --loss_function logistic/hinge -f model
And this is how I try the predictions:
vw -d ../test_set.txt --loss_function logistic/hinge -i model -t -p /dev/stdout
However, this is where I'm running into problems. If I use the hinge loss function, all the predictions are -1. When I use the logistic loss function, I get arbitrary values between 5 and 11. There is a general trend for data points that should be 0 to be lower values, 5-7, and for data points that should be 1 to be from 6-11. What am I doing wrong? I've looked around the documentation and checked a bunch of articles about VW to see if I can identify what my problem is, but I can't figure it out. Ideally I would get a 0,1 value, or a value between 0 and 1 which corresponds to how strong VW thinks the result is. Any help would be appreciated!
If the output should be just -1 and +1 labels, use the --binary option (when testing).
If the output should be a real number between 0 and 1, use --loss_function=logistic --link=logistic. The loss_function=logistic is needed when training, so the number can be interpreted as probability.
If the output should be a real number between -1 and 1, use --link=glf1.
If your training data is unbalanced, e.g. 10 times more positive examples than negative, but your test data is balanced (and you want to get the best loss on this test data), set the importance weight of the positive examples to 0.1 (because there are 10 times more positive examples).
Independently of your tool and/or specific algorithm you can use "learning curves" ,and train/cross validation/test splitting to diagnose your algorithm and determine whats your problem . After diagnosing your problem you can apply adjustments to your algorithm, for example if you find you have over-fitting you can apply some actions like:
Add regularization
Get more training data
Reduce the complexity of your model
Eliminate redundant features.
You can reference Andrew Ng. "Advice for machine learning" videos on YouTube to more details on this subject.
I'm trying to train a logistic regression model in mahout. The command I use is this:mahout trainlogistic --input /home/cloudera/Desktop/final.csv --output /home/cloudera/Desktop/model/model --target Action --predictors Open High Close --types word --features 20 --passes 100 --rate 50 --categories 2
The files I use actually exist. I'm reading a book that says that I should expect an output that looks like
Action ~ 647.186*Close+-44.975*High+3.269*Intercept term +-601.454*Open
and then a 4x2 matrix.
What I actually get is a terminal being filled with calculations, no Action ~, and a 5x4 matrix.
What am I doing wrong?
Well, the type of my predictors was numeric; why did the book I referenced call them words I have no idea.
I am new to Weka. I am trying to classify text documents after OCR process. The training corpus contains 286 mortgage documents and 57 note documents. The test dataset contains 1-100 text pages. So each line of the training and test dataset contains few paragraphs of text data. After classification text documents should be classified into mortgage or note properly.
I am doing a StringToWordVector operation combining both Training and Test dataset with missing values from Test dataset i.e. "?".
Steps are as follows:
Create training Arff file using following command line:
java -cp weka.jar weka.core.converters.TextDirectoryLoader -dir <text directory>
This creates a training dataset with known classes i.e. mortgage, note
Create test Arff file with missing classes i.e "?"
Combine both training and test dataset
Run the classifier with following command line:
java -cp weka.jar weka.classifiers.meta.FilteredClassifier -t train.arff -test.arff -F "weka.filters.MultiFilter -F weka.filters.unsupervised.attribute.StringToWordVector -F weka.filters.unsupervised.attribute.Standardize" -d trained.model -p 0
I am running the above example from both Weka GUI and from command line as well. Everything works fine as far as commands are concerned. The results are abnormal. Not at all correct.
I have also tried to run StringToWordVector operation separately and tested through NaiveBayes, NaiveBayesMultiNomial, J48 and other multiclass classifiers on the dataset but classification prediction is not correct. Always giving abnormal results.
Please help me to get the proper prediction result. Let me know if the above steps are correct and if I am doing anything wrong.
I am trying to figure out WEKA and perform some experiments with data that I have.
Basically what I want to do is take Data Set 1, use it as a training set. Run a J48 Decision Tree on it. Then take Data Set 2 and run the trained tree on it, with the output of the original data set with a extra column for what the prediction was.
Then do the same thing again with the Bayes Neural Network.
Can someone point me to a link of detail instructions on how exactly I would accomplish this? I seem to be missing some steps and cannot get the output of the original data set with the extra column.
Here is one way to do it with the command-line. This information is found in Chapter 1 ("A command-line primer") of the Weka manual that comes with the software.
java weka.classifiers.trees.J48 -t training_data.arff -T test_data.arff -p 1-N
where:
-t <training_data.arff> specifies the training data in ARFF format
-T <test_data.arff> specifies the test data in ARFF format
-p 1-N specifies that you want to output the feature vector and the prediction,
where N is the number of features in your feature vector.
For example, here I am using soybean.arff for both training and testing. There are 35 features in the feature vector:
java weka.classifiers.trees.J48 -t soybean.arff -T soybean.arff -p 1-35
The first few lines of the output look like:
=== Predictions on test data ===
inst# actual predicted error prediction (date,plant-stand,precip,temp,hail,crop-hist,area-damaged,severity,seed-tmt,germination,plant-growth,leaves,leafspots-halo,leafspots-marg,leafspot-size,leaf-shread,leaf-malf,leaf-mild,stem,lodging,stem-cankers,canker-lesion,fruiting-bodies,external-decay,mycelium,int-discolor,sclerotia,fruit-pods,fruit-spots,seed,mold-growth,seed-discolor,seed-size,shriveling,roots)
1 1:diaporth 1:diaporth 0.952 (october,normal,gt-norm,norm,yes,same-lst-yr,low-areas,pot-severe,none,90-100,abnorm,abnorm,absent,dna,dna,absent,absent,absent,abnorm,no,above-sec-nde,brown,present,firm-and-dry,absent,none,absent,norm,dna,norm,absent,absent,norm,absent,norm)
2 1:diaporth 1:diaporth 0.952 (august,normal,gt-norm,norm,yes,same-lst-two-yrs,scattered,severe,fungicide,80-89,abnorm,abnorm,absent,dna,dna,absent,absent,absent,abnorm,yes,above-sec-nde,brown,present,firm-and-dry,absent,none,absent,norm,dna,norm,absent,absent,norm,absent,norm)
The columns are: (1) data instance number; (2) ground truth label; (3) predicted label; (4) error; (5) prediction confidence; and (6) feature vector.