As much as I understand, I could parse the support vectors from the model produced by training with a set of data with LibSVM.
What would be the formula, to produce the classifier?
Do I need the data in the headers of the file, like the following (kernel etc...before the listed support vectors):
svm_type c_svc
kernel_type rbf
gamma 0.125
nr_class 4
total_sv 1038
rho -0.859244 -0.876628 -0.958343 0.543365 -1.10722 -1.79433
label 2 1 3 0
nr_sv 364 276 242 156
SV
My case is
I want to do classification from Node.JS. But there isn't any bindings for LibSVM for it, yet.
Since my models are not going to change, I would like to do the classification in Node.JS, holding the model in-memory.
If this proves to be slow, I rather write the same classification from scratch in C++ and create a wrapper module if it's only a matter of a simple computation (as I suspect it is).
Thanks.
You should be able to translate the C function to Javascript.
Here is the relevant code:
double svm_predict_values(const svm_model *model, const svm_node *x, double* dec_values)
{
int i;
int nr_class = model->nr_class;
int l = model->l;
double *kvalue = Malloc(double,l);
for(i=0;i<l;i++)
kvalue[i] = Kernel::k_function(x,model->SV[i],model->param);
int *start = Malloc(int,nr_class);
start[0] = 0;
for(i=1;i<nr_class;i++)
start[i] = start[i-1]+model->nSV[i-1];
int *vote = Malloc(int,nr_class);
for(i=0;i<nr_class;i++)
vote[i] = 0;
int p=0;
for(i=0;i<nr_class;i++)
for(int j=i+1;j<nr_class;j++)
{
double sum = 0;
int si = start[i];
int sj = start[j];
int ci = model->nSV[i];
int cj = model->nSV[j];
int k;
double *coef1 = model->sv_coef[j-1];
double *coef2 = model->sv_coef[i];
for(k=0;k<ci;k++)
sum += coef1[si+k] * kvalue[si+k];
for(k=0;k<cj;k++)
sum += coef2[sj+k] * kvalue[sj+k];
sum -= model->rho[p];
dec_values[p] = sum;
if(dec_values[p] > 0)
++vote[i];
else
++vote[j];
p++;
}
int vote_max_idx = 0;
for(i=1;i<nr_class;i++)
if(vote[i] > vote[vote_max_idx])
vote_max_idx = i;
free(kvalue);
free(start);
free(vote);
return model->label[vote_max_idx];
}
Notice that you have to recreate this equation:
The only difference is since your model has 4 classes, you need to implement the vote system which is basically the code above.
Hope it helps.
Related
Currently i'm working with an custom implementation of the Mersenne Twister, and i'd like to improve my understanding of vector operations.
I have the following code:
#define N 624
#define M 397
for( k = N -1; k; k-- )
{
array[i] = (array[i] ^ ((array[i-1] ^ (array[i-1] >> 30)) * 1566083941UL)) - i;
array[i] &= 0xffffffffUL;
++i;
if ( i >= N )
{
array[0] = array[N-1];
i = 1;
}
}
Here i'm working with 32 bit integers only, so as i understand, I could perform 8 times as much operations at the same time, using AVX2 instructions? How can I do that in practice?
I know how to deal with addition of 2 vectors, but this case seems to be more complicated. I don't know how to begin.
For a scalar approach i'd work like that, but i'd like to get sure how to perform these actions in my case.
for (i = 0; i < 1024; i++)
{
C[i] = A[i]*B[i];
}
for (i = 0; i < 1024; i+=4)
{
C[i:i+3] = A[i:i+3]*B[i:i+3];
}
Unfortunately at my university there are no lessons about intrinsics, but i'm quite curious in order to get an improvement.
I'm also doing some thoughts, about how to create the array using vectors? Maybe matrix? (Maybe _mm256_setr_epi32)
I hope to get some advice regarding this topic!
I want to modify the existing softmaxloss in Caffe. The idea is to add a weight factor to the loss. For instance, if we are processing a pixel that belongs to car class, I want to put a factor 2 to the loss, because in my case, the detection of car class is more important than the dog class(for example). This is the original source code:
__global__ void SoftmaxLossForwardGPU(const int nthreads,
const Dtype* prob_data, const Dtype* label, Dtype* loss,
const int num, const int dim, const int spatial_dim,
const bool has_ignore_label_, const int ignore_label_,
Dtype* counts) {
CUDA_KERNEL_LOOP(index, nthreads) {
const int n = index / spatial_dim;
const int s = index % spatial_dim;
const int label_value = static_cast<int>(label[n * spatial_dim + s]);
if (has_ignore_label_ && label_value == ignore_label_) {
loss[index] = 0;
counts[index] = 0;
} else {
loss[index] = -log(max(prob_data[n * dim + label_value * spatial_dim + s],
Dtype(FLT_MIN)));
counts[index] = 1;
}
}
}
You can find this code in https://github.com/BVLC/caffe/blob/master/src/caffe/layers/softmax_loss_layer.cu
In the following code you can find the modifications that I do in order to achieve my objective:
__global__ void SoftmaxLossForwardGPU(const int nthreads,
const Dtype* prob_data, const Dtype* label, Dtype* loss,
const int num, const int dim, const int spatial_dim,
const bool has_ignore_label_, const int ignore_label_,
Dtype* counts) {
const float weights[4]={3.0, 1.0, 1.0, 0.5}
CUDA_KERNEL_LOOP(index, nthreads) {
const int n = index / spatial_dim;
const int s = index % spatial_dim;
const int label_value = static_cast<int>(label[n * spatial_dim + s]);
if (has_ignore_label_ && label_value == ignore_label_) {
loss[index] = 0;
counts[index] = 0;
} else {
loss[index] = -log(max(prob_data[n * dim + label_value * spatial_dim + s],
Dtype(FLT_MIN))) * weights[label_value];
counts[index] = 1;
}
}
}
I am not sure if this modification is doing what I want to do. For several reasons:
I am not sure what means each values of this function. I am supposing for instance the label_value corresponds to the ground truth value, but
I am not sure.
I completely do not understand this line: prob_data[n * dim + label_value * spatial_dim + s]. Where is the loss estimated here? I am supposing the loss calculation is happening in this line, and for that reason I'm putting my weights here, but I can't see the calculation here. Here I can see an access to a specific position of the vector prob_dat.
I know my code proposal is not the best one, I would like at some point to convert these weights into an input of the layer, but right now I don't have enough knowledge to do it (if you can also give me some hints in order to achieve it, that would be great).
Implementing your own layer in caffe is a very nice skill to have, but you should do this as a "last resort". There are quite a few existing layers and you can usually achieve what you want using existing layer(s).
You cannot modify the forward_gpu implementation without modifying forward_cpu as well. More importantly, you MUST modify the backward functions as well - otherwise the gradients updating your weights will not reflect your modified loss.
"SoftmaxWithLoss" layer is a special case of the loss "InfogainLoss" layer. If you want to have different "weight" for each class, you can simply use "InfogainLoss" with weight matrix H according to your weights.
If you want to have spatially varying weight (different weight for different location) you can look at PR #5828, implementing "WeightedSoftmaxWithLoss".
I'm trying to write a MLP that classifies input into three objects.
I have a number that represents each object.
1-10 : Banana
11-20 : Apple
21:30 : Carrot
There's only two layers in MLP: one hidden layer (2 units) and one output layer (3 units).
Each unit has:
inputs[] (inputs that were passed to this unit)
weights[]
delta
sum (summed up weights with inputs)
output (activated sum)
also each unit has an activation function:
double activate(double[] inputs) {
this.inputs = inputs;
sum = 0;
for (int i = 0; i < inputs.length; i++)
sum += weights[i] * inputs[i];
output = 1.0 / (1.0 + (Math.exp(-sum))); // activation
return output;
}
and a function to correct weights:
void correctWeights(double momentum, double learningRate) {
for (int i = 0; i < weights.length; i++) {
weights[i] = weights[i] * momentum + learningRate * delta * (output * (1 - output)) * inputs[i];
}
}
where (output * (1 - output)) is the derivative.
To train the network I have a function that loops N times, in the loop I generate the input relative to the object, then propagate it to the network and use back-propagation.
private void train() {
for (int i = 0; i < 10000; i++) {
int[] expectedOutput = new int[3];
double[] inputs = {ThreadLocalRandom.current().nextInt(1, 30 + 1)};
if (inputs[0] <= 10) {
expectedOutput[0] = 1;
expectedOutput[1] = 0;
expectedOutput[2] = 0;
}
if (inputs[0] <= 20 && inputs[0] > 10) {
expectedOutput[0] = 0;
expectedOutput[1] = 1;
expectedOutput[2] = 0;
}
if (inputs[0] <= 30 && inputs[0] > 20) {
expectedOutput[0] = 0;
expectedOutput[1] = 0;
expectedOutput[2] = 1;
}
double[] outputs = propagate(inputs);
backPropagate(expectedOutput, outputs);
}
}
Propagation function just goes through the whole net and activates the units.
private double[] propagate(double[] inputs) {
double[] hiddenOutputs = new double[hiddenLayer.length];
for (int i = 0; i < hiddenLayer.length; i++)
hiddenOutputs[i] = hiddenLayer[i].activate(inputs);
double[] outputs = new double[outputLayer.length];
for (int i = 0; i < outputs.length; i++)
outputs[i] = outputLayer[i].activate(hiddenOutputs);
return outputs;
}
Back-propagation algorithm was taken from http://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html
private void backPropagate(int[] expectedOutput, double[] output) {
for (int i = 0; i < outputLayer.length; i++) {
outputLayer[i].setDelta(expectedOutput[i] - output[i]);
}
for (int i = 0; i < hiddenLayer.length; i++) {
double delta = 0;
for (int j = 0; j < outputLayer.length; j++) {
delta += outputLayer[j].getDelta() * outputLayer[j].getWeight(i);
}
hiddenLayer[i].setDelta(delta);
}
for (int i = 0; i < hiddenLayer.length; i++)
hiddenLayer[i].correctWeights(momentum, learningRate);
for (int i = 0; i < outputLayer.length; i++)
outputLayer[i].correctWeights(momentum, learningRate);
}
It also has a function to recognize objects after it got trained
private void recognize(String number) {
double[] inputs = {Double.parseDouble(number)};
double[] outputs = propagate(inputs);
System.out.println("Banana: " + outputs[0]);
System.out.println("Apple: " + outputs[1]);
System.out.println("Carrot: " + outputs[2]);
}
So the problem is that when I pass any number to the recognize function I get the output similar to this:
Banana: 0.49984367018594233
Apple: 0.49984367018594233
Carrot: 0.5001563298140577
Carrot is being chosen every time (also carrot is the last trained object by the net). So if I input 5 it will output that it's a carrot. if I input 15 it will output that it's a carrot. If I change the order of the objects that are being learned in train function and make the banana to be the last learned object then the net will always pick the banana as its answer.
I've been working on this for a few days now and I couldn't find any solution to it, please help me, what am I doing wrong?
I notice that you select a random number between 0-30 and then determine an output for it, however you are forgetting to normalize the input. Neural networks function best if the input is within the range of 0-1 (depends on which activation function you use).
So what is left for you to do is to do normalize this input. That means, converting the inputs equally to a number between 0 and 1.
Your input is a numerical value, so all you have to do is choose a maximum value with which you divide all the values. In your case, this could be 30, as there is no input higher than 30. So each number gets converted as follows:
10 -> 10 / 30 -> 0.33
15 -> 15 / 30 -> 0.50
etc.
Read more about normalization here.
I have a real 2d matrix. I am taking its fft using fftw. But the result of using a real to complex fft is different from a complex ( with imaginary part equal to zero) to complex fft.
real matrix
0 1 2
3 4 5
6 7 8
result of real to complex fft
36 -4.5+2.59808i -13.5+7.79423i
0 -13.5-7.79423i 0
0 0 0
Code:
int r = 3, c = 3;
int sz = r * c;
double *in = (double*) malloc(sizeof(double) * sz);
fftw_complex *out = (fftw_complex*) fftw_malloc(sizeof(fftw_complex) * sz);
fftw_plan p = fftw_plan_dft_r2c_2d(r, c, in, out, FFTW_MEASURE);
for ( int i=0; i<r; ++i ){
for ( int j=0; j<c; ++j ){
in[i*c+j] = i*c + j;
}
}
fftw_execute(p);
using a complex matrix with imaginary part of zero
complex matrix
0+0i 1+0i 2+0i
3+0i 4+0i 5+0i
6+0i 7+0i 8+0i
result of complex to complex fft
36 -4.5 + 2.59808i -4.5 - 2.59808i
-13.5 + 7.79423i 0 0
-13.5 - 7.79423i 0 0
Code:
int r = 3, c = 3;
int sz = r * c;
fftw_complex *out = (fftw_complex*) fftw_malloc(sizeof(fftw_complex) * sz);
fftw_complex *inc = (fftw_complex*) fftw_malloc(sizeof(fftw_complex) * sz);
p = fftw_plan_dft_2d( r,c, inc, out, FFTW_FORWARD,FFTW_MEASURE);
for ( int i=0; i<r; ++i ){
for ( int j=0; j<c; ++j ){
inc[i*c+j][0] = i*c+j;
inc[i*c+j][1] = 0;
}
}
fftw_execute(p);
I am after the result of complex to complex fft. But the real to complex fft is much faster and my data is real. Am I making a programming mistake or the result should be different?
As indicated in FFTW documentation
Then, after an r2c transform, the output is an n0 × n1 × n2 × … × (nd-1/2 + 1) array of fftw_complex values in row-major order
In other words, the output for your real-to-complex transform of your sample real matrix really is:
36 -4.5+2.59808i
-13.5+7.79423i 0
-13.5-7.79423i 0
You may notice that these two columns match exactly the first two columns of your complex-to-complex transform. The missing column is omitted from the real-to-complex transform since it is redundant due to symmetry. As such, the full 3x3 matrix including the missing column could be constructed using:
fftw_complex *outfull = (fftw_complex*) fftw_malloc(sizeof(fftw_complex) * sz);
int outc = (c/2+1);
for ( int i=0; i<r; ++i ){
// copy existing columns
for ( int j=0; j<outc; ++j ){
outfull[i*c+j][0] = out[i*outc+j][0];
outfull[i*c+j][1] = out[i*outc+j][1];
}
// generate missing column(s) from symmetry
for ( int j=outc; j<c; ++j){
int row = (r-i)%r;
int col = c-j;
outfull[i*c+j][0] = out[row*outc+col][0];
outfull[i*c+j][1] = -out[row*outc+col][1];
}
}
I've prototyped an algorithm for my iOS game in Python, and I need to rewrite in in ObjC. Basically, I have a board of 16 numbers, and I want to loop through every number three times and the four functions I'm using (add, subtract, multiply, exponentiate). 1+2+3, 2*3-4, 3^4-5, 9-4^3, etc., but without order of operations (first operation is always done first).
What I would like is an overview of how this might be implemented in Objective-C. Specifically, what is the equivalent of an array of functions in Objective-C? Is there an easy way to implement it with selectors? What's the best structure to use for loops with numbers? Array of NSIntegers, array of ints, NSArray/NSMutableArray of NSNumbers?
import random as rand
min = 0
max = 9
max_target = 20
maximum_to_calculate = 100
def multiply(x, y):
return x * y
def exponate(x, y):
return x ** y
def add(x, y):
return x + y
def subtract(x, y):
return x - y
function_array = [multiply, exponate, add, subtract]
board = [rand.randint(min, max) for i in xrange(0, 16)]
dict_of_frequencies = {}
for a in board:
for b in board:
for first_fun in function_array:
first_result = first_fun(a, b)
for c in board:
for second_fun in function_array:
final_result = second_fun(first_result, c)
if final_result not in dict_of_frequencies:
dict_of_frequencies[final_result] = 0
dict_of_frequencies[final_result] += 1
The most convenient way in Objective-C to construct an array of functions would be to use Blocks:
typedef NSInteger (^ArithmeticBlock)(NSInteger, NSInteger);
ArithmeticBlock add = ^NSInteger (NSInteger x, NSInteger y){
return x + y;
};
ArithmeticBlock sub = ^NSInteger (NSInteger x, NSInteger y){
return x - y;
};
NSArray * operations = #[add, sub];
Since there's no great way to perform arithmetic on NSNumbers, it would probably be best to create and store the board's values as primitives, such as NSIntegers, in a plain C array. You can box them up later easily enough, if necessary -- #(boardValue) gives you an NSNumber.
If you want to do it with straight C function pointers, something like this will do it:
#include <stdio.h>
#include <math.h>
long add(int a, int b) {
return a + b;
}
long subtract(int a, int b) {
return a - b;
}
long multiply(int a, int b) {
return a * b;
}
long exponate(int a, int b) {
return pow(a, b);
}
int main(void) {
long (*mfunc[4])(int, int) = {add, subtract, multiply, exponate};
char ops[4] = {'+', '-', '*', '^'};
for ( int i = 0; i < 4; ++i ) {
printf("5 %c 9 = %ld\n", ops[i], mfunc[i](5, 9));
}
return 0;
}
and gives the output:
paul#MacBook:~/Documents/src$ ./rndfnc
5 + 9 = 14
5 - 9 = -4
5 * 9 = 45
5 ^ 9 = 1953125
paul#MacBook:~/Documents/src$
Function pointer syntax can be slightly convoluted. long (*mfunc[4])(int, int) basically translates to defining a four-element array, called mfunc, of pointers to functions returning long and taking two arguments of type int.
Maddy is right. Anyway, I'll give it a try just for the fun of it.
This has never seen a compiler. So please forgive me all the typos and minor syntax errors in advance.
#include <stdlib.h>
...
const int MIN = 0;
const int MAX = 9;
const int MAX_TARGET = 20;
const int MAX_TO_CALCULATE = 100;
...
- (int) multiply:(int)x with:(int)y { return x * y; }
- (int) exponate:(int)x with:(int)y { return x ^ y; }
- (int) add:(int)x to:(int)y { return x + y; }
- (int) substract:(int)x by:(int)y { return x - y; }
// some method should start here, probably with
-(void) someMethod {
NSArray *functionArray = [NSArray arrayWithObjects: #selector(multiply::), #selector(exponate::), #selector(add::), #substract(multiply::), nil]; // there are other ways of generating an array of objects
NSMutableArray *board = [NSMutableArray arrayWithCapacity:16]; //Again, there are other ways available.
for (int i = 0; i < 16; i++) {
[board addObject:#(arc4random() % (MAX-MIN) + MIN)];
}
NSMutableDictionary dictOfFrequencies = [[NSMutableDictionary alloc] init];
for (NSNumber a in board)
for (NSNumber b in board)
for (SEL firstFun in functionArray) {
NSNumber firstResult = #([self performSelector:firstFun withObject:a withObject:b]);
NSNumber countedResults = [dictOfFrequencies objectForKey:firstResult];
if (countedResults) {
[dictOfFrequencies removeObjectForKey:firstResult];
countedResults = #(1 + [countedResults intValue]);
} else {
countedResults = #1; // BTW, using the # followed by a numeric expression creates an NSNumber object with the value 1.
}
[dictOfFrequencies setObject:countedResults forKey:firstResult];
}
}
Well, let me add some comments before others do. :-)
There is no need for objective c. You python code is iterative therefore you can implement it in plain C. Plain C is available where ever Objective C is.
If you really want to go for Objective-C here then you should forget your python code and implement the same logic (aiming for the same result) in Objective-C in an OOP style. My code really tries to translate your code as close as possible. Therefore my code is far far away from neither beeing good style nor maintainable nor proper OOP. Just keep that in mind before you think, ObjC was complicated compared to python :-)