does onnx model support well defined output format? I have seen that many model eg PMML provides input and output fields and data types in the model itself. So the ONNX model input/output are also well defined and can it be deduced from metadata information.
Yes.
When representing models using the ONNX format, the neural network is stored according to a predefined protobuf format.
This contains fields like Graph with all nodes sorted in topological order but also Input and Output, which contains information about the inputs and outputs of the model.
Input and Output are both ValueInfoProto, which holds information about the name of the input/output, and also the datatype. They have an enum with all sorts of datatypes, hence you can easily determine the datatype of the inputs and outputs.
If you want more information (I'm not sure what you mean by format) it of course depends: shapes might differ when using dynamic dimensions (for batch sizes of timesteps for examples). However, if you export to onnx with fixed dimension sizes you can also find this in ValueInfoProto.
This information can all be found in: https://github.com/onnx/onnx/blob/main/onnx/onnx.proto3
Related
For a multi class classification, I used label encoder to convert the categorical target variable to a numerical variable. got some good accuracy and saved the model to my local drive using joblib module.
So, in the future, if I load the model from my drive and try to predict using model.predict(['laptop not charging']) I will get numerical value as output because I have trained the model using numerical data only.
I know it won't work but mentioning here for reference that even this also won't work
le=LabelEncoder()
print(le.inverse_transform(model.predict(['laptop not charging'])))
I want the output as asset issue.
How to get asset issue as a output?
I am quite new to nvidia-tlt. Currently, I have trained, pruned and retrained the model with the kitti dataset, also am able to do these steps on any datasets with the required kitti format. What I want to do is used a previously trained model on kitti and fine tune it to my own data. The config file have the options pretrained_model_path, resume_model_path and pruned_model_path, So there is no option for the fine-tune in config. If I try to use pretrained_model_path, it throws an exception for the shape.
Invalid argument: Incompatible shapes: [6,29484,3] vs. [6,29484,12]
That error is expected.
Technically the pretrained model that we download from ngc comes without final layer which represents the total number of classes and their respective bboxes.
Once you train that model with any dataset, then the trained model will be frozen with the top layer. Now, if you want to finetune the same model with different number of classes you will get error related to invalid shapes.
You need to train the model on the new dataset from the beginning.
If you want to finetune the model with different dataset but of the same classes then you can use the previously trained model.
Which dimension of the input should be used for batching in torch?
I have 1000 examples for training and each training example is of the dimension 10*5. Now, i want to feed this data into a Sequencer as a batch of 100 examples each.
How should i structure my input? Should the dimension of each batch of input be 100*10*5 (first dimension used for batch) or 10*100*5 (second dimension used for batch) ?
Would appreciate links to relevant documents explaining the followed convention.
Does the convention change for containers and modules?
It is usually a Tensor of size 100*10*5. If it is an image it could be that you have to consider the number of channels so it would be batchSize*channels*width*height. This will make the data easy to access, you just need to do inputs[{i}] to retrieve your data. Consider to create another Tensor to store the labels (if you use labelled data). You can find the example here. https://github.com/torch/tutorials/blob/master/2_supervised/4_train.lua#L131
I'd recommend you to have a look at the tutorials, there you will see how the data has to be "prepared" before feeding the network https://github.com/torch/tutorials
The LibSVM on WEKA isn't loading with my dataset.
I am using WEKA and LibSVM. Every time I open my dataset and then try to chose an algorithm, the LibSVM algorithm isn't enabled (the option is gray). But if for instance I load the weather.arff example dataset that comes with the WEKA then the LibSVM algorithm works...
I don't know if there is anything wrong with my dataset. Are there any limitations that I should be aware of when dealing with LibSVM? For instance, number of attributes, etc.
The strange thing is that when I run my dataset with the SMO algorith that comes with WEKA it works without any problem.
In my dataset I have 76 attributes and my class attribute has 100 possible values.
Am I doing anything wrong? Thanks, very appreciated.
Your dataset doesn't match input format required by LibSVM. The capabilities are as follows:
CAPABILITIES
Class -- Nominal class, Missing class values, Binary class
Attributes -- Empty nominal attributes, Nominal attributes, Unary attributes, Binary attributes, Date attributes, Numeric attributes
Additional
min # of instances: 1
So the class in your .arff file should be either nominal or binary (allowed to miss some values) and your attributes should be nominal, unary or binary (allowed to be empty).
I made a classifier to classify search queries into one of the following classes: {Artist, Actor, Politician, Athlete, Facility, Geo, Definition, QA}. I have two csv files: one for training the classifier (contains 300 queries) and one for testing the classifier (currently contains about 200 queries). When I use the trainingset and testset for training/evaluating the classifier with weka knowledgeflow, most classes reach a pretty good accuracy. Setup of Weka knowledge flow training/testing situation:
After training I saved the MultiLayer Perceptron classifier from the knowledgeflow into classifier.model, which I used in java code to classify queries.
When I deserialize this model in java code and use it to classify all the queries of the testing set CSV-file (using the distributionForInstance()-method on the deserialized classifier) in the knowledgeflow it classifies all 'Geo' queries as 'Facility' queries and all 'QA' queries as 'Definition' queries. This surprised me a bit, as the ClassifierPerformanceEvaluator showed me a confusion matrix in which 'Geo' and 'QA' queries scored really well and the testing-queries are the same (the same CSV file was used). All other query classifications using the distributionForInstance()-method seem to work normally and so show the behavior that could be expected looking at the confusion matrix in the knowledgeflow. Does anyone know what could be possible causes for the classification difference between distributionForInstance()-method in the java code and the knowledgeflow evaluation results?
One thing that I can think of is the following:
The testing-CSV-file contains among other attributes a lot of nominal value attributes in all-capital casing. When I print out the values of all attributes of the instances before classification in the java code these values seem to be converted to lower capital letters (it seems like the DataSource.getDataSet() method behaves like this). Could it be that the casing of these attributes is the cause that some instances of my testing-CSV-file get classified differently? I read in Weka specification that nominal value attributes are case sensitive. I change these values to uppercase in the java file though, as weka then throws an exception that these values are not predefined for the nominal attribute.
Weka is likely using the same class in the knowledge flow as in your weka code to interpret the csv. This is why it works (produces data sets -- Instances objects -- that match) without tweaking and fails when you change things: the items don't match any more. This is to say that weka is handling the case of the input strings consistently, and does not require you to change it.
Check that you are looking at the Error on Test Data value and not the Error on Training Data value in the knowledge flow output, because the second one will be artificially high given that you built the model using those exact examples. It is possible that your classifier is performing the same in both places, but you are looking at different statistics.