Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 1 year ago.
Improve this question
I have a basic question about heavy-tailed distributions.
Suppose there are 50,000 cities in Spain and the population of each is denoted by p(1), p(2), …, p(n). Based on the mean of the distribution 𝜇 and the deviation 𝜎, how can we tell if the distribution is heavy-tailed or not? What procedure should we consider?
If you have all 50,000 observations then you can calculate the central moments about the mean.
In particular, the fourth central moment divided by the variance squared is the kurtosis. This number will tell you if the distribution is platykurtic or not. If it is greater than three, it means that your distribution has heavier tails than a standard normal distribution.
So if you are working in Python and all 50K observations are stored in x:
from scipy import stats
# Calculate kurtosis
k = stats.moment(x, 4) / x.var()**2
# Evaluate
if k > 3:
print('Distribution has heavy tails')
else:
print('Distribution does not have heavy tails')
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 1 year ago.
Improve this question
I look for examples about encoding high cardinal or sparse datasets using Neural Networks but I cannot find it. Also I search about embedding numerical variables (not categorical) however I couldn't find any examples either. Can you send me a GitHub link etc. if you have about these issues?
Working with neural networks I am assuming that tensorflow with Keras backend is being used?
If so here is a reference snippet, main library used tf.feature_column
import tensorflow as tf
from tensorflow.keras import layers
feature_columns=[]
for col in list(df_train_numerical.columns):
col = tf.feature_column.numeric_column(col)
feature_columns.append(col)
for col in list(df_train_categorical.columns):
col = tf.feature_column.embedding_column(tf.feature_column.categorical_column_with_hash_bucket(col, hash_bucket_size=8000), dimension=8)
#above hash bucket size is specified (cardinality) with dimension
feature_columns.append(col)
feature_layer = layers.DenseFeatures(feature_columns)
Following that the feature_layer is basically the first layer of the neural network-
model = tf.keras.models.Sequential()
model.add(feature_layer)
reference git code
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 1 year ago.
Improve this question
For example i want to develop a Deep Learning model for classification of images and I have thousands of images . Since training the model with the whole dataset takes a long amount of time i would like to take a sample (10%) of the original dataset for initial training . How to do this?
If the dataset is contained into a folder, I will try in the following:
import os
import numpy as np
images = os.listdir('Path to your dataset') # list of all the images
n_test_images = int(len(images) * 0.1) # 10% of the total images
subset_images = np.random.choice(images, size=n_test_images, replace=False)
I used replace=True to avoid picking twice the same element.
After I selected the 10% of the images, I load them.
Actually I am not sure if this way is the most optimal one, but it could be a good starting point.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 2 years ago.
Improve this question
What is the partial derivative for MAE? I understand that for mean squared error (MSE) the partial derivative with respect to some x1 would be -x1 * (y_pred - y_actual) assuming the the following version of MSE is used.
What is the partial derivative for x1 when the loss function is MAE instead of MSE? I've been trying to find this but I haven't had any luck. Would it just be -(y_pred - y_actual) when x1 is greater than 0, and (y_pred - y_actual) when x1 is less than 0? Or is there something else that I'm missing?
Unless you're having a single neuron, there's no fixed formula for partial derivative of loss function with respect to each weight; it depends strictly on the connections between neurons in the network. And the partial derivative formula is not only one, each weight has a different one.
For small network with kinda 2, 3 layers, apply chain rule, and sum rule to find the partial derivative of loss function manually, otherwise, dynamic programming in backpropagation is needed.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 9 years ago.
Improve this question
I am working on a past exam paper. I am given a data set as follows:
Hair {brown, red} = {B,R},
Height {tall, short} = {T,S} and
Country {UK, Italy} = {U,I}
(B,T,U) (B,T,U) (B,T,I)
(R,T,U) (R,T,U) (B,T,I)
(R,T,U) {R,T,U) (B,T,I)
(R,S,U) (R,S,U) (R,S,I)
Question: Estimate the probabilities P(B,T|U), P(B|U), P(T|U), P(U) and P(I)
As the question states estimate, I am guessing that I don't need to calculate any values. Is it just a case of adding up how many times P(B,T|U) occurs over the whole data set e.g. (2/12) = 16%.
Then would the probability of P(U) be 0?
I don't think so. Out of your 12 records, 8 are from the country UK. So P(U) should be 8/12 = 2/3 ~= .66
Bayes' theorem is P(A|B) = P(B|A)P(A)/P(B) , which you're going to need to estimate some of those probabilities.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
Anybody please help me to interpret the following result generated in weka for classification using naive bayes.
Please explain clearly what is
Normal Distribution
Mean
StandardDev
WeightSum
Precision.
Please help me. I am new in weka.
**
Naive Bayes Classifier
Class Normal: Prior probability = 0.5
1374195_at: Normal Distribution. Mean = 218.06 StandardDev = 6.0572 WeightSum = 3 Precision = 36.34333334
1373315_at: Normal Distribution. Mean = 1142.58 StandardDev = 21.1589 WeightSum = 3 Precision = 126.95333339999999
Normal distribution is the classic gaussian distribution. Mean and Standard deviation are properties of a normal/gaussian distribution. Look to basic statistics texts about this.
Weight Sum. This value is calculated for numerical values. Its value is equal to class distribution. For iris dataset there are 3 classes (50,50,50) and this value is 50 for all of them. For weather dataset it is 9 5. Same as class instance number. Your attribute value affects your result according to class distribution.
Precision : TP / (TP + FP) The percentage of positive predictions that are correct.
More resources :
Classifier Evaluation