scikitlearn remove less frequent categorical classes - machine-learning

Question

I'm doing a classification task where the number of distinct classes are 1500. From these, I would like to remove those classes (and corresponding record) whose frequency is less than 10.
I can write a function something like this:
code_freq_hash = {}
for code in y:
code_freq_hash.setdefault(code, 0)
code_freq_hash[code] += 1
to get the frequency for each class and then remove corresponding records.
However, I'm wondering whether there is an inbuilt function to do this in scikit learn or keras

Answer 1

Here's a sample solution using numpy and pandas.
Creating a data set with two features and a one class column
data = np.hstack((np.array(np.random.randn(20,2)), np.random.choice(np.arange(20), (20,1))))
Numpy
val, count = np.unique(data[:,-1], return_counts=True)
val[count>2]
out = data[np.isin(data[:, -1], val[np.isin(val, val[count>2])])] # replace 2 with 10 for your problem
Pandas
Converting the dataset (numpy array) to a pandas dataframe
df = pd.DataFrame(data)
# renamming the last column to the name "class"
df.rename(columns={ df.columns[-1]: "class" }, inplace=True)
0 1 class
0 0.542154 -0.434981 3.0
1 1.513857 -0.606722 17.0
2 0.372834 -0.120914 0.0
3 -1.357369 1.575805 5.0
4 0.547217 0.719883 4.0
5 0.818016 -0.243919 9.0
6 -0.400552 0.066519 19.0
7 0.463596 1.020041 6.0
8 0.850465 -0.814260 14.0
9 1.693060 0.186741 17.0
10 -0.287775 -0.190247 3.0
11 -0.390932 -0.418964 6.0
12 0.209542 0.797151 5.0
13 0.126585 -0.345196 5.0
14 -0.151729 -1.260708 4.0
15 -1.042408 1.050194 6.0
16 -0.221668 1.763742 5.0
17 -0.045617 1.159383 5.0
18 1.452508 -0.785115 5.0
19 2.125601 1.745009 2.0
Counting the occurrences and filter only the classes that occur more than twice (set 2 to 10 in your case)
d = df.loc[df['class'].isin(df['class'].value_counts().index[df['class'].value_counts() > 2])]
You can obtain the numpy array as d.values
array([[-1.35736852, 1.57580524, 5. ],
[ 0.46359614, 1.02004142, 6. ],
[-0.39093188, -0.41896435, 6. ],
[ 0.20954221, 0.79715056, 5. ],
[ 0.12658469, -0.34519613, 5. ],
[-1.04240815, 1.05019427, 6. ],
[-0.2216682 , 1.76374209, 5. ],
[-0.0456175 , 1.15938322, 5. ],
[ 1.45250806, -0.78511526, 5. ]])

Answer 2

There is no direct solution for this in Sklearn but as you mentioned it could be achieved by custom functions.
import pandas as pd
import numpy as np
df = pd.DataFrame({'labels': np.random.randint(0,10,size=50000),
'input': np.random.choice(['sample text 1','sample text 1'],size=50000)})
threshold = 5000
labels_df=df.labels.value_counts()
filtered_labels = labels_df[labels_df>threshold].index
new_df = df.loc[df['labels'].isin(filtered_labels),:]
new_df.shape
#(25290, 2)

Answer 3

One solution could be the following code snippet:
import numpy as np
unique, appearances = np.unique(a, return_counts=True)
code_freq_hash = [(unique[i], appearances[i]) for i in range(len(unique)) if appearances[i] >= 10]
Even more elegant, as stated below, relevant_labels = unique[appearances >= 10]