Scikit-learn pipeline : possibility of rows aggregation (pandas groupby)? - machine-learning

Question

I wish to run all preprocessing and model optimisation tasks in a single pipeline with the following steps :
onehot encoding
SVD dimension reduction
aggregation (pandas groupby)
Random Forest modelisation
my input variables are :
X_train with 349 rows, which will become 338 rows after step3 (aggregation)
y_train with 338 rows
I get the error "Found input variables with inconsistent numbers of samples."
It's because sklearn doesn't allow a difference of rows number between X_train and y_train.
Do you know another method, if possible, to have an aggregation in a pipeline ?
here is my code :
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder
from sklearn.decomposition import TruncatedSVD
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.compose import ColumnTransformer
# does nothing, but is here to collect numerical columns
class nothing(BaseEstimator, TransformerMixin):
def fit(self, X, y=None):
return self
def transform(self, X):
return X
class Aggregator(BaseEstimator, TransformerMixin):
def fit(self, X, y=None):
return self
def transform(self, X):
X = pd.DataFrame(X)
X = X.rename(columns = {0 :'InvoiceNo', 1 : 'amount', 2:'Quantity',
3:'UnitPrice',4:'CustomerID' })
X['InvoiceNo'] = X['InvoiceNo'].astype('int')
X['Quantity'] = X['Quantity'].astype('float64')
X['UnitPrice'] = X['UnitPrice'].astype('float64')
aggregations = dict()
for col in range(5, X.shape[1]-1) :
aggregations[col] = 'max'
aggregations.update({ 'CustomerID' : 'first',
'amount' : "sum",'Quantity' : 'mean', 'UnitPrice' : 'mean'})
# aggregating all basket lines
result = X.groupby('InvoiceNo').agg(aggregations)
# add number of lines in the basket
result['lines_nb'] = X.groupby('InvoiceNo').size()
return result
numeric_features = ['InvoiceNo','amount', 'Quantity', 'UnitPrice',
'CustomerID']
numeric_transformer = Pipeline(steps=[('nothing', nothing())])
categorical_features = ['StockCode', 'Country']
preprocessor = ColumnTransformer(
[
# 'num' transformer does nothing, but is here to
# collect numerical columns
('num', numeric_transformer ,numeric_features ),
('cat', Pipeline([
('onehot', OneHotEncoder(handle_unknown='ignore')),
('best', TruncatedSVD(n_components=100)),
]), categorical_features)
]
)
pipe = Pipeline(steps=[
('preprocessor', preprocessor),
('aggregator', Aggregator()),
('rf', RandomForestClassifier(n_estimators=400,
max_features='auto',
class_weight=class_weights)),
])
X_train_transformed = pipe.fit_transform(X_train)
ValueError: Found input variables with inconsistent numbers of samples: [349, 338]
more detail to answer to #desertnaut comment :
example :
X_train contains 4 rows :
customer_num : 1 article_ref : 1 money : 10$
customer_num : 1 article_ref : 2 money : 15$
customer_num : 2 article_ref : 5 money : 5$
customer_num : 3 article_ref : 4 money : 11$
I aggregate the 4 rows with pandas groupby=cucstomer_num, the resulting dataframe , X_train_transformed , has 3 rows, one per customer
y_train has 3 rows, containing the group (label to predict) for customer_num 1, customer_num 2 et customer_num 3.
The standard method is :
pipeline 1 : transform X_train (4 rows) to X_train_transformed (3 rows)
pipeline 2 : fit a model to (X_train_transformed (3 rows), y_train(3 rows))
I whish to have a single pipeline to handle pipeline 1 and pipeline 2