How to do feature engineering of real time data? - machine-learning

Question

I have made a good linear regression model with following step:
Data Integration
Data normalization/scaling(data preprocessing & feature engineering)
Model Building(using linear regression with SGD using cross validation)
Testing
My question is if we use this model in production environment then how can we do feature engineering of real time data because this model is built with feature normalization and scaling so how real time data can be normalized and scaled to get a good prediction? We don't need explicit feature engineering for cross validation and testing step because this can be done in data preprocessing step before building a model. What about real time data feature engineering?

Answer 1

This data could lend itself quite nicely to Featuretools. It is an open-source automated feature engineering library that explicitly deals with time to make sure you don't introduce label leakage.
For your music data, you could create two entities: "users" and "artist_plays", and then apply featuretools.dfs (Deep Feature Synthesis) to generate features. Think of an entity as being the same as a table in a relational database. Deep Feature Synthesis creates a single-table feature matrix ready for modeling from multiple different tables complete with high-level statistical features. Here is a short post explaining how it works.
This example is using plain Python, but could be adapted for use with Spark or Dask
# Create entityset
import featuretools as ft
from sklearn.preprocessing import Imputer, StandardScaler
import pandas as pd
import pickle
def load_entityset(user_df, artist_plays_df):
es = ft.EntitySet("artist plays")
es.entity_from_dataframe("users", user_df, index="user_id")
es.entity_from_dataframe("artist_plays", artist_plays_df, index="artist_id")
es.add_relationship(ft.Relationship(es['users']['user_id'], es['artist_plays']['user_id']))
return es
user_df = pd.read_csv("training_user.csv")
artist_plays_df = pd.read_csv("training_artist_plays.csv")
es = load_entityset(user_df, artist_plays_df)
feature_matrix, feature_defs = ft.dfs(entityset=es,
target_entity='artist_plays',
ignore_variables={'artist_plays': ['play']})
# encode categoricals
encoded_fm, encoded_fl = ft.encode_features(feature_matrix, feature_defs)
# Impute/scale using SKLearn
imputer = Imputer()
scaler = StandardScaler()
imputed_fm = imputer.fit_transform(encoded_fm)
scaled_fm = scaler.fit_transform(imputed_fm)
# Now, save the encoded feature list, and the imputer/scaler to files to reuse in production
ft.save_features(encoded_fl, 'fl.p')
with open('imputer.p', 'wb') as f:
pickle.dump(imputer, f)
with open('scaler.p', 'wb') as f:
pickle.dump(scaler, f)
Then in production:
import featuretools as ft
import pickle
import pandas as pd
# load previous data
old_user_df = pd.read_csv("training_user.csv")
old_artist_plays_df = pd.read_csv("training_artist_plays.csv")
es_old = load_entityset(old_user_df, old_artist_plays_df)
# load new data
user_df = pd.read_csv("new_user.csv")
artist_plays_df = pd.read_csv("new_artist_plays.csv")
es_updated = load_entityset(user_df, artist_plays_df)
# merge both data sources
es = es_old.concat(es_updated)
# load back in encoded features
features = ft.load_features('fl.p', es)
fm = ft.calculate_feature_matrix(features,
entityset=es,
instance_ids=es_updated['artist_plays'].get_all_instances())
# impute and scale
with open('imputer.p', 'r') as f:
imputer = pickle.load(f)
imputed_fm = imputer.transform(fm)
with open('scaler.p', 'r') as f:
scaler = pickle.load(f)
scaled_fm = scaler.transform(imputed_fm)
We have a few demos using this workflow, check out this example to predict what a grocery shopper will buy in the future.
I have also used this workflow in a real time production environment to do prediction of delivery metrics for large scale software projects- check out this white paper we have published that goes through this method and the results in a live deployment in gory detail.