Input Features ARIMA model - time-series

I am learning about the ARIMA model. My training set consists of 1) a date, 2) about 20 input features for each date, 3) output variable. Do ARIMA models take in as input multiple input features and then predict one of the features? Or do they only operate on a single variable?

ARIMA models are time series models, so they do not allow exogenous variables. There are various extensions of ARIMA models that do include exogenous variables including ARIMAX models, transfer function models, dynamic regression models, etc.

Related

Log-Likelihood for Random Forest models

I'm trying to compare multiple species distribution modeling approaches via k-fold cross-validation. Currently I'm calculating the RSME and AUC to compare model-performance. A friend suggested to further use the sum of log-likelihoods as metric to compare models. However, one of the models is a random forest fitted with the ranger package. If actually possible how would I calculate the log-likelihood for a random forest model and would it actually be a comparable metric to use with other models (GAM, GLM).
Thanks for your help.

Sagemaker: Training everytime I need make a prediction: how should I structure the solution?

I have been asked to migrate a custom model to Sagemaker. This model is a forecasting script that trains everytime it is run and then predicts after training. (It is a two-layer forecasting prediction with SARIMAX). The flow is as explained below:
train arima model to get exogenous variables (training algorithm 1)
predict with that trained model
use the output variables to train the second layer (training algorithm 2)
predict with this last trained model and output the solution
This is not what im used to do in Sagemaker (I train a model once that will be invoked multiple times), so how could I frame this? Train models separately from two separate docker images and create two endpoints? The whole train-predict-train-predict workflow would no longer be automatic, right? How would I trigger this workflow? Please help!

What's the correct approach to evaluate a model using K-fold cross validation over multiple seeds?

I am training a deep learning model using a 5-fold CV over three random seeds (random seeds are for model initialization, CV is split once). For each fold, I save the best model. Hence, I get 15 models after the simulation. To assess the performance, I take the best of these 15 models (unchanged during the entire evaluation process) and evaluate it using the validation fold of all the 5-folds for each seed. I then average the results across these seeds.
I would like to know if I am doing the right thing here.
I have read that there are two ways to compute CV performance: [1] pooling, where the performance is calculated globally over the union of all the test sets [2] averaging, where the performance is computed for every test set separately, with results being the average of these.
I intend to use method two (averaging).
Yes, you can use the averaging method for the 5-fold CV, but I don't understand what you mean by "For each fold, I save the best model". Moreover, three random seed values are not enough. You should use at least 10 different values and plot a boxplot for the corresponding results across these seeds.

Machine learning: training model from test data

I was wondering if a model trains itself from the test data as well while evaluating it multiple times, leading to a over-fitting scenario. Normally we split the training data into train-test splits and I noticed some people split it into 3 sets of data - train, test and eval. eval is for final evaluation of the model. I might be wrong but my point is that if the above mentioned scenario is not true, then there is no need for an eval data set.
Need some clarification.
The best way to evaluate how well a model will perform in the 'wild' is to evaluate its performance on a data set it has not seen (i.e., been trained on) -- assuming you have the labels in a supervised learning problem.
People split their data into train/test/eval and use the training data to estimate/learn the model parameters and the test set to tune the model (e.g., by trying different hyperparameter combinations). A model is usually selected based on the hyperparameter combination that optimizes a test metric (regression - MSE, R^2, etc.; classification - AUC, accuracy, etc.). Then the selected model is usually retrained on the combined train + test data set. After retraining, the model is evaluated based on its performance on the eval data set (assuming you have some ground truth labels to evaluate your predictions). The eval metric is what you report as the generalization metric -- that is, how well your model performs on novel data.
Does this help?
Consider you have train and test datasets. Train dataset is the one in which you know the output and you train your model on train dataset and you try to predict the output of Test dataset.
Most people split train dataset into train and validation. So first you run your model on train data and evaluate it on validation set. Then again you run the model on test dataset.
Now you are wondering how this will help and of any use?
This helps you to understand your model performance on seen data(validation data) and unseen data(your test data).
Here comes bias-variance trade-off into picture.
https://machinelearningmastery.com/gentle-introduction-to-the-bias-variance-trade-off-in-machine-learning/
Let's consider a binary classification example where a student's previous semester grades, Sports achievements, Extracurriculars etc are used to predict whether or not he will pass the final semester.
Let's say we have around 10000 samples (data of 10000 students).
Now we split them:
Training set - 6000 samples
Validation set - 2000 samples
Test set - 1000 samples
The training data is generally split into three (training set, validation set, and test set) for the following reasons:
1) Feature Selection: Let's assume you have trained the model using some algorithm. You calculate the training accuracy and validation accuracy. You plot the learning curves and find if the model is overfitting or underfitting and make changes (add or remove features, add more samples etc). Repeat until you have the best validation accuracy. Now test the model with the test set to get your final score.
2) Parameter Selection: When you use algorithms like KNN, And you need to find the best K value which fits the model properly. You can plot the accuracy of different K value and choose the best validation accuracy and use it for your test set. (same applies when you find n_estimators for Random forests etc)
3) Model Selection: Also you can train the model with different algorithms and choose the model which better fits the data by testing out the accuracy using validation set.
So basically the Validation set helps you evaluate your model's performance how you must fine-tune it for best accuracy.
Hope you find this helpful.

k-fold cross validation model selection method

I want to know about how we select model from k-fold cross validation method. In k-fold cross validation we can get k models and an accuracy score using the average of k models' accuracies. Can you please provide a method to get the final best model from cross validation?
K-fold cross validation is for comparing the performance of two models not for building models. Say, we desingned two 2 seq2seq generative models with different structures and our dataset is small, and we want to choose one model. We can follow the k-fold cross-validation method and get an average score for each model and choose the superior one with the higher score.
We don't need to choose a model from the k models, but we can ensemble the k modles into one by utilizing bagging(one of the three Ensemble methods). For more information please refer to this blog: Bagging and Random Forest Ensemble Algorithms for Machine Learning.
Reference:
1. https://stats.stackexchange.com/a/52277/103153
2. https://stats.stackexchange.com/a/19053/103153

Resources