SageMaker deploy error "serve" executable file not found in $PATH - docker

Question

In Amazon SageMaker, I'm trying to deploy a custom created Docker container with a Scikit-Learn model, but deploying keeps giving errors.
These are my steps:
On my local machine created a script (script.py) and splitted training and test data. The script contains a main section, accepts parameters 'output-train-dir', 'model-dir', 'train' and 'test', and contains the functions model_fn, input_fn, output_fn and predict_fn
Tested the script locally, which worked
python script.py --train . --test . --model-dir .
Created a Docker image based on the default Python image (Python 3.9) and push to Amazon ECR, below are the commands I've used
> docker pull python
create Dockerfile, containing
FROM python:3.9
RUN pip3 install --no-cache scikit-learn numpy pandas joblib sagemaker-training
> docker build -t mymodel .
> aws ecr create-repository --repository-name mymodel
> docker tag 123456789012 123456789123.dkr.ecr.eu-central-1.amazonaws.com/mymodel
> docker push 123456789123.dkr.ecr.eu-central-1.amazonaws.com/mymodel
Uploaded the training and test data to s3 (mybucket)
Trained the script with local modus
aws_sklearn = SKLearn(entry_point='script.py',
framework_version='0.23-1',
image_uri='123456789123.dkr.ecr.eu-central-1.amazonaws.com/mymodel',
instance_type='local',
role=role)
aws_sklearn.fit({'train': mybucket_train_path, 'test': mybucket_test_path, 'model-dir': mybucket_model_path})
which was successful
Next I trained on AWS
aws_sklearn = SKLearn(entry_point='script.py',
framework_version='0.23-1',
image_uri='123456789123.dkr.ecr.eu-central-1.amazonaws.com/mymodel',
instance_type='ml.m4.xlarge',
role=role)
aws_sklearn.fit({'train': mybucket_train_path, 'test': mybucket_test_path})
which also was successful (however, providing the model-dir paramater gave errors, so I omitted it)
deploying however gave an error:
aws_sklearn_predictor = aws_sklearn.deploy(instance_type='ml.t2.medium',
initial_instance_count=1)
Error message:
UnexpectedStatusException: Error hosting endpoint
mymodel-2021-01-24-12-52-02-790: Failed. Reason: The primary
container for production variant AllTraffic did not pass the ping
health check. Please check CloudWatch logs for this endpoint..
And Cloudwatch said:
AWS sagemaker exec: "serve": executable file not found in $PATH
I somewhere read that I should add RUN chmod +x /opt/program/serve to the Dockerfile, but in my local image, there is no serve file present, this is something that SageMaker creates, right ?
How or where should I add serve to the $PATH environment variable or grant execute rights to the serve script ?

Answer 1

The serve file isn't something SageMaker creates automatically; you have to have it be part of the Docker container. This is technically true for the Estimator job too (there should be a similar train file as well; however you overwrite this by manually specifying an entry_point).
This page should help explain what SageMaker is actually trying to run when you run training and batch_transform jobs. That page references this repo which you can use as an example.
In short, if you want to continue to use your custom docker container, you'll have to build in functionality for the serve command (See additional scripts in the repo for launching the gunicorn server which runs multiple instances of the Flask app) and add those files to your Dockerfile.
The RUN chmod +x /opt/program/serve command will also make more sense after you've added that serve command functionality.