About Automatic playing YouTube video into clubhouse - youtube

I am trying to write some codes for automatically playing one online radio into clubhouse Android application, so if possible I like to have some comment for:
Finding some python or php codes to play automatically live-streaming from YouTube by
using the Google Colaboratory environment, like the result of the below Google search results:
Automate clubhouse Messages With Python
Update 1:
I have found two result from the Google, which could be seen below:
A third-part web application based on flask to play Clubhouse
audio
A simple script that will watch a stream for you and earn the
channel points.
And tried to run the above first instruction on the Google colab (link) but get the below error:
!git clone https://github.com/ai-eks/OpenClubhouse-Worker
%cd OpenClubhouse-Worker
!pip install -r requirements.txt
Cloning into 'OpenClubhouse-Worker'...
remote: Enumerating objects: 61, done.
remote: Counting objects: 100% (61/61), done.
remote: Compressing objects: 100% (47/47), done.
remote: Total 61 (delta 31), reused 38 (delta 14), pack-reused 0
Unpacking objects: 100% (61/61), done.
/content/OpenClubhouse-Worker
Collecting pymongo==3.11.3
Downloading pymongo-3.11.3-cp37-cp37m-manylinux2014_x86_64.whl (512 kB)
|████████████████████████████████| 512 kB 5.2 MB/s
Collecting requests==2.25.1
Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
|████████████████████████████████| 61 kB 6.8 MB/s
Collecting requests-openapi==0.9.7
Downloading requests_openapi-0.9.7-py3-none-any.whl (5.1 kB)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests==2.25.1->-r requirements.txt (line 2)) (1.24.3)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests==2.25.1->-r requirements.txt (line 2)) (2.10)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests==2.25.1->-r requirements.txt (line 2)) (2021.5.30)
Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests==2.25.1->-r requirements.txt (line 2)) (3.0.4)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from requests-openapi==0.9.7->-r requirements.txt (line 3)) (3.13)
Installing collected packages: requests, requests-openapi, pymongo
Attempting uninstall: requests
Found existing installation: requests 2.23.0
Uninstalling requests-2.23.0:
Successfully uninstalled requests-2.23.0
Attempting uninstall: pymongo
Found existing installation: pymongo 3.12.0
Uninstalling pymongo-3.12.0:
Successfully uninstalled pymongo-3.12.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
google-colab 1.0.0 requires requests~=2.23.0, but you have requests 2.25.1 which is incompatible.
datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.
Successfully installed pymongo-3.11.3 requests-2.25.1 requests-openapi-0.9.7
So I am trying to make it by updating the question and working based of the comments.
Update 2:
Based on the help of Marco the first error fixed, and I have got a new error, as you can see below:
Traceback (most recent call last):
File "main.py", line 38, in <module>
main()
File "main.py", line 26, in main
chh = ClubHouseHelper(phone=phone, url=api_uri, device_id=device_id)
File "/content/OpenClubhouse-Worker/OpenClubhouse-Worker/OpenClubhouse-Worker/OpenClubhouse-Worker/ch_helper.py", line 9, in __init__
self.client.load_spec_from_file(url)
File "/usr/local/lib/python3.7/dist-packages/requests_openapi/core.py", line 252, in load_spec_from_file
spec = load_spec_from_file(file_path)
File "/usr/local/lib/python3.7/dist-packages/requests_openapi/core.py", line 159, in load_spec_from_file
return yaml.load(spec_str, Loader=yaml.Loader)
File "/usr/local/lib/python3.7/dist-packages/yaml/__init__.py", line 70, in load
loader = Loader(stream)
File "/usr/local/lib/python3.7/dist-packages/yaml/loader.py", line 34, in __init__
Reader.__init__(self, stream)
File "/usr/local/lib/python3.7/dist-packages/yaml/reader.py", line 74, in __init__
self.check_printable(stream)
File "/usr/local/lib/python3.7/dist-packages/yaml/reader.py", line 144, in check_printable
'unicode', "special characters are not allowed")
yaml.reader.ReaderError: unacceptable character #x1f579: special characters are not allowed
in "<unicode string>", position 6373
Thanks.

Related

Im having problems while running alphacode on ubuntu

I am using Docker Ubuntu.
I have installed the full dataset(dm-code_contests) to /tmp folder and cloned the git repository on /home folder(the repository is code_contests). When I try to run bazel run -c opt \ :print_names_and_sources /tmp/dm-code_contests/code_contests_valid.riegeli(in /home/code_contests folder), it shows error:
Starting local Bazel server and connecting to it...
INFO: Repository local_config_python instantiated at:
/home/code_contests/WORKSPACE:12:10: in <toplevel>
/root/.cache/bazel/_bazel_root/24a36d3f089e715b642fd688d4461183/external/com_github_grpc_grpc/bazel/grpc_deps.bzl:414:21: in grpc_deps
/root/.cache/bazel/_bazel_root/24a36d3f089e715b642fd688d4461183/external/com_github_grpc_grpc/bazel/grpc_python_deps.bzl:43:21: in grpc_python_deps
Repository rule python_configure defined at:
/root/.cache/bazel/_bazel_root/24a36d3f089e715b642fd688d4461183/external/com_github_grpc_grpc/third_party/py/python_configure.bzl:365:35: in <toplevel>
ERROR: An error occurred during the fetch of repository 'local_config_python':
Traceback (most recent call last):
File "/root/.cache/bazel/_bazel_root/24a36d3f089e715b642fd688d4461183/external/com_github_grpc_grpc/third_party/py/python_configure.bzl", line 355, column 35, in _python_autoconf_impl
_create_single_version_package(
File "/root/.cache/bazel/_bazel_root/24a36d3f089e715b642fd688d4461183/external/com_github_grpc_grpc/third_party/py/python_configure.bzl", line 304, column 45, in _create_single_version_package
python_include = _get_python_include(repository_ctx, python_bin)
File "/root/.cache/bazel/_bazel_root/24a36d3f089e715b642fd688d4461183/external/com_github_grpc_grpc/third_party/py/python_configure.bzl", line 236, column 22, in _get_python_include
result = _execute(
File "/root/.cache/bazel/_bazel_root/24a36d3f089e715b642fd688d4461183/external/com_github_grpc_grpc/third_party/py/python_configure.bzl", line 62, column 14, in _execute
_fail("\n".join([
File "/root/.cache/bazel/_bazel_root/24a36d3f089e715b642fd688d4461183/external/com_github_grpc_grpc/third_party/py/python_configure.bzl", line 35, column 9, in _fail
fail("%sPython Configuration Error:%s %s\n" % (red, no_color, msg))
Error in fail: Python Configuration Error: Problem getting python include path for /usr/bin/python3.
<string>:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
<string>:1: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
Is the Python binary path set up right? (See ./configure or /usr/bin/python3.) Is distutils installed? Are Python headers installed? Try installing python-dev or python3-dev on Debian-based systems. Try python-devel or python3-devel on Redhat-based systems.
ERROR: /home/code_contests/WORKSPACE:12:10: fetching python_configure rule //external:local_config_python: Traceback (most recent call last):
File "/root/.cache/bazel/_bazel_root/24a36d3f089e715b642fd688d4461183/external/com_github_grpc_grpc/third_party/py/python_configure.bzl", line 355, column 35, in _python_autoconf_impl
_create_single_version_package(
File "/root/.cache/bazel/_bazel_root/24a36d3f089e715b642fd688d4461183/external/com_github_grpc_grpc/third_party/py/python_configure.bzl", line 304, column 45, in _create_single_version_package
python_include = _get_python_include(repository_ctx, python_bin)
File "/root/.cache/bazel/_bazel_root/24a36d3f089e715b642fd688d4461183/external/com_github_grpc_grpc/third_party/py/python_configure.bzl", line 236, column 22, in _get_python_include
result = _execute(
File "/root/.cache/bazel/_bazel_root/24a36d3f089e715b642fd688d4461183/external/com_github_grpc_grpc/third_party/py/python_configure.bzl", line 62, column 14, in _execute
_fail("\n".join([
File "/root/.cache/bazel/_bazel_root/24a36d3f089e715b642fd688d4461183/external/com_github_grpc_grpc/third_party/py/python_configure.bzl", line 35, column 9, in _fail
fail("%sPython Configuration Error:%s %s\n" % (red, no_color, msg))
Error in fail: Python Configuration Error: Problem getting python include path for /usr/bin/python3.
<string>:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
<string>:1: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
Is the Python binary path set up right? (See ./configure or /usr/bin/python3.) Is distutils installed? Are Python headers installed? Try installing python-dev or python3-dev on Debian-based systems. Try python-devel or python3-devel on Redhat-based systems.
ERROR: /root/.cache/bazel/_bazel_root/24a36d3f089e715b642fd688d4461183/external/com_google_riegeli/python/riegeli/records/BUILD:8:13: #com_google_riegeli//python/riegeli/records:record_writer_cc depends on #local_config_python//:python_headers in repository #local_config_python which failed to fetch. no such package '#local_config_python//': Python Configuration Error: Problem getting python include path for /usr/bin/python3.
<string>:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
<string>:1: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
Is the Python binary path set up right? (See ./configure or /usr/bin/python3.) Is distutils installed? Are Python headers installed? Try installing python-dev or python3-dev on Debian-based systems. Try python-devel or python3-devel on Redhat-based systems.
ERROR: Analysis of target '//:print_names_and_sources' failed; build aborted:
INFO: Elapsed time: 3.701s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (49 packages loaded, 348 targets configured)
FAILED: Build did NOT complete successfully (49 packages loaded, 348 targets configured)
Fetching #com_google_absl; Cloning tags/20211102.0 of https://github.com/abseil/abseil-cpp.git
root#c89a94de94ce://home/code_contests# bazel run -c opt \ :print_names_and_sources /tmp/dm-code_contests/code_contests
_valid.riegeli
ERROR: Skipping ' :print_names_and_sources': no such package ' ': BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.
- /home/code_contests/
WARNING: Target pattern parsing failed.
ERROR: no such package ' ': BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.
- /home/code_contests/
INFO: Elapsed time: 0.174s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)
Im new to Ubuntu(as well as bazel). So how can I fix this error and run the project?
link to the source code: https://github.com/deepmind/code_contests
You should make sure your gcc is the latest version.
For python2 you should install it like this:
#Remove bazel and reinstall
bazel clean --expunge
rm -rf ~/.cache/bazel
To re-install follow this instruction
#Install python2 dependency
sudo apt update && sudo apt install python-dev
For a detailed explanation kindly refer to this document. Thank you!

SagemakerTraining job catboost-classification-model , ErrorMessage "TypeError: Cannot convert 'xxx'' to float

When I performed the following AWS tutorial, I got an error when training the model. https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/lightgbm_catboost_tabular/Amazon_Tabular_Classification_LightGBM_CatBoost.ipynb
The error that occurred is
UnexpectedStatusException: Error for Training job jumpstart-catboost-classification-model-2022-07-22-07-33-18-038: Failed. Reason: AlgorithmError: ExecuteUserScriptError:
ExitCode 1 ErrorMessage "TypeError: Cannot convert 'b'BROOKLYN'' to float
These are all the files that I have upload in S3 bucket : Amazon S3 --> Buckets---> R-sandbox-sagemaker--->ml/---> train/ and in the train folder 'data.csv' and 'categorical_index.json' are uploaded based on the mentioned tutorial. Could you give me some advice on how to solve it?
Also, here all the code and full traceback of the issue are provided below, I have used the same data and used the Catboost binary classifier in the local machine; everything works, including training and prediction, however using the Catboost classifier build-in algorithm in Sagemamker have some issues. I doable check the directory, data.csv and JSON file, and searched a lot about this error, but no success so far. Could you give me some advice on how to solve it?
!pip install sagemaker ipywidgets --upgrade –quiet
import sagemaker, boto3, json
from sagemaker import get_execution_role
aws_role = get_execution_role()
aws_region = boto3.Session().region_name
sess = sagemaker.Session()
##2.1 Retrieve Training Artifacts-
#retrieve the training docker container, the training algorithm source, and the tabular algorithm. Note that model_version="*" fetches the latest model.
# Currently, not all the object detection models in jumpstart support finetuning. Thus, we manually select a model
# which supports finetuning.
from sagemaker import image_uris, model_uris, script_uris
train_model_id, train_model_version, train_scope = "catboost-classification-model", "*", "training"
training_instance_type = "ml.m5.xlarge"
# Retrieve the docker image
train_image_uri = image_uris.retrieve(
region=None,
framework=None,
model_id=train_model_id,
model_version=train_model_version,
image_scope=train_scope,
instance_type=training_instance_type,
)
# Retrieve the training script
train_source_uri = script_uris.retrieve(
model_id=train_model_id, model_version=train_model_version, script_scope=train_scope
)
# Retrieve the pre-trained model tarball to further fine-tune
train_model_uri = model_uris.retrieve(
model_id=train_model_id, model_version=train_model_version, model_scope=train_scope
)
## 2.2 Set Training Parameters
# Sample training data is available in this bucket
training_data_bucket = "R-sandbox-sagemaker"
training_data_prefix = "ml"
training_dataset_s3_path = f"s3://{training_data_bucket}/{training_data_prefix}"
output_bucket = sess.default_bucket()
output_prefix = "jumpstart-example-tabular-training"
s3_output_location = f"s3://{output_bucket}/{output_prefix}/output"
from sagemaker import hyperparameters
# Retrieve the default hyper-parameters for fine-tuning the model
hyperparameters = hyperparameters.retrieve_default(
model_id=train_model_id, model_version=train_model_version
)
# [Optional] Override default hyperparameters with custom values
hyperparameters[
"iterations"
] = "500" # The same hyperparameter is named as "iterations" for CatBoost
print(hyperparameters)
## 2.3. Train with Automatic Model Tuning
from sagemaker.tuner import ContinuousParameter, IntegerParameter, HyperparameterTuner
use_amt = True
if train_model_id == "lightgbm-classification-model":
hyperparameter_ranges = {
"learning_rate": ContinuousParameter(1e-4, 1, scaling_type="Logarithmic"),
"num_boost_round": IntegerParameter(2, 30),
"early_stopping_rounds": IntegerParameter(2, 30),
"num_leaves": IntegerParameter(10, 50),
"feature_fraction": ContinuousParameter(0, 1),
"bagging_fraction": ContinuousParameter(0, 1),
"bagging_freq": IntegerParameter(1, 10),
"max_depth": IntegerParameter(5, 30),
"min_data_in_leaf": IntegerParameter(5, 50),
}
if train_model_id == "catboost-classification-model":
hyperparameter_ranges = {
"learning_rate": ContinuousParameter(0.00001, 0.1, scaling_type="Logarithmic"),
"iterations": IntegerParameter(50, 1000),
"early_stopping_rounds": IntegerParameter(1, 10),
"depth": IntegerParameter(1, 10),
"l2_leaf_reg": IntegerParameter(1, 10),
"random_strength": ContinuousParameter(0.01, 10, scaling_type="Logarithmic"),
}
## 2.4. Start Training
from sagemaker.estimator import Estimator
from sagemaker.utils import name_from_base
training_job_name = name_from_base(f"jumpstart-{'catboost-classification-model'}-training")
# Create SageMaker Estimator instance
tabular_estimator = Estimator(
role=aws_role,
image_uri=train_image_uri,
source_dir=train_source_uri,
model_uri=train_model_uri,
entry_point="transfer_learning.py",
instance_count=1,
instance_type=training_instance_type,
max_run=360000,
#hyperparameters=hyperparameters,
output_path=s3_output_location,
)
# Launch a SageMaker Training job by passing s3 path of the training data
tabular_estimator.fit(
{"training": training_dataset_s3_path}, logs=True, job_name=training_job_name
)
2022-07-22 07:33:18 Starting - Starting the training job...
2022-07-22 07:33:46 Starting - Preparing the instances for trainingProfilerReport-1658475198: InProgress
.........
2022-07-22 07:35:06 Downloading - Downloading input data...
2022-07-22 07:35:46 Training - Downloading the training image...
2022-07-22 07:36:11 Training - Training image download completed. Training in progress..bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
2022-07-22 07:36:14,025 sagemaker-training-toolkit INFO Imported framework sagemaker_pytorch_container.training
2022-07-22 07:36:14,027 sagemaker-training-toolkit INFO No GPUs detected (normal if no gpus installed)
2022-07-22 07:36:14,036 sagemaker_pytorch_container.training INFO Block until all host DNS lookups succeed.
2022-07-22 07:36:14,041 sagemaker_pytorch_container.training INFO Invoking user training script.
2022-07-22 07:36:15,901 sagemaker-training-toolkit INFO Installing dependencies from requirements.txt:
/opt/conda/bin/python3.8 -m pip install -r requirements.txt
Processing ./catboost/tenacity-8.0.1-py3-none-any.whl
Processing ./catboost/plotly-5.1.0-py2.py3-none-any.whl
Processing ./catboost/graphviz-0.17-py3-none-any.whl
Processing ./catboost/catboost-1.0.1-cp38-none-manylinux1_x86_64.whl
Processing ./sagemaker_jumpstart_script_utilities-1.0.0-py2.py3-none-any.whl
Requirement already satisfied: six in /opt/conda/lib/python3.8/site-packages (from plotly==5.1.0->-r requirements.txt (line 2)) (1.16.0)
Requirement already satisfied: numpy>=1.16.0 in /opt/conda/lib/python3.8/site-packages (from catboost==1.0.1->-r requirements.txt (line 4)) (1.19.1)
Requirement already satisfied: scipy in /opt/conda/lib/python3.8/site-packages (from catboost==1.0.1->-r requirements.txt (line 4)) (1.7.1)
Requirement already satisfied: matplotlib in /opt/conda/lib/python3.8/site-packages (from catboost==1.0.1->-r requirements.txt (line 4)) (3.4.3)
Requirement already satisfied: pandas>=0.24.0 in /opt/conda/lib/python3.8/site-packages (from catboost==1.0.1->-r requirements.txt (line 4)) (1.2.4)
Requirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/lib/python3.8/site-packages (from pandas>=0.24.0->catboost==1.0.1->-r requirements.txt (line 4)) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.8/site-packages (from pandas>=0.24.0->catboost==1.0.1->-r requirements.txt (line 4)) (2021.3)
Requirement already satisfied: pillow>=6.2.0 in /opt/conda/lib/python3.8/site-packages (from matplotlib->catboost==1.0.1->-r requirements.txt (line 4)) (8.3.2)
Requirement already satisfied: pyparsing>=2.2.1 in /opt/conda/lib/python3.8/site-packages (from matplotlib->catboost==1.0.1->-r requirements.txt (line 4)) (2.4.7)
Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.8/site-packages (from matplotlib->catboost==1.0.1->-r requirements.txt (line 4)) (0.10.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.8/site-packages (from matplotlib->catboost==1.0.1->-r requirements.txt (line 4)) (1.3.2)
tenacity is already installed with the same version as the provided wheel. Use --force-reinstall to force an installation of the wheel.
Installing collected packages: plotly, graphviz, sagemaker-jumpstart-script-utilities, catboost
Attempting uninstall: plotly
Found existing installation: plotly 5.3.1
Uninstalling plotly-5.3.1:
Successfully uninstalled plotly-5.3.1
Successfully installed catboost-1.0.1 graphviz-0.17 plotly-5.1.0 sagemaker-jumpstart-script-utilities-1.0.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
2022-07-22 07:36:32,568 sagemaker-training-toolkit INFO No GPUs detected (normal if no gpus installed)
2022-07-22 07:36:32,580 sagemaker-training-toolkit INFO No GPUs detected (normal if no gpus installed)
2022-07-22 07:36:32,594 sagemaker-training-toolkit INFO No GPUs detected (normal if no gpus installed)
2022-07-22 07:36:32,604 sagemaker-training-toolkit INFO Invoking user script
Training Env:
{
"additional_framework_parameters": {},
"channel_input_dirs": {
"model": "/opt/ml/input/data/model",
"training": "/opt/ml/input/data/training"
},
"current_host": "algo-1",
"framework_module": "sagemaker_pytorch_container.training:main",
"hosts": [
"algo-1"
],
"hyperparameters": {},
"input_config_dir": "/opt/ml/input/config",
"input_data_config": {
"model": {
"ContentType": "application/x-sagemaker-model",
"TrainingInputMode": "File",
"S3DistributionType": "FullyReplicated",
"RecordWrapperType": "None"
},
"training": {
"TrainingInputMode": "File",
"S3DistributionType": "FullyReplicated",
"RecordWrapperType": "None"
}
},
"input_dir": "/opt/ml/input",
"is_master": true,
"job_name": "jumpstart-catboost-classification-model-2022-07-22-07-33-18-038",
"log_level": 20,
"master_hostname": "algo-1",
"model_dir": "/opt/ml/model",
"module_dir": "s3://jumpstart-cache-prod-us-east-1/source-directory-tarballs/catboost/transfer_learning/classification/v1.1.3/sourcedir.tar.gz",
"module_name": "transfer_learning",
"network_interface_name": "eth0",
"num_cpus": 4,
"num_gpus": 0,
"output_data_dir": "/opt/ml/output/data",
"output_dir": "/opt/ml/output",
"output_intermediate_dir": "/opt/ml/output/intermediate",
"resource_config": {
"current_host": "algo-1",
"current_instance_type": "ml.m5.xlarge",
"current_group_name": "homogeneousCluster",
"hosts": [
"algo-1"
],
"instance_groups": [
{
"instance_group_name": "homogeneousCluster",
"instance_type": "ml.m5.xlarge",
"hosts": [
"algo-1"
]
}
],
"network_interface_name": "eth0"
},
"user_entry_point": "transfer_learning.py"
}
Environment variables:
SM_HOSTS=["algo-1"]
SM_NETWORK_INTERFACE_NAME=eth0
SM_HPS={}
SM_USER_ENTRY_POINT=transfer_learning.py
SM_FRAMEWORK_PARAMS={}
SM_RESOURCE_CONFIG={"current_group_name":"homogeneousCluster","current_host":"algo-1","current_instance_type":"ml.m5.xlarge","hosts":["algo-1"],"instance_groups":[{"hosts":["algo-1"],"instance_group_name":"homogeneousCluster","instance_type":"ml.m5.xlarge"}],"network_interface_name":"eth0"}
SM_INPUT_DATA_CONFIG={"model":{"ContentType":"application/x-sagemaker-model","RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"},"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}}
SM_OUTPUT_DATA_DIR=/opt/ml/output/data
SM_CHANNELS=["model","training"]
SM_CURRENT_HOST=algo-1
SM_MODULE_NAME=transfer_learning
SM_LOG_LEVEL=20
SM_FRAMEWORK_MODULE=sagemaker_pytorch_container.training:main
SM_INPUT_DIR=/opt/ml/input
SM_INPUT_CONFIG_DIR=/opt/ml/input/config
SM_OUTPUT_DIR=/opt/ml/output
SM_NUM_CPUS=4
SM_NUM_GPUS=0
SM_MODEL_DIR=/opt/ml/model
SM_MODULE_DIR=s3://jumpstart-cache-prod-us-east-1/source-directory-tarballs/catboost/transfer_learning/classification/v1.1.3/sourcedir.tar.gz
SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{"model":"/opt/ml/input/data/model","training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_pytorch_container.training:main","hosts":["algo-1"],"hyperparameters":{},"input_config_dir":"/opt/ml/input/config","input_data_config":{"model":{"ContentType":"application/x-sagemaker-model","RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"},"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"jumpstart-catboost-classification-model-2022-07-22-07-33-18-038","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://jumpstart-cache-prod-us-east-1/source-directory-tarballs/catboost/transfer_learning/classification/v1.1.3/sourcedir.tar.gz","module_name":"transfer_learning","network_interface_name":"eth0","num_cpus":4,"num_gpus":0,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_group_name":"homogeneousCluster","current_host":"algo-1","current_instance_type":"ml.m5.xlarge","hosts":["algo-1"],"instance_groups":[{"hosts":["algo-1"],"instance_group_name":"homogeneousCluster","instance_type":"ml.m5.xlarge"}],"network_interface_name":"eth0"},"user_entry_point":"transfer_learning.py"}
SM_USER_ARGS=[]
SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
SM_CHANNEL_MODEL=/opt/ml/input/data/model
SM_CHANNEL_TRAINING=/opt/ml/input/data/training
PYTHONPATH=/opt/ml/code:/opt/conda/bin:/opt/conda/lib/python38.zip:/opt/conda/lib/python3.8:/opt/conda/lib/python3.8/lib-dynload:/opt/conda/lib/python3.8/site-packages
Invoking script with the following command:
/opt/conda/bin/python3.8 transfer_learning.py
INFO:root:Validation data is not found. 20.0% of training data is randomly selected as validation data. The seed for random sampling is 200.
Traceback (most recent call last):
File "_catboost.pyx", line 2167, in _catboost.get_float_feature
File "_catboost.pyx", line 1125, in _catboost._FloatOrNan
File "_catboost.pyx", line 949, in _catboost._FloatOrNanFromString
TypeError: Cannot convert 'b'BROOKLYN'' to float
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "transfer_learning.py", line 221, in <module>
run_with_args(args)
File "transfer_learning.py", line 182, in run_with_args
cat_train, cat_eval = Pool(data=X_train, label=y_train, cat_features=cat_features), Pool(
File "/opt/conda/lib/python3.8/site-packages/catboost/core.py", line 628, in __init__
self._init(data, label, cat_features, text_features, embedding_features, pairs, weight, group_id, group_weight, subgroup_id, pairs_weight, baseline, timestamp, feature_names, thread_count)
File "/opt/conda/lib/python3.8/site-packages/catboost/core.py", line 1171, in _init
self._init_pool(data, label, cat_features, text_features, embedding_features, pairs, weight, group_id, group_weight, subgroup_id, pairs_weight, baseline, timestamp, feature_names, thread_count)
File "_catboost.pyx", line 3755, in _catboost._PoolBase._init_pool
File "_catboost.pyx", line 3803, in _catboost._PoolBase._init_pool
File "_catboost.pyx", line 3638, in _catboost._PoolBase._init_features_order_layout_pool
File "_catboost.pyx", line 2664, in _catboost._set_features_order_data_pd_data_frame
File "_catboost.pyx", line 2208, in _catboost.create_num_factor_data
File "_catboost.pyx", line 2169, in _catboost.get_float_feature
_catboost.CatBoostError: Bad value for num_feature[non_default_doc_idx=0,feature_idx=2]="BROOKLYN": Cannot convert 'b'BROOKLYN'' to float
2022-07-22 07:36:34,967 sagemaker-training-toolkit ERROR Reporting training FAILURE
2022-07-22 07:36:34,967 sagemaker-training-toolkit ERROR ExecuteUserScriptError:
ExitCode 1
ErrorMessage "TypeError: Cannot convert 'b'BROOKLYN'' to float
During handling of the above exception, another exception occurred: Traceback (most recent call last): File "transfer_learning.py", line 221, in <module> run_with_args(args) File "transfer_learning.py", line 182, in run_with_args cat_train, cat_eval = Pool(data=X_train, label=y_train, cat_features=cat_features), Pool( File "/opt/conda/lib/python3.8/site-packages/catboost/core.py", line 628, in __init__ self._init(data, label, cat_features, text_features, embedding_features, pairs, weight, group_id, group_weight, subgroup_id, pairs_weight, baseline, timestamp, feature_names, thread_count) File "/opt/conda/lib/python3.8/site-packages/catboost/core.py", line 1171, in _init self._init_pool(data, label, cat_features, text_features, embedding_features, pairs, weight, group_id, group_weight, subgroup_id, pairs_weight, baseline, timestamp, feature_names, thread_count) File "_catboost.pyx", line 3755, in _catboost._PoolBase._init_pool File "_catboost.pyx", line 3803, in _catboost._PoolBase._init_pool File "_catboost.pyx", line 3638, in _catboost._PoolBase._init_features_order_layout_pool File "_catboost.pyx", line 2664, in _catboost._set_features_order_data_pd_data_frame File "_catboost.pyx", line 2208, in _catboost.create_num_factor_data File "_catboost.pyx", line 2169, in _catboost.get_float_feature _catboost.CatBoostError: Bad value for num_feature[non_default_doc_idx=0,feature_idx=2]="BROOKLYN": Cannot convert 'b'BROOKLYN'' to float"
Command "/opt/conda/bin/python3.8 transfer_learning.py"
2022-07-22 07:36:34,967 sagemaker-training-toolkit ERROR Encountered exit_code 1
2022-07-22 07:36:47 Uploading - Uploading generated training model
2022-07-22 07:36:47 Failed - Training job failed
---------------------------------------------------------------------------
UnexpectedStatusException Traceback (most recent call last)
/tmp/ipykernel_13950/1137348215.py in <cell line: 23>()
21
22 # Launch a SageMaker Training job by passing s3 path of the training data
---> 23 tabular_estimator.fit(
24 {"training": training_dataset_s3_path}, logs=True, job_name=training_job_name
25 )
~/anaconda3/envs/python3/lib/python3.8/site-packages/sagemaker/estimator.py in fit(self, inputs, wait, logs, job_name, experiment_config)
953 ):
954 """Train a model using the input training dataset.
--> 955
956 The API calls the Amazon SageMaker CreateTrainingJob API to start
957 model training. The API uses configuration you provided to create the
~/anaconda3/envs/python3/lib/python3.8/site-packages/sagemaker/estimator.py in wait(self, logs)
1954 logger.debug(
1955 "Selecting TrainingInput's input_mode (%s) for TrainingInputMode.",
-> 1956 inputs.config["InputMode"],
1957 )
1958 train_args["input_mode"] = inputs.config["InputMode"]
~/anaconda3/envs/python3/lib/python3.8/site-packages/sagemaker/session.py in logs_for_job(self, job_name, wait, poll, log_type)
3796 color_wrap,
3797 )
-> 3798 if state == LogState.COMPLETE:
3799 break
3800
~/anaconda3/envs/python3/lib/python3.8/site-packages/sagemaker/session.py in _check_job_status(self, job, desc, status_key_name)
3334
3335 Args:
-> 3336 name (str): Name of the Amazon SageMaker batch transform job.
3337
3338 Raises:
UnexpectedStatusException: Error for Training job jumpstart-catboost-classification-model-2022-07-22-07-33-18-038: Failed. Reason: AlgorithmError: ExecuteUserScriptError:
ExitCode 1
ErrorMessage "TypeError: Cannot convert 'b'BROOKLYN'' to float
During handling of the above exception, another exception occurred: Traceback (most recent call last): File "transfer_learning.py", line 221, in <module> run_with_args(args) File "transfer_learning.py", line 182, in run_with_args cat_train, cat_eval = Pool(data=X_train, label=y_train, cat_features=cat_features), Pool( File "/opt/conda/lib/python3.8/site-packages/catboost/core.py", line 628, in __init__ self._init(data, label, cat_features, text_features, embedding_features, pairs, weight, group_id, group_weight, subgroup_id, pairs_weight, baseline, timestamp, feature_names, thread_count) File "/opt/conda/lib/python3.8/site-packages/catboost/core.py", line 1171, in _init self._init_pool(data, label, cat_features, text_features, embedding_features, pairs, weight, group_id, group_weight, subgroup_id, pairs_weight, baseline, timestamp, feature_names, thread_count) File "_catboost.pyx", lin

Running deeppavlov model in a container results in TypeError: Descriptors cannot not be created directly

I'm trying to run one of deeppavlov's models in a docker container on Windows 10, but I'm getting an error: 'TypeError: Descriptors cannot not be created directly.'
Could someone please explain what's going wrong here?
At first I typed in "docker pull deeppavlov/base-cpu" to get the image, and then this:
PS C:\Users\user> docker run -e CONFIG=ner_ontonotes -p 5555:5000 -v ~/my_dp_components:/root/.deeppavlov -v ~/my_dp_envs:/venv deeppavlov/base-cpu
2022-07-10 12:13:50.324 INFO in 'deeppavlov.core.common.file'['file'] at line 32: Interpreting 'ner_ontonotes' as '/base/DeepPavlov/deeppavlov/configs/ner/ner_ontonotes.json'
Collecting tensorflow==1.15.2
Downloading tensorflow-1.15.2-cp37-cp37m-manylinux2010_x86_64.whl (110.5 MB)
Collecting keras-applications>=1.0.8
Downloading Keras_Applications-1.0.8-py3-none-any.whl (50 kB)
Collecting tensorboard<1.16.0,>=1.15.0
Downloading tensorboard-1.15.0-py3-none-any.whl (3.8 MB)
Collecting astor>=0.6.0
Downloading astor-0.8.1-py2.py3-none-any.whl (27 kB)
Requirement already satisfied: six>=1.10.0 in ./venv/lib/python3.7/site-packages/six-1.16.0-py3.7.egg (from tensorflow==1.15.2) (1.16.0)
Collecting opt-einsum>=2.3.2
Downloading opt_einsum-3.3.0-py3-none-any.whl (65 kB)
Collecting absl-py>=0.7.0
Downloading absl_py-1.1.0-py3-none-any.whl (123 kB)
Collecting keras-preprocessing>=1.0.5
Downloading Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB)
Collecting grpcio>=1.8.6
Downloading grpcio-1.47.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.5 MB)
Requirement already satisfied: wheel>=0.26 in ./venv/lib/python3.7/site-packages (from tensorflow==1.15.2) (0.36.2)
Collecting wrapt>=1.11.1
Downloading wrapt-1.14.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (75 kB)
Collecting tensorflow-estimator==1.15.1
Downloading tensorflow_estimator-1.15.1-py2.py3-none-any.whl (503 kB)
Requirement already satisfied: numpy<2.0,>=1.16.0 in ./venv/lib/python3.7/site-packages/numpy-1.18.0-py3.7-linux-x86_64.egg (from tensorflow==1.15.2) (1.18.0)
Collecting google-pasta>=0.1.6
Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB)
Collecting termcolor>=1.1.0
Downloading termcolor-1.1.0.tar.gz (3.9 kB)
Collecting protobuf>=3.6.1
Downloading protobuf-4.21.2-cp37-abi3-manylinux2014_x86_64.whl (407 kB)
Collecting gast==0.2.2
Downloading gast-0.2.2.tar.gz (10 kB)
Requirement already satisfied: h5py in ./venv/lib/python3.7/site-packages/h5py-2.10.0-py3.7-linux-x86_64.egg (from keras-applications>=1.0.8->tensorflow==1.15.2) (2.10.0)
Requirement already satisfied: setuptools>=41.0.0 in ./venv/lib/python3.7/site-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow==1.15.2) (57.0.0)
Collecting werkzeug>=0.11.15
Downloading Werkzeug-2.1.2-py3-none-any.whl (224 kB)
Collecting markdown>=2.6.8
Downloading Markdown-3.3.7-py3-none-any.whl (97 kB)
Collecting importlib-metadata>=4.4
Downloading importlib_metadata-4.12.0-py3-none-any.whl (21 kB)
Collecting zipp>=0.5
Downloading zipp-3.8.0-py3-none-any.whl (5.4 kB)
Requirement already satisfied: typing-extensions>=3.6.4 in ./venv/lib/python3.7/site-packages/typing_extensions-3.10.0.0-py3.7.egg (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<1.16.0,>=1.15.0->tensorflow==1.15.2) (3.10.0.0)
Building wheels for collected packages: gast, termcolor
Building wheel for gast (setup.py): started
Building wheel for gast (setup.py): finished with status 'done'
Created wheel for gast: filename=gast-0.2.2-py3-none-any.whl size=7553 sha256=669a2d92bdd23f624a8ead4e4353fa016514b23fad922f801b1109678bfd7d78
Stored in directory: /root/.cache/pip/wheels/21/7f/02/420f32a803f7d0967b48dd823da3f558c5166991bfd204eef3
Building wheel for termcolor (setup.py): started
Building wheel for termcolor (setup.py): finished with status 'done'
Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=4847 sha256=fed5779a43e12fb9fcc5daab1ad2edd126970ddcf1c270954e198d2000f28e42
Stored in directory: /root/.cache/pip/wheels/3f/e3/ec/8a8336ff196023622fbcb36de0c5a5c218cbb24111d1d4c7f2
Successfully built gast termcolor
Installing collected packages: zipp, importlib-metadata, werkzeug, protobuf, markdown, grpcio, absl-py, wrapt, termcolor, tensorflow-estimator, tensorboard, opt-einsum, keras-preprocessing, keras-applications, google-pasta, gast, astor, tensorflow
Successfully installed absl-py-1.1.0 astor-0.8.1 gast-0.2.2 google-pasta-0.2.0 grpcio-1.47.0 importlib-metadata-4.12.0 keras-applications-1.0.8 keras-preprocessing-1.1.2 markdown-3.3.7 opt-einsum-3.3.0 protobuf-4.21.2 tensorboard-1.15.0 tensorflow-1.15.2 tensorflow-estimator-1.15.1 termcolor-1.1.0 werkzeug-2.1.2 wrapt-1.14.1 zipp-3.8.0
WARNING: You are using pip version 21.1.2; however, version 22.1.2 is available.
You should consider upgrading via the '/base/venv/bin/python -m pip install --upgrade pip' command.
Collecting gensim==3.8.1
Downloading gensim-3.8.1-cp37-cp37m-manylinux1_x86_64.whl (24.2 MB)
Collecting smart-open>=1.8.1
Downloading smart_open-6.0.0-py3-none-any.whl (58 kB)
Requirement already satisfied: numpy>=1.11.3 in ./venv/lib/python3.7/site-packages/numpy-1.18.0-py3.7-linux-x86_64.egg (from gensim==3.8.1) (1.18.0)
Requirement already satisfied: scipy>=0.18.1 in ./venv/lib/python3.7/site-packages/scipy-1.4.1-py3.7-linux-x86_64.egg (from gensim==3.8.1) (1.4.1)
Requirement already satisfied: six>=1.5.0 in ./venv/lib/python3.7/site-packages/six-1.16.0-py3.7.egg (from gensim==3.8.1) (1.16.0)
Installing collected packages: smart-open, gensim
Successfully installed gensim-3.8.1 smart-open-6.0.0
WARNING: You are using pip version 21.1.2; however, version 22.1.2 is available.
You should consider upgrading via the '/base/venv/bin/python -m pip install --upgrade pip' command.
2022-07-10 12:14:42.20 INFO in 'deeppavlov.core.common.file'['file'] at line 32: Interpreting 'ner_ontonotes' as '/base/DeepPavlov/deeppavlov/configs/ner/ner_ontonotes.json'
2022-07-10 12:14:43.7 INFO in 'deeppavlov.core.data.utils'['utils'] at line 95: Downloading from http://files.deeppavlov.ai/embeddings/glove.6B.100d.txt?config=ner_ontonotes to /root/.deeppavlov/downloads/embeddings/glove.6B.100d.txt
347MB [00:13, 25.1MB/s]
2022-07-10 12:14:57.596 INFO in 'deeppavlov.core.data.utils'['utils'] at line 95: Downloading from http://files.deeppavlov.ai/deeppavlov_data/ner_ontonotes_v3_cpu_compatible.tar.gz?config=ner_ontonotes to /root/.deeppavlov/ner_ontonotes_v3_cpu_compatible.tar.gz
100%|██████████| 8.13M/8.13M [00:01<00:00, 7.53MB/s]
2022-07-10 12:14:59.90 INFO in 'deeppavlov.core.data.utils'['utils'] at line 272: Extracting /root/.deeppavlov/ner_ontonotes_v3_cpu_compatible.tar.gz archive into /root/.deeppavlov/models
2022-07-10 12:14:59.749 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 115: [loading vocabulary from /root/.deeppavlov/models/ner_ontonotes/tag.dict]
2022-07-10 12:14:59.751 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 115: [loading vocabulary from /root/.deeppavlov/models/ner_ontonotes/char.dict]
2022-07-10 12:14:59.825 INFO in 'deeppavlov.models.embedders.glove_embedder'['glove_embedder'] at line 52: [loading GloVe embeddings from `/root/.deeppavlov/downloads/embeddings/glove.6B.100d.txt`]
Traceback (most recent call last):
File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/base/DeepPavlov/deeppavlov/__main__.py", line 4, in <module>
main()
File "/base/DeepPavlov/deeppavlov/deep.py", line 113, in main
start_model_server(pipeline_config_path, args.https, args.key, args.cert, port=args.port)
File "/base/DeepPavlov/deeppavlov/utils/server/server.py", line 179, in start_model_server
model = build_model(model_config)
File "/base/DeepPavlov/deeppavlov/core/commands/infer.py", line 62, in build_model
component = from_params(component_config, mode=mode, serialized=component_serialized)
File "/base/DeepPavlov/deeppavlov/core/common/params.py", line 95, in from_params
obj = get_model(cls_name)
File "/base/DeepPavlov/deeppavlov/core/common/registry.py", line 74, in get_model
return cls_from_str(_REGISTRY[name])
File "/base/DeepPavlov/deeppavlov/core/common/registry.py", line 42, in cls_from_str
return getattr(importlib.import_module(module_name), cls_name)
File "/base/venv/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/base/DeepPavlov/deeppavlov/models/ner/network.py", line 19, in <module>
import tensorflow as tf
File "/base/venv/lib/python3.7/site-packages/tensorflow/__init__.py", line 99, in <module>
from tensorflow_core import *
File "/base/venv/lib/python3.7/site-packages/tensorflow_core/__init__.py", line 28, in <module>
from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import
File "<frozen importlib._bootstrap>", line 1019, in _handle_fromlist
File "/base/venv/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
module = self._load()
File "/base/venv/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
module = _importlib.import_module(self.__name__)
File "/base/venv/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/base/venv/lib/python3.7/site-packages/tensorflow_core/python/__init__.py", line 52, in <module>
from tensorflow.core.framework.graph_pb2 import *
File "/base/venv/lib/python3.7/site-packages/tensorflow_core/core/framework/graph_pb2.py", line 16, in <module>
from tensorflow.core.framework import node_def_pb2 as tensorflow_dot_core_dot_framework_dot_node__def__pb2
File "/base/venv/lib/python3.7/site-packages/tensorflow_core/core/framework/node_def_pb2.py", line 16, in <module>
from tensorflow.core.framework import attr_value_pb2 as tensorflow_dot_core_dot_framework_dot_attr__value__pb2
File "/base/venv/lib/python3.7/site-packages/tensorflow_core/core/framework/attr_value_pb2.py", line 16, in <module>
from tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2
File "/base/venv/lib/python3.7/site-packages/tensorflow_core/core/framework/tensor_pb2.py", line 16, in <module>
from tensorflow.core.framework import resource_handle_pb2 as tensorflow_dot_core_dot_framework_dot_resource__handle__pb2
File "/base/venv/lib/python3.7/site-packages/tensorflow_core/core/framework/resource_handle_pb2.py", line 16, in <module>
from tensorflow.core.framework import tensor_shape_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__shape__pb2
File "/base/venv/lib/python3.7/site-packages/tensorflow_core/core/framework/tensor_shape_pb2.py", line 42, in <module>
serialized_options=None, file=DESCRIPTOR),
File "/base/venv/lib/python3.7/site-packages/google/protobuf/descriptor.py", line 560, in __new__
_message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
1. Downgrade the protobuf package to 3.20.x or lower.
2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
The image has just been updated. Please, try again.

Problems extending Airflow image for Helm

I'm using HELM to install Airflow in a Kubernetes cluster. I wanted the pods to have an additional apt dependency (sshpass) and as such, I was trying to extend the Airflow image Helm uses and add that apt dependency using Docker. My Dockerfile looks like this:
FROM apache/airflow:1.10.12-python3.6
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
sshpass \
&& apt-get autoremove -yqq --purge \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
USER airflow
I followed the example given by Airflow doc here: https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html#extending-the-image, but replaced the image with the one Helm uses.
I then built and published the new image, I made Helm use it by editing the values.yaml file like so:
airflow:
## configs for the docker image of the web/scheduler/worker
##
image:
repository: myaccount/myrepo
tag: mytag
## values: Always or IfNotPresent
pullPolicy: IfNotPresent
pullSecret: ""
I then ran the installation command but the pods that are launched go into CrashLoopBackOff and Error states. I checked the logs of one of the pods which is:
*** installing requirements...
Collecting airflow-multi-dagrun==1.2 (from -r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/af/41/e60dff951d002dbf14daf601b1323dfc48c0d24d2bc4e7d19ac72b19c3f6/airflow_multi_dagrun-1.2-py3-none-any.whl
Collecting azure-storage-blob==12.4.0 (from -r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/60/dc/3c25aeab827266c019e13abc07f31b5f47d93f1b8548a417c81c89c9d021/azure_storage_blob-12.4.0-py2.py3-none-any.whl (326kB)
Collecting azure-cosmosdb-table==1.0.6 (from -r requirements.txt (line 3))
Downloading https://files.pythonhosted.org/packages/f0/e4/15a59108883cc47460b1475aeac935e2d975b5def42f2c0a8b8fd48b3304/azure_cosmosdb_table-1.0.6-py2.py3-none-any.whl (125kB)
Collecting pandas==1.1.2 (from -r requirements.txt (line 4))
Downloading https://files.pythonhosted.org/packages/1c/11/e1f53db0614f2721027aab297c8afd2eaf58d33d566441a97ea454541c5e/pandas-1.1.2-cp36-cp36m-manylinux1_x86_64.whl (10.5MB)
Collecting pyarrow==1.0.1 (from -r requirements.txt (line 5))
Downloading https://files.pythonhosted.org/packages/1f/32/34b44246e671a2ba672fc2551dd3a85472ba07e58a75da1dbc655810fbfa/pyarrow-1.0.1-cp36-cp36m-manylinux2010_x86_64.whl (17.5MB)
Collecting azure-core<2.0.0,>=1.6.0 (from azure-storage-blob==12.4.0->-r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/12/9e/6bb67fe85f6a89d71f50c86a0da778a5064f749a485ed9ba498067034227/azure_core-1.10.0-py2.py3-none-any.whl (125kB)
Collecting msrest>=0.6.10 (from azure-storage-blob==12.4.0->-r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/fa/f5/9e315fe8cb985b0ce052b34bcb767883dc739f46fadb62f05a7e6d6eedbe/msrest-0.6.19-py2.py3-none-any.whl (84kB)
Collecting cryptography>=2.1.4 (from azure-storage-blob==12.4.0->-r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/c9/de/7054df0620b5411ba45480f0261e1fb66a53f3db31b28e3aa52c026e72d9/cryptography-3.3.1-cp36-abi3-manylinux2010_x86_64.whl (2.6MB)
Collecting azure-common>=1.1.5 (from azure-cosmosdb-table==1.0.6->-r requirements.txt (line 3))
Downloading https://files.pythonhosted.org/packages/19/2b/46ada1753c4a640bc3ad04a1e20b1a5ea52a8f18079e1b8238e536aa0c98/azure_common-1.1.26-py2.py3-none-any.whl
Collecting requests (from azure-cosmosdb-table==1.0.6->-r requirements.txt (line 3))
Downloading https://files.pythonhosted.org/packages/29/c1/24814557f1d22c56d50280771a17307e6bf87b70727d975fd6b2ce6b014a/requests-2.25.1-py2.py3-none-any.whl (61kB)
Collecting azure-cosmosdb-nspkg>=2.0.0 (from azure-cosmosdb-table==1.0.6->-r requirements.txt (line 3))
Downloading https://files.pythonhosted.org/packages/58/0d/1329b47e5386b0acf4e42ada2284851eff60ef3337a87e5b2dfedabbfcb1/azure_cosmosdb_nspkg-2.0.2-py2.py3-none-any.whl
Collecting python-dateutil (from azure-cosmosdb-table==1.0.6->-r requirements.txt (line 3))
Downloading https://files.pythonhosted.org/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl (227kB)
Collecting pytz>=2017.2 (from pandas==1.1.2->-r requirements.txt (line 4))
Downloading https://files.pythonhosted.org/packages/89/06/2c2d3034b4d6bf22f2a4ae546d16925898658a33b4400cfb7e2c1e2871a3/pytz-2020.5-py2.py3-none-any.whl (510kB)
Collecting numpy>=1.15.4 (from pandas==1.1.2->-r requirements.txt (line 4))
Downloading https://files.pythonhosted.org/packages/14/32/d3fa649ad7ec0b82737b92fefd3c4dd376b0bb23730715124569f38f3a08/numpy-1.19.5-cp36-cp36m-manylinux2010_x86_64.whl (14.8MB)
Collecting six>=1.6 (from azure-core<2.0.0,>=1.6.0->azure-storage-blob==12.4.0->-r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from msrest>=0.6.10->azure-storage-blob==12.4.0->-r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/5e/a0/5f06e1e1d463903cf0c0eebeb751791119ed7a4b3737fdc9a77f1cdfb51f/certifi-2020.12.5-py2.py3-none-any.whl (147kB)
Collecting requests-oauthlib>=0.5.0 (from msrest>=0.6.10->azure-storage-blob==12.4.0->-r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/a3/12/b92740d845ab62ea4edf04d2f4164d82532b5a0b03836d4d4e71c6f3d379/requests_oauthlib-1.3.0-py2.py3-none-any.whl
Collecting isodate>=0.6.0 (from msrest>=0.6.10->azure-storage-blob==12.4.0->-r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/9b/9f/b36f7774ff5ea8e428fdcfc4bb332c39ee5b9362ddd3d40d9516a55221b2/isodate-0.6.0-py2.py3-none-any.whl (45kB)
Collecting cffi>=1.12 (from cryptography>=2.1.4->azure-storage-blob==12.4.0->-r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/1c/1a/90fa7e7ee05d91d0339ef264bd8c008f57292aba4a91ec512a0bbb379d1d/cffi-1.14.4-cp36-cp36m-manylinux1_x86_64.whl (401kB)
Collecting chardet<5,>=3.0.2 (from requests->azure-cosmosdb-table==1.0.6->-r requirements.txt (line 3))
Downloading https://files.pythonhosted.org/packages/19/c7/fa589626997dd07bd87d9269342ccb74b1720384a4d739a1872bd84fbe68/chardet-4.0.0-py2.py3-none-any.whl (178kB)
Collecting urllib3<1.27,>=1.21.1 (from requests->azure-cosmosdb-table==1.0.6->-r requirements.txt (line 3))
Downloading https://files.pythonhosted.org/packages/f5/71/45d36a8df68f3ebb098d6861b2c017f3d094538c0fb98fa61d4dc43e69b9/urllib3-1.26.2-py2.py3-none-any.whl (136kB)
Collecting idna<3,>=2.5 (from requests->azure-cosmosdb-table==1.0.6->-r requirements.txt (line 3))
Downloading https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl (58kB)
Collecting azure-nspkg>=2.0.0 (from azure-cosmosdb-nspkg>=2.0.0->azure-cosmosdb-table==1.0.6->-r requirements.txt (line 3))
Downloading https://files.pythonhosted.org/packages/c4/0c/c562be95a9a2ed52454f598571cf300b1114d0db2aa27f5b8ed3bb9cd0c0/azure_nspkg-3.0.2-py3-none-any.whl
Collecting oauthlib>=3.0.0 (from requests-oauthlib>=0.5.0->msrest>=0.6.10->azure-storage-blob==12.4.0->-r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/05/57/ce2e7a8fa7c0afb54a0581b14a65b56e62b5759dbc98e80627142b8a3704/oauthlib-3.1.0-py2.py3-none-any.whl (147kB)
Collecting pycparser (from cffi>=1.12->cryptography>=2.1.4->azure-storage-blob==12.4.0->-r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl (112kB)
Installing collected packages: airflow-multi-dagrun, six, chardet, urllib3, idna, certifi, requests, azure-core, oauthlib, requests-oauthlib, isodate, msrest, pycparser, cffi, cryptography, azure-storage-blob, azure-common, azure-nspkg,azure-cosmosdb-nspkg, python-dateutil, azure-cosmosdb-table, pytz, numpy, pandas, pyarrow
The script chardetect is installed in '/root/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
The scripts f2py, f2py3 and f2py3.6 are installed in '/root/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
The script plasma_store is installed in '/root/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed airflow-multi-dagrun-1.2 azure-common-1.1.26 azure-core-1.10.0 azure-cosmosdb-nspkg-2.0.2 azure-cosmosdb-table-1.0.6 azure-nspkg-3.0.2 azure-storage-blob-12.4.0 certifi-2020.12.5 cffi-1.14.4 chardet-4.0.0 cryptography-3.3.1 idna-2.10 isodate-0.6.0 msrest-0.6.19 numpy-1.19.5 oauthlib-3.1.0 pandas-1.1.2 pyarrow-1.0.1 pycparser-2.20 python-dateutil-2.8.1 pytz-2020.5 requests-2.25.1 requests-oauthlib-1.3.0 six-1.15.0 urllib3-1.26.2
You are using pip version 19.0.2, however version 20.3.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
*** installing extra pip packages...
Requirement already satisfied: azure-storage-blob==12.4.0 in /root/.local/lib/python3.6/site-packages (12.4.0)
Requirement already satisfied: azure-core<2.0.0,>=1.6.0 in /root/.local/lib/python3.6/site-packages (from azure-storage-blob==12.4.0) (1.10.0)
Requirement already satisfied: msrest>=0.6.10 in /root/.local/lib/python3.6/site-packages (from azure-storage-blob==12.4.0) (0.6.19)
Requirement already satisfied: cryptography>=2.1.4 in /root/.local/lib/python3.6/site-packages (from azure-storage-blob==12.4.0) (3.3.1)
Requirement already satisfied: six>=1.6 in /root/.local/lib/python3.6/site-packages (from azure-core<2.0.0,>=1.6.0->azure-storage-blob==12.4.0) (1.15.0)
Requirement already satisfied: requests>=2.18.4 in /root/.local/lib/python3.6/site-packages (from azure-core<2.0.0,>=1.6.0->azure-storage-blob==12.4.0) (2.25.1)
Requirement already satisfied: requests-oauthlib>=0.5.0 in /root/.local/lib/python3.6/site-packages (from msrest>=0.6.10->azure-storage-blob==12.4.0) (1.3.0)
Requirement already satisfied: certifi>=2017.4.17 in /root/.local/lib/python3.6/site-packages (from msrest>=0.6.10->azure-storage-blob==12.4.0) (2020.12.5)
Requirement already satisfied: isodate>=0.6.0 in /root/.local/lib/python3.6/site-packages (from msrest>=0.6.10->azure-storage-blob==12.4.0) (0.6.0)
Requirement already satisfied: cffi>=1.12 in /root/.local/lib/python3.6/site-packages (from cryptography>=2.1.4->azure-storage-blob==12.4.0) (1.14.4)
Requirement already satisfied: idna<3,>=2.5 in /root/.local/lib/python3.6/site-packages (from requests>=2.18.4->azure-core<2.0.0,>=1.6.0->azure-storage-blob==12.4.0) (2.10)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /root/.local/lib/python3.6/site-packages (from requests>=2.18.4->azure-core<2.0.0,>=1.6.0->azure-storage-blob==12.4.0) (1.26.2)
Requirement already satisfied: chardet<5,>=3.0.2 in /root/.local/lib/python3.6/site-packages (from requests>=2.18.4->azure-core<2.0.0,>=1.6.0->azure-storage-blob==12.4.0) (4.0.0)
Requirement already satisfied: oauthlib>=3.0.0 in /root/.local/lib/python3.6/site-packages (from requests-oauthlib>=0.5.0->msrest>=0.6.10->azure-storage-blob==12.4.0) (3.1.0)
Requirement already satisfied: pycparser in /root/.local/lib/python3.6/site-packages (from cffi>=1.12->cryptography>=2.1.4->azure-storage-blob==12.4.0) (2.20)
You are using pip version 19.0.2, however version 20.3.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
*** running webserver...
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 23, in <module>
import argcomplete
ModuleNotFoundError: No module named 'argcomplete'
From what I can gather here, it seems that the python dependencies Airflow needs are being installed with the root user (/root/.local/lib/python3.6/site-packages) instead of the airflow user, that is then used to execute airflow that therefore doesn't have said dependencies. Without me trying to extend the Airflow image and just using the default one in Helm chart, the installation python path is /home/airflow/.local/lib/python3.6/site-packages and everything else goes well. Why does my custom image behave like this?
Do I need to add anything else to Docker file? Am I doing something wrong? How can I fix this?
Thank you in advance

Still loads previous version after upgrading latest version OpenCV 4.4.0

I upgraded OpenCV 4.4.0 from OpenCV 4.2.x with pip command :
pip install --upgrade opencv-python==4.4.0.40
and checked the upgrade completed :
Collecting opencv-python==4.4.0.40
|████████████████████████████████| 33.5 MB 283 kB/s
Requirement already satisfied, skipping upgrade: numpy>=1.17.3 in c:\users\kangs\anaconda3\lib\site-packages (from opencv-python==4.4.0.40) (1.18.5)
Found existing installation: opencv-python 4.3.0.36
Successfully uninstalled opencv-python-4.3.0.36
PS C:\Users\kangs\Documents\TELPA\Source\numberplateRecognition> python
Python 3.8.3 (default, Jul 2 2020, 17:30:36) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cv2.__version__
'4.4.0'
>>> exit()
However, I still get this error message when I run this code :
cv2.dnn.readNet(WEIGHTS, CFG)
Error message :
Traceback (most recent call last):
File "c:\Users\kangs\Documents\TELPA\Source\numberplateRecognition\number_plate_v4c.py", line 54, in <module>
File "c:\Users\kangs\Documents\TELPA\Source\numberplateRecognition\algorithm\detection\yolodetector.py", line 39, in __init__
self.load_dir(dir_path)
File "c:\Users\kangs\Documents\TELPA\Source\numberplateRecognition\algorithm\detection\yolodetector.py", line 50, in load_dir
self.load_files(file_list)
File "c:\Users\kangs\Documents\TELPA\Source\numberplateRecognition\algorithm\detection\yolodetector.py", line 78, in load_files
self.net = cv2.dnn.readNet(WEIGHTS, CFG)
cv2.error: OpenCV(4.2.0) C:\projects\opencv-python\opencv\modules\dnn\src\darknet\darknet_io.cpp:686: error: (-212:Parsing error) Unsupported activation: mish in function 'cv::dnn::darknet::ReadDarknetFromCfgStream'
I found out that the code still linked to the previous library OpenCV 4.2.0.
Can someone please let me know why this happens and how to solve this problem?

Resources