Creating Singularity container from local private docker registry - docker

When creating a Singularity image from a Docker image, how do I correctly reference a Docker image in a local insecure private registry?
Using Singularity 2.2.1, attempted using specfile:
Bootstrap: docker
Registry: localhost:5000
From: tensorflow:latest
%runscript
exec /usr/bin/python "$#"
%post
echo "Post install stuffs!"
With the following results:
Executing Prebootstrap module
VERBOSE [U=0,P=22966] message.c:52:init() : Set messagelevel to: 5
DEBUG [U=0,P=22966] get-section.c:66:main() : Iterating through /proc/mounts
Executing Bootstrap 'docker' module
From: tensorflow:latest
Registry: https://localhost:5000
library/tensorflow:latest
scope=repository:library/tensorflow:pull
URL: https://localhost:5000/v2/library/tensorflow/manifests/latest
Traceback (most recent call last):
File "/usr/lib/x86_64-linux-gnu/singularity/python/cli.py", line 198, in <module>
main()
File "/usr/lib/x86_64-linux-gnu/singularity/python/cli.py", line 146, in main
auth=doauth)
File "/usr/lib/x86_64-linux-gnu/singularity/python/docker/api.py", line 156, in get_manifest
response = api_get(base,headers=token,default_header=True)
File "/usr/lib/x86_64-linux-gnu/singularity/python/utils.py", line 139, in api_get
response = urllib2.urlopen(request)
File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 429, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 447, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1241, in https_open
context=self._context)
File "/usr/lib/python2.7/urllib2.py", line 1198, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:661)>
Executing Postbootstrap module
ERROR: Container does not contain the valid minimum requirement of /bin/sh
DEBUG [U=0,P=22936] fork.c:52:handle_sigchld() : Checking child pids: 22941 22941
Singularity is making an assumption about HTTPS on the private registry.
Thanks,
Piers.

In answer to my own question - it turns out that I need to do two things:
explicitly add http:// to the local registry URL
prefix the Docker image name with "/" so that it doesn't get prepended with "/library/"
Example specfile:
Bootstrap: docker
Registry: http://localhost:5000
From: /tensorflow:latest
%runscript
exec /usr/bin/python "$#"
%post
echo "Post install stuffs!"

Related

airflow worker crashing in helms upgrade with Temporary failure in name resolution for postgres

I have been trying to use a custom dockerfile for mounting dags and plugins as follows:
FROM apache/airflow:2.3.0-python3.7
COPY ./dags/ /opt/airflow/dags/
COPY ./plugins/ /opt/airflow/plugins/docker push
COPY requirements.txt .
RUN pip install -r requirements.txt
EXPOSE 5555
which I am building as:
docker build -f base.dockerfile --pull --tag lqc-airflow:0.0.1 .
minikube image load lqc-airflow:0.0.1
and then doing a helm install
helm upgrade $RELEASE_NAME apache-airflow/airflow --namespace $NAMESPACE --set images.airflow.repository=lqc-airflow --set images.airflow.tag=0.0.1
which however is making just the airflow-worker-0 pod fail due to the following error:
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py", line 38, in main
args.func(args)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 51, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 99, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/celery_command.py", line 130, in worker
session = celery_app.backend.ResultSession()
File "/home/airflow/.local/lib/python3.7/site-packages/celery/backends/database/__init__.py", line 109, in ResultSession
**self.engine_options)
File "/home/airflow/.local/lib/python3.7/site-packages/celery/backends/database/session.py", line 88, in session_factory
self.prepare_models(engine)
File "/home/airflow/.local/lib/python3.7/site-packages/celery/backends/database/session.py", line 72, in prepare_models
ResultModelBase.metadata.create_all(engine)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/schema.py", line 4745, in create_all
ddl.SchemaGenerator, self, checkfirst=checkfirst, tables=tables
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 3007, in _run_ddl_visitor
with self.begin() as conn:
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2923, in begin
conn = self.connect(close_with_result=close_with_result)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 3095, in connect
return self._connection_cls(self, close_with_result=close_with_result)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 91, in __init__
else engine.raw_connection()
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 3174, in raw_connection
return self._wrap_pool_connect(self.pool.connect, _connection)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 3145, in _wrap_pool_connect
e, dialect, self
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2004, in _handle_dbapi_exception_noconnection
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
raise exception
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 3141, in _wrap_pool_connect
return fn()
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 301, in connect
return _ConnectionFairy._checkout(self)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 755, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 419, in checkout
rec = pool._do_get()
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 259, in _do_get
return self._create_connection()
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 247, in _create_connection
return _ConnectionRecord(self)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 362, in __init__
self.__connect(first_connect_check=True)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 605, in __connect
pool.logger.debug("Error on connect(): %s", e)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 72, in __exit__
with_traceback=exc_tb,
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
raise exception
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 599, in __connect
connection = pool._invoke_creator(self)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/create.py", line 578, in connect
return dialect.connect(*cargs, **cparams)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 583, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/home/airflow/.local/lib/python3.7/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "postgres" to address: Temporary failure in name resolution
I am just following the reading advisory from airflow: https://airflow.apache.org/docs/helm-chart/stable/manage-dags-files.html
please note that there are no such name resolution errors if I dont use my custom docker file. Kindly help!
thanks #Hussein also for your support, but I could solve it by myself. I saw that the logs of the airflow-migrations pods, complaint of a revision id not being found. So somehow I chanced upon: https://airflow.apache.org/docs/apache-airflow/stable/migrations-ref.html
and there I saw the airflow tag which was above/carrying that alembic revision.
My revision id was ecb43d2a1842 and thus the changes to my docker file were:
FROM apache/airflow:2.4.3
COPY ./dags/ /opt/airflow/dags/
COPY ./plugins/ /opt/airflow/plugins/
COPY requirements.txt .
RUN pip install -r requirements.txt
thus 2.4.3 was the catch.

Configure gsutil s3 in Google Cloud SDK Docker

I have Google Cloud SDK docker configured and running on my windows machine after following this.
https://hub.docker.com/r/google/cloud-sdk/
I'm trying to run this command to list a s3 bucket
docker run --rm -ti --volumes-from gcloud-config google/cloud-sdk gsutil ls s3://bucketname
Authentication fails due to not setting the AWS keys. I presume from .boto file not having aws_access_key_id and aws_secret_access_key set. I can't seem to figure out how to set those variables.
I tried to run this to generate a .boto file but the bucket was shared with me and I don't have access keys.
docker run --rm -ti --volumes-from gcloud-config google/cloud-sdk gsutil config -a
Am I missing something or is there any other way to set these AWS credentials? Maybe with gcloud config set?
Here is the error log
ERROR 1202 03:16:07.326810 utils.py] Caught exception reading instance data
Traceback (most recent call last):
File "/usr/lib/python3.7/urllib/request.py", line 1324, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/usr/lib/python3.7/http/client.py", line 1260, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1306, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1255, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1030, in _send_output
self.send(msg)
File "/usr/lib/python3.7/http/client.py", line 970, in send
self.connect()
File "/usr/lib/python3.7/http/client.py", line 942, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/usr/lib/python3.7/socket.py", line 727, in create_connection
raise err
File "/usr/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/utils.py", line 220, in retry_url
r = opener.open(req, timeout=timeout)
File "/usr/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/usr/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/usr/lib/python3.7/urllib/request.py", line 1352, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/usr/lib/python3.7/urllib/request.py", line 1326, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 111] Connection refused>
ERROR 1202 03:16:07.328018 utils.py] Unable to read instance data, giving up
Traceback (most recent call last):
File "/usr/lib/google-cloud-sdk/platform/gsutil/gsutil", line 21, in <module>
gsutil.RunMain()
File "/usr/lib/google-cloud-sdk/platform/gsutil/gsutil.py", line 122, in RunMain
sys.exit(gslib.__main__.main())
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 444, in main
user_project=user_project)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 783, in _RunNamedCommandAndHandleExceptions
_HandleUnknownFailure(e)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 640, in _RunNamedCommandAndHandleExceptions
user_project=user_project)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 412, in RunNamedCommand
return_code = command_inst.RunCommand()
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/ls.py", line 683, in RunCommand
listing_helper.ExpandUrlAndPrint(storage_url))
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/utils/ls_helper.py", line 372, in ExpandUrlAndPrint
print_initial_newline=False)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/utils/ls_helper.py", line 449, in _RecurseExpandUrlAndPrint
bucket_listing_fields=self.bucket_listing_fields):
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/wildcard_iterator.py", line 540, in IterAll
expand_top_level_buckets=expand_top_level_buckets):
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/wildcard_iterator.py", line 174, in __iter__
fields=bucket_listing_fields):
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/boto_translation.py", line 447, in ListObjects
headers=headers)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/storage_uri.py", line 166, in list_bucket
bucket = self.get_bucket(headers=headers)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/storage_uri.py", line 181, in get_bucket
conn = self.connect()
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/storage_uri.py", line 117, in connect
**connection_args)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/s3/connection.py", line 205, in __init__
validate_certs=validate_certs, profile_name=profile_name)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/connection.py", line 573, in __init__
host, config, self.provider, self._required_auth_capability())
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/auth.py", line 1032, in get_auth_handler
'Check your credentials' % (len(names), str(names)))
boto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['S3HmacAuthV4Handler'] Check your credentials
I edited the .boto file in the legacy config and got this to work.
docker restart gcloud-config
docker exec -u 0 -it <container-id-here>/bin/bash
apt-get install nano
nano root/.config/gcloud/legacy_credentials/***/.boto
add under [Credentials]
aws_access_key_id = ***
aws_secret_access_key = ***
enjoy
docker run --rm -ti --volumes-from gcloud-config google/cloud-sdk gsutil ls s3://bucketname

Unable to connect to MLFLOW_TRACKING_URI when running MLflow run in Docker container

I have setup a mlflow server locally at http://localhost:5000
I followed the instructions at https://github.com/mlflow/mlflow/tree/master/examples/docker and tried to run the example docker with
/mlflow/examples/docker$ mlflow run . -P alpha=0.5
but I encountered the following error.
2021/05/09 17:11:20 INFO mlflow.projects.docker: === Building docker image docker-example:7530274 ===
2021/05/09 17:11:20 INFO mlflow.projects.utils: === Created directory /tmp/tmp9wpxyzd_ for downloading remote URIs passed to arguments of type 'path' ===
2021/05/09 17:11:20 INFO mlflow.projects.backend.local: === Running command 'docker run --rm -v /home/mlf/mlf/0/ae69145133bf49efac22b1d390c354f1/artifacts:/home/mlf/mlf/0/ae69145133bf49efac22b1d390c354f1/artifacts -e MLFLOW_RUN_ID=ae69145133bf49efac22b1d390c354f1 -e MLFLOW_TRACKING_URI=http://localhost:5000 -e MLFLOW_EXPERIMENT_ID=0 docker-example:7530274 python train.py --alpha 0.5 --l1-ratio 0.1' in run with ID 'ae69145133bf49efac22b1d390c354f1' ===
/opt/conda/lib/python2.7/site-packages/mlflow/__init__.py:55: DeprecationWarning: MLflow support for Python 2 is deprecated and will be dropped in a future release. At that point, existing Python 2 workflows that use MLflow will continue to work without modification, but Python 2 users will no longer get access to the latest MLflow features and bugfixes. We recommend that you upgrade to Python 3 - see https://docs.python.org/3/howto/pyporting.html for a migration guide.
"for a migration guide.", DeprecationWarning)
Traceback (most recent call last):
File "train.py", line 56, in <module>
with mlflow.start_run():
File "/opt/conda/lib/python2.7/site-packages/mlflow/tracking/fluent.py", line 122, in start_run
active_run_obj = MlflowClient().get_run(existing_run_id)
File "/opt/conda/lib/python2.7/site-packages/mlflow/tracking/client.py", line 96, in get_run
return self._tracking_client.get_run(run_id)
File "/opt/conda/lib/python2.7/site-packages/mlflow/tracking/_tracking_service/client.py", line 49, in get_run
return self.store.get_run(run_id)
File "/opt/conda/lib/python2.7/site-packages/mlflow/store/tracking/rest_store.py", line 92, in get_run
response_proto = self._call_endpoint(GetRun, req_body)
File "/opt/conda/lib/python2.7/site-packages/mlflow/store/tracking/rest_store.py", line 32, in _call_endpoint
return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)
File "/opt/conda/lib/python2.7/site-packages/mlflow/utils/rest_utils.py", line 133, in call_endpoint
host_creds=host_creds, endpoint=endpoint, method=method, params=json_body)
File "/opt/conda/lib/python2.7/site-packages/mlflow/utils/rest_utils.py", line 70, in http_request
url=url, headers=headers, verify=verify, **kwargs)
File "/opt/conda/lib/python2.7/site-packages/mlflow/utils/rest_utils.py", line 51, in request_with_ratelimit_retries
response = requests.request(**kwargs)
File "/opt/conda/lib/python2.7/site-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/conda/lib/python2.7/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/opt/conda/lib/python2.7/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/opt/conda/lib/python2.7/site-packages/requests/adapters.py", line 508, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /api/2.0/mlflow/runs/get?run_uuid=ae69145133bf49efac22b1d390c354f1&run_id=ae69145133bf49efac22b1d390c354f1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5cbd80d690>: Failed to establish a new connection: [Errno 111] Connection refused',))
2021/05/09 17:11:22 ERROR mlflow.cli: === Run (ID 'ae69145133bf49efac22b1d390c354f1') failed ===
Any ideas how to fix this? I tried adding the following in MLproject file but it doesn't help
environment: [["network", "host"], ["add-host", "host.docker.internal:host-gateway"]]
Thanks for your help! =)
Run MLflow server such was that it will use your machine IP instead of localhost. Then point the mlflow run to that IP instead of http://localhost:5000. The main reason is that localhost of Docker process is its own, not your machine.

Docker Compose, Conda, and non-standard certificates

I'm having some difficulty figuring out how to run conda install ... in a Dockerfile, while referencing a non-standard certificates file. In my Dockerfile, I have:
RUN REQUESTS_CA_BUNDLE=/non-standard-certificates.pem conda update -n base conda -y
which appears to run fine. But then I have:
RUN REQUESTS_CA_BUNDLE=/non-standard-certificates.pem CURL_CA_BUNDLE=/non-standard-certificates.pem conda install -n base -c defaults -c conda-forge <list-of-packages>
which ends in:
CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/conda-forge/linux-64/current_repodata.json>
Elapsed: -
An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
'https://conda.anaconda.org/conda-forge/linux-64'
Can anyone see what is incorrect here?
Update:
I have since figured out that I should probably be using RUN conda config --set client_ssl_cert ..., and that the certificate file in question had Windows carriage-returns in it (which I removed with dos2unix), but now I'm getting a different error:
Step 7/29 : RUN conda update -n base conda -y
---> Running in 052e36266aef
Collecting package metadata (current_repodata.json): ...working... failed
# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/conda/exceptions.py", line 1074, in __call__
return func(*args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/conda/cli/main.py", line 84, in _main
exit_code = do_call(args, p)
File "/opt/conda/lib/python3.7/site-packages/conda/cli/conda_argparse.py", line 82, in do_call
exit_code = getattr(module, func_name)(args, parser)
File "/opt/conda/lib/python3.7/site-packages/conda/cli/main_update.py", line 20, in execute
install(args, parser, 'update')
File "/opt/conda/lib/python3.7/site-packages/conda/cli/install.py", line 265, in install
should_retry_solve=(_should_retry_unfrozen or repodata_fn != repodata_fns[-1]),
File "/opt/conda/lib/python3.7/site-packages/conda/core/solve.py", line 117, in solve_for_transaction
should_retry_solve)
File "/opt/conda/lib/python3.7/site-packages/conda/core/solve.py", line 158, in solve_for_diff
force_remove, should_retry_solve)
File "/opt/conda/lib/python3.7/site-packages/conda/core/solve.py", line 262, in solve_final_state
ssc = self._collect_all_metadata(ssc)
File "/opt/conda/lib/python3.7/site-packages/conda/common/io.py", line 88, in decorated
return f(*args, **kwds)
File "/opt/conda/lib/python3.7/site-packages/conda/core/solve.py", line 415, in _collect_all_metadata
index, r = self._prepare(prepared_specs)
File "/opt/conda/lib/python3.7/site-packages/conda/core/solve.py", line 1004, in _prepare
self.subdirs, prepared_specs, self._repodata_fn)
File "/opt/conda/lib/python3.7/site-packages/conda/core/index.py", line 214, in get_reduced_index
repodata_fn=repodata_fn)
File "/opt/conda/lib/python3.7/site-packages/conda/core/subdir_data.py", line 97, in query_all
result = tuple(concat(executor.map(subdir_query, channel_urls)))
File "/opt/conda/lib/python3.7/concurrent/futures/_base.py", line 598, in result_iterator
yield fs.pop().result()
File "/opt/conda/lib/python3.7/concurrent/futures/_base.py", line 435, in result
return self.__get_result()
File "/opt/conda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/opt/conda/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/conda/lib/python3.7/site-packages/conda/core/subdir_data.py", line 90, in <lambda>
package_ref_or_match_spec))
File "/opt/conda/lib/python3.7/site-packages/conda/core/subdir_data.py", line 102, in query
self.load()
File "/opt/conda/lib/python3.7/site-packages/conda/core/subdir_data.py", line 166, in load
_internal_state = self._load()
File "/opt/conda/lib/python3.7/site-packages/conda/core/subdir_data.py", line 240, in _load
repodata_fn=self.repodata_fn)
File "/opt/conda/lib/python3.7/site-packages/conda/core/subdir_data.py", line 477, in fetch_repodata_remote_request
timeout=timeout)
File "/opt/conda/lib/python3.7/site-packages/requests/sessions.py", line 546, in get
return self.request('GET', url, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 343, in _make_request
self._validate_conn(conn)
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
conn.connect()
File "/opt/conda/lib/python3.7/site-packages/urllib3/connection.py", line 344, in connect
ssl_context=context)
File "/opt/conda/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 338, in ssl_wrap_socket
context.load_cert_chain(certfile, keyfile)
File "/opt/conda/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 439, in load_cert_chain
self._ctx.use_privatekey_file(keyfile or certfile)
File "/opt/conda/lib/python3.7/site-packages/OpenSSL/SSL.py", line 990, in use_privatekey_file
self._raise_passphrase_exception()
File "/opt/conda/lib/python3.7/site-packages/OpenSSL/SSL.py", line 967, in _raise_passphrase_exception
_raise_current_error()
File "/opt/conda/lib/python3.7/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
raise exception_type(errors)
OpenSSL.SSL.Error: [('PEM routines', 'get_name', 'no start line'), ('SSL routines', 'SSL_CTX_use_PrivateKey_file', 'PEM lib')]
`$ /opt/conda/bin/conda update -n base conda -y`
environment variables:
CIO_TEST=<not set>
CONDA_ROOT=/opt/conda
PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin
:/bin
REQUESTS_CA_BUNDLE=<not set>
SSL_CERT_FILE=<not set>
active environment : None
user config file : /root/.condarc
populated config files : /root/.condarc
conda version : 4.7.12
conda-build version : not installed
python version : 3.7.4.final.0
virtual packages :
base environment : /opt/conda (writable)
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /opt/conda/pkgs
/root/.conda/pkgs
envs directories : /opt/conda/envs
/root/.conda/envs
platform : linux-64
user-agent : conda/4.7.12 requests/2.22.0 CPython/3.7.4 Linux/3.10.0-1160.6.1.el7.x86_64 debian/10 glibc/2.28
UID:GID : 0:0
netrc file : None
offline mode : False
An unexpected error has occurred. Conda has prepared the above report.
Upload did not complete.
ERROR: Service 'base_image' failed to build: The command '/bin/sh -c conda update -n base conda -y' returned a non-zero code: 1

Rancher: Adding another Host to Rancher fails on CERTIFICATE_VERIFY_FAILED

I have setup a rancher with a host. Now i'm trying to add another host to the same, here is what I did, installed the docker-ce on the new host and then ran the following:
sudo docker run --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher 10.10.18.35:5000/rancher/ant http://10.10.18.35:8080/v1/scripts/9F4687F125CF02E0ACF1:1546214400000:L1JlsOJwk5weSux3NHNHNhLWkI
I get the following error (requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)):
INFO: Running Agent Registration Process, CATTLE_URL=http://10.10.18.35:8080/v1 INFO: Attempting to connect to: http://10.10.18.35:8080/v1 INFO: http://10.10.18.35:8080/v1 is accessible INFO: Inspecting host capabilities INFO: Boot2Docker: false INFO: Host writable: true INFO: Token: xxxxxxxx INFO: Running registration Traceback (most recent call last): File "./register.py", line 11, in <module>
secret_key=os.environ['CATTLE_REGISTRATION_SECRET_KEY']) File "/usr/local/lib/python2.7/dist-packages/cattle.py", line 45, in from_env
return gdapi.from_env(prefix=prefix, factory=Client, **kw) File "/usr/local/lib/python2.7/dist-packages/gdapi.py", line 613, in from_env
return _from_env(prefix=prefix, factory=factory, **args) File "/usr/local/lib/python2.7/dist-packages/gdapi.py", line 632, in
_from_env
return factory(**result) File "/usr/local/lib/python2.7/dist-packages/cattle.py", line 12, in
__init__
super(Client, self).__init__(*args, **kw) File "/usr/local/lib/python2.7/dist-packages/gdapi.py", line 197, in
__init__
self._load_schemas() File "/usr/local/lib/python2.7/dist-packages/gdapi.py", line 315, in
_load_schemas
response = self._get_response(self._url) File "/usr/local/lib/python2.7/dist-packages/gdapi.py", line 264, in
_get_response
headers=self._headers) File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 501, in get
return self.request('GET', url, **kwargs) File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 609, in send
r = adapter.send(request, **kwargs) File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 497, in send
raise SSLError(e, request=request) requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
Any clues to resolve this please
It is required to access Rancher via ssl. I see you’re using http instead. To generate a self-signed cert, follow this documentation; https://rancher.com/docs/rancher/v2.x/en/installation/single-node-install/#option-a-default-self-signed-certificate

Resources