I am using use the ice command line interface for IBM Container Services, and I am seeing a couple of different problems from a couple of different boxes I am testing with. Here is one example:
[root#cds-legacy-monitor ~]# ice --verbose login --org chrisr#ca.ibm.com --space dev --user chrisr#ca.ibm.com --registry registry-ice.ng.bluemix.net
#2015-11-26 01:38:26.092288 - Namespace(api_key=None, api_url=None, cf=False, cloud=False, host=None, local=False, org='chrisr#ca.ibm.com', psswd=None, reg_host='registry-ice.ng.bluemix.net', skip_docker=False, space='dev', subparser_name='login', user='chrisr#ca.ibm.com', verbose=True)
#2015-11-26 01:38:26.092417 - Executing: cf login -u chrisr#ca.ibm.com -o chrisr#ca.ibm.com -s dev -a https://api.ng.bluemix.net
API endpoint: https://api.ng.bluemix.net`
Password>
Authenticating...
OK
Targeted org chrisr#ca.ibm.com
Targeted space dev
API endpoint: https://api.ng.bluemix.net (API version: 2.40.0)
User: chrisr#ca.ibm.com
Org: chrisr#ca.ibm.com
Space: dev
#2015-11-26 01:38:32.186204 - cf exit level: 0
#2015-11-26 01:38:32.186340 - config.json path: /root/.cf/config.json
#2015-11-26 01:38:32.186640 - Bearer: <long string omitted>
#2015-11-26 01:38:32.186697 - cf login succeeded. Can access: https://api-ice.ng.bluemix.net/v3/containers
Authentication with container cloud service at https://api-ice.ng.bluemix.net/v3/containers completed successfully
You can issue commands now to the container service
Proceeding to authenticate with the container cloud registry at registry-ice.ng.bluemix.net
#2015-11-26 01:38:32.187317 - using bearer token
#2015-11-26 01:38:32.187350 - config.json path: /root/.cf/config.json
#2015-11-26 01:38:32.187489 - Bearer: <long pw string omitted>
#2015-11-26 01:38:32.187517 - Org Guid: dae00d7c-1c3d-4bfd-a207-57a35a2fb42b
#2015-11-26 01:38:32.187551 - docker login -u bearer -p '<long pw string omitted>' -e a#b.c registry-ice.ng.bluemix.net
FATA[0012] Error response from daemon: </html>
#2015-11-26 01:38:44.689721 - docker call exit level: 256
docker is not available on your system or is not properly configured
Could not authenticate with cloud registry at registry-ice.ng.bluemix.net
You can still use IBM Containers but will not be able to run local docker containers, push, or pull images
#2015-11-26 01:38:44.689842 - Exit err level = 2**
On the other box, it also fails, but the final error is slightly different.
#2015-11-26 01:44:48.916034 - docker login -u bearer -p '<long pw string omitted>' -e a#b.c registry-ice.ng.bluemix.net
Error response from daemon: Unexpected status code [502] : <html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>
#2015-11-26 01:45:02.582753 - docker call exit level: 256
docker is not available on your system or is not properly configured
Could not authenticate with cloud registry at registry-ice.ng.bluemix.net
You can still use IBM Containers but will not be able to run local docker containers, push, or pull images
#2015-11-26 01:45:02.582868 - Exit err level = 2
Any thoughts on what might be causing these issues?
The errors are referring the same problem, ice isn't finding any docker env locally.
It doesn't prevent working remotely on Bluemix but without a local docker env ice cannot work with local containers
Related
I'm facing the same problem as here - I have set up a private Docker Registry with TLS certification (certificates generated via Certbot), and I can interact with it directly via curl etc. (thus proving that the certificate is correct), but the Docker Plugin in my Drone flow gives an error x509: certificate signed by unknown authority.
As per this StackOverflow answer, I believe that putting the certificate at /etc/docker/certs.d/<my_registry_address:port>/ca.crt should fix this problem, but it doesn't appear to (neither does adding the certificate into the standard /etc/ssl/certs/ca-certificates.crt location)
Demonstration that the certificates work as-expected, having already built the Docker Drone Plugin locally as per https://github.com/drone-plugins/drone-docker:
$ docker run --rm -v <path_to_directory_containing_pems>:/custom-certs -it --entrypoint /bin/sh plugins/docker
/ # ls /custom-certs
accounts archive csr keys live renewal renewal-hooks
/ # apk add curl
...
OK: 28 MiB in 56 packages
/ # curl https://docker-registry.scubbo.org:8843/v2/_catalog
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
/ # curl https://docker-registry.scubbo.org:8843/v2/_catalog --cacert /custom-certs/live/docker-registry.scubbo.org/fullchain.pem
{"repositories":[...]}
/ # cat /custom-certs/live/docker-registry.scubbo.org/fullchain.pem >> /etc/ssl/certs/ca-certificates.crt
/ # curl https://docker-registry.scubbo.org:8843/v2/_catalog
{"repositories":[...]}
Here's my .drone.yml, for a Runner instantiated with --env=DRONE_RUNNER_VOLUMES=/var/run/docker.sock:/var/run/docker.sock,<path_to_directory_containing_pems>:/custom-certs:
kind: pipeline
name: hello-world
type: docker
platform:
os: linux
arch: arm64
steps:
- name: copy-cert-into-place
image: busybox
volumes:
- name: docker-cert-persistence
path: /etc/docker/certs.d/
commands:
# https://stackoverflow.com/a/56410355/1040915
# Note that we need to mount the whole `custom-certs` directory into the workflow and then copy the file to `/etc/...`,
# rather than mounting the file directly into `/etc/...`, because the original file is a symlink and it's not possible (AFAIK)
# to instruct Docker to "mount the eventual-target-of this symlink into <location>"
- mkdir -p /etc/docker/certs.d/docker-registry.scubbo.org:8843
- cp -L /custom-certs/live/docker-registry.scubbo.org/fullchain.pem /etc/docker/certs.d/docker-registry.scubbo.org:8843/ca.crt
- name: check-cert-persists-between-stages
image: alpine
volumes:
- name: docker-cert-persistence
path: /etc/docker/certs.d/
commands:
- apk add curl
# The command below would fail if the cert was unavailable or invalid
- curl https://docker-registry.scubbo.org:8843/v2/_catalog --cacert /etc/docker/certs.d/docker-registry.scubbo.org:8843/ca.crt
- name: build-image
# ...contents irrelevant to this question...
- name: push-built-image
image: plugins/docker
volumes:
- name: docker-cert-persistence
path: /etc/docker/certs.d/
settings:
repo: docker-registry.scubbo.org:8843/scubbo/blog_nginx
tags: built_in_ci
debug: true
launch_debug: true
volumes:
- name: docker-cert-persistence
temp: {}
giving these logs from push-built-image step - ending in...
+ /usr/local/bin/docker tag 472d41d9c03ee60fe9c1965ad9cfd36a1cdb6cbf docker-registry.scubbo.org:8843/scubbo/blog_nginx:built_in_ci
+ /usr/local/bin/docker push docker-registry.scubbo.org:8843/scubbo/blog_nginx:built_in_ci
The push refers to repository [docker-registry.scubbo.org:8843/scubbo/blog_nginx]
Get "https://docker-registry.scubbo.org:8843/v2/": x509: certificate signed by unknown authority
exit status 1
How should I go about providing the CA Certificate to my Drone Docker Plugin step to permit it to communicate over TLS with a secure Docker registry? This answer suggests simply reverting to insecure integration, which works but is unsatisfactory.
EDIT: After re-reading this documentation, I extended the copy-cert-into-place commands to copy all 3 certificate-related files:
commands:
- mkdir -p /etc/docker/certs.d/docker-registry.scubbo.org:8843
- cp -L /custom-certs/live/docker-registry.scubbo.org/fullchain.pem /etc/docker/certs.d/docker-registry.scubbo.org:8843/ca.crt
- cp -L /custom-certs/live/docker-registry.scubbo.org/privkey.pem /etc/docker/certs.d/docker-registry.scubbo.org:8843/client.key
- cp -L /custom-certs/live/docker-registry.scubbo.org/cert.pem /etc/docker/certs.d/docker-registry.scubbo.org:8843/client.cert
but that did not resolve the problem - same x509: certificate signed by unknown authority error.
EDIT2: I directly confirmed (directly on a host, outside the context of a plugin or docker container) that adding the certificate to the path used above is sufficient to permit interaction with the registry:
$ docker pull docker-registry.scubbo.org:8843/scubbo/blog_nginx:built_in_ci
Error response from daemon: Get "https://docker-registry.scubbo.org:8843/v2/": x509: certificate signed by unknown authority
$ sudo cp -L <path_to_directory_containing_pems>/live/docker-registry.scubbo.org/chain.pem /etc/docker/certs.d/docker-registry.scubbo.org\:8843/ca.crt
$ docker pull docker-registry.scubbo.org:8843/scubbo/blog_nginx:built_in_ci
built_in_ci: Pulling from scubbo/blog_nginx
Digest: sha256:3a17f86f23050303d94443f24318b49fb1a5e2d0cc9228270678c8aa55b4d2c2
Status: Image is up to date for docker-registry.scubbo.org:8843/scubbo/blog_nginx:built_in_ci
docker-registry.scubbo.org:8843/scubbo/blog_nginx:built_in_ci
This isn't a complete answer, but I was able to get secure registry access working by switching from mounting a directory, to mounting the file directly:
I changed the docker run option to --env=DRONE_RUNNER_VOLUMES=/var/run/docker.sock:/var/run/docker.sock,$(readlink -f <path_to_directory_containing_pems>/live/docker-registry.scubbo.org/chain.pem):/registry_cert.crt
I changed the commands in copy-cert-into-place to:
- mkdir -p /etc/docker/certs.d/docker-registry.scubbo.org:8843
- cp /registry_cert.crt /etc/docker/certs.d/docker-registry.scubbo.org:8843/ca.crt
I don't consider this a complete answer (and would love further input or advice!), because:
I don't know why copying the file out of the mounted directory into /etc/docker/... (as in the original question) didn't work, but mounting the file directly from the host filesystem worked. (Note that the check-cert-persists-between-stages stage confirms that the certificate is correct, so it's not a mistake of copying a wrong or empty file)
I don't know how to mount the file directly into an in-stage path that contains a colon - this answer indicates how to mount a path containing a colon directly into a container, but in this case we're passing the path to DRONE_RUNNER_VOLUMES
Please am trying to deploy an image in Azure Container Registry(ACR) on my Function App, but am not able to do it.
I pushed the latest image from pc to the ACR after creating it. Also the admin under access key is enable. Please advise how to resolve this. The log result can be found below.
Please use https://aka.ms/linux-diagnostics to enable logging to see container logs here.
2020-11-02T15:05:03.452Z INFO - Pulling image: myacrqa.azurecr.io/myimage:v1
2020-11-02T15:05:03.462Z ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://ifckpacrqa.azurecr.io/v2/: read tcp IP:17045->IP:443: read: connection reset by peer"}
2020-11-02T15:05:03.462Z ERROR - Pulling docker image myacrqa.azurecr.io/offlinekpqa:v1 failed:
2020-11-02T15:05:03.462Z INFO - Pulling image from Docker hub: myacrqa.azurecr.io/offlinekpqa:v1
2020-11-02T15:05:03.470Z ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://ifckpacrqa.azurecr.io/v2/: read tcp IP:17047->IP:443: read: connection reset by peer"}
2020-11-02T15:05:03.471Z ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
2020-11-02T15:05:08.847Z INFO - Stopping site ifc-kp-ml-qa because it failed during startup.
2020-11-02T15:10:01.023Z INFO - Starting container for site
2020-11-02T15:10:01.024Z INFO - docker run -d -p 8356:8081 --name ifc-kp-func_app_0_22558c6b_msiProxy -e WEBSITE_CORS_ALLOWED_ORIGINS=https://functions.azure.com,https://functions-staging.azure.com,https://functions-next.azure.com,https://storage.z13.web.core.windows.net -e WEBSITE_CORS_SUPPORT_CREDENTIALS=False -e WEBSITES_ENABLE_APP_SERVICE_STORAGE=false -e WEBSITE_SITE_NAME=IFC-KP-ML-QA -e WEBSITE_AUTH_ENABLED=True -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=xxxx -e WEBSITE_INSTANCE_ID=65631c3af46c684539e2d9f55e37247be307daaa00f59cdf3231284117e30b40 appsvc/msitokenservice:2007200210
2020-11-02T15:10:01.025Z INFO - Logging is not enabled for this container.
Please use https://aka.ms/linux-diagnostics to enable logging to see container logs here.
2020-11-02T15:10:04.362Z INFO - Pulling image: ifckpacrqa.azurecr.io/offlinekpqa:v1
2020-11-02T15:10:04.372Z ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://myacrqa.azurecr.io/v2/: read tcp IP:17489->IP:443: read: connection reset by peer"}
2020-11-02T15:10:04.373Z ERROR - Pulling docker image myacrqa.azurecr.io/myimage:v1 failed:
2020-11-02T15:10:04.373Z INFO - Pulling image from Docker hub: myacrqa.azurecr.io/myimage:v1
2020-11-02T15:10:04.398Z ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://ifckpacrqa.azurecr.io/v2/: read tcp 10.168.216.12:17491->52.168.114.2:443: read: connection reset by peer"}
2020-11-02T15:10:04.401Z ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
2020-11-02T15:10:09.853Z INFO - Stopping site ifc-kp-ml-qa because it failed during startup.
2020-11-02T15:15:02.120Z INFO - Starting container for site
2020-11-02T15:15:02.121Z INFO - docker run -d -p 7603:8081 --name ifc-kp-ml-qa_0_969b061e_msiProxy -e WEBSITE_CORS_ALLOWED_ORIGINS=https://functions.azure.com,https://functions-staging.azure.com,https://functions-next.azure.com,https://storage.z13.web.core.windows.net -e WEBSITE_CORS_SUPPORT_CREDENTIALS=False -e WEBSITES_ENABLE_APP_SERVICE_STORAGE=false -e WEBSITE_SITE_NAME=IFC-KP-ML-QA -e WEBSITE_AUTH_ENABLED=True -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=ifc-kp-ml-qa.aseqa.ifc.org -e WEBSITE_INSTANCE_ID=65631c3af46c684539e2d9f55e37247be307daaa00f59cdf3231284117e30b40 appsvc/msitokenservice:2007200210
2020-11-02T15:15:02.122Z INFO - Logging is not enabled for this container.
Please use https://aka.ms/linux-diagnostics to enable logging to see container logs here.
2020-11-02T15:15:04.469Z INFO - Pulling image: myacrqa.azurecr.io/myimage:v1
2020-11-02T15:15:04.479Z ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://myacrqa.azurecr.io/v2/: read tcp IP:17953->IP:443: read: connection reset by peer"}
2020-11-02T15:15:04.479Z ERROR - Pulling docker image myacrqa.azurecr.io/myimage:v1 failed:
2020-11-02T15:15:04.479Z INFO - Pulling image from Docker hub: myacrqa.azurecr.io/myimage:v1
2020-11-02T15:15:04.487Z ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://myacrqa.azurecr.io/v2/: read tcp IP:17955->IP:443: read: connection reset by peer"}
2020-11-02T15:15:04.490Z ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
2020-11-02T15:15:09.926Z INFO - Stopping site ifc-kp-ml-qa because it failed during startup.
To pull the docker images from the ACR or other private registry, you need to set the environment variables like this:
You can set these environment variables in the Azure portal, function settings. Or use the Azure CLI command az functionapp create with the paramteres:
--deployment-container-image-name
--docker-registry-server-password
--docker-registry-server-user
actually I'm using gitlab runners, with docker executor, and I'm trying to pull some docker images to do some tests, and to preserve my network connection, I've created a private docker registry, to "cache" the images .
So, my registry is linked to my gitlab runner (with configuration in the config.toml https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnersdocker-section ) .
This work, my image can ask the registry :
$ wget http://registry:5000/v2/_catalog
--2019-02-15 10:40:54-- http://registry:5000/v2/_catalog
Resolving registry... 172.17.0.3
Connecting to registry|172.17.0.3|:5000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 20 [application/json]
Saving to: '_catalog'
0K 100% 1.17M=0s
2019-02-15 10:40:54 (1.17 MB/s) - '_catalog' saved [20/20]
but the DIND service can't :
pull registry:5000/arminc/clair-db:latest
Error response from daemon: Get http://registry:5000/v2/: dial tcp: lookup registry on 192.168.9.254:53: no such host
My gitlab-ci conf for this task
scan:image:
stage: scans
image: docker:git
services:
- name: docker:dind
command: ["--insecure-registry=registry:5000"]
variables:
DOCKER_DRIVER: overlay2
allow_failure: true
script:
- chmod 777 ./docker/scan.sh
- docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD $DOCKER_REGISTRY
- ./docker/scan.sh
artifacts:
paths: [gl-container-scanning-report.json]
only:
- master
Probably, you might need to add a DNS entry to your DNS server or dockers host file:
192.168.xx.xxx registry
I use an ansible script to load & start the https://hub.docker.com/r/rastasheep/ubuntu-sshd/ container.
so it starts well of course :
bash-4.4$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8bedbd3b7d88 rastasheep/ubuntu-sshd "/usr/sbin/sshd -D" 37 minutes ago Up 36 minutes 0.0.0.0:49154->22/tcp test
bash-4.4$
so after ansible failure on ssh access to it I tested manually from shell
this is also ok.
bash-4.4$ ssh root#172.17.0.2
The authenticity of host '172.17.0.2 (172.17.0.2)' can't be established.
ECDSA key fingerprint is SHA256:YtTfuoRRR5qStSVA5UuznGamA/dvf+djbIT6Y48IYD0.
ECDSA key fingerprint is MD5:43:3f:41:e9:89:45:06:6f:f6:42:c4:6a:70:37:f8:1d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.17.0.2' (ECDSA) to the list of known hosts.
root#172.17.0.2's password:
root#8bedbd3b7d88:~# logout
Connection to 172.17.0.2 closed.
bash-4.4$
so the step that failed is trying to get on it from ansible script & make access to ssh-copy-id
ansible error message is :
Fatal: [172.17.0.2]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '172.17.0.2' (ECDSA) to the list of known hosts.\r\nPermission denied (publickey,password).\r\n", "unreachable": true}
---
- hosts: 127.0.0.1
tasks:
- name: start docker service
service:
name: docker
state: started
- name: load and start the container we wanna use
docker_container:
name: test
image: rastasheep/ubuntu-sshd
state: started
ports:
- "49154:22"
- name: Wait maximum of 300 seconds for ports to be available
wait_for:
host: 0.0.0.0
port: 49154
state: started
- hosts: 172.17.0.2
vars:
passwordadmin: $6$pbE6yznA$AeFIdI.....K0
passwordroot: $6$TMrxQUxT$I8.JIzR.....TV1
ansible_ssh_extra_args: "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
tasks:
- name: Build test container root user rsa ssh-key
shell: docker exec test ssh-keygen -b 2048 -t rsa -f /root/.ssh/id_rsa -q -N ""
so I cannot even run the needed step to build ssh
how to do then ??
1st step (ansible task) : load docker container
2cd step (ansible task on only 172.17.0.2) : connect to it & setup it
there will be 3rd step to run application on it after that.
the problem occurs only when starting the 2cd step
Ok after many trys on a second container
conclusion is my procedure was bad
what I have done to solve that :
build a diroctory tree separating ./ ./inventory ./includes
build 1 yaml file by host (local, docker, labo)
build 1 main yaml file on ./
build 1 new host file in ./inventory
connect forced by sshpass to docker on default password
changed it
add the host key on authorized key to a login dedicated usage
installed pyhton (needed to answer ansible host else it makes
randomly module errors or refused connections depending on current
action)
setup a ssh login user in sudoers
then I can un the docker.yaml actions
then only at last I can run the labo.yaml actions.
Thanks for help
now I'm able to build the missing tools.
I've successfully logged in bluemix container service via command ice login with following output:
Authentication with container cloud service at https://api-ice.ng.bluemix.net/v1.0/containers completed successfully
You can issue commands now to the container service
Proceeding to authenticate with the container cloud registry at registry-ice.ng.bluemix.net
Login Succeeded
And ice ps -a works as well, but when issuing ice images, it failed with:
$ ice --verbose images
#2015-05-07 13:59:29.221306 - Namespace(cloud=False, local=False, subparser_name='images', verbose=True)
#2015-05-07 13:59:29.221370 - request url: https://api-ice.ng.bluemix.net/v1.0/containers/images/json
#2015-05-07 13:59:30.012412 - Return code: 404 Return reason: NOT FOUND
#2015-05-07 13:59:30.012439 - Req-ID: a382f2f79d54b157
#2015-05-07 13:59:30.012451 - Exit err level = 1
Here's the command line version:
$ ice version
ICE CLI Version : 2.0.1 000 2015-03-26T19:51:27
Notice that ice images works last week.
is there anything changed in the server side?
Try login to ice with this
ice login -a https://api.ng.bluemix.net -H https://api-ice.ng.bluemix.net/v2/containers -R registry-ice.ng.bluemix.net
This is what I get when running ice --verbose images
bash-3.2$ ice --verbose images
#2015-05-08 14:54:49.692386 - Namespace(cloud=False, local=False, subparser_name='images', verbose=True)
#2015-05-08 14:54:49.692455 - request url: https://api-ice.ng.bluemix.net/v2/containers/images/json
#2015-05-08 14:54:49.692466 - using bearer token and space id
#2015-05-08 14:54:49.692482 - config.json path: /Users/stanli/.cf/config.json
It seems that your ice command was pointing to v1 of the api.
-Stan