Please am trying to deploy an image in Azure Container Registry(ACR) on my Function App, but am not able to do it.
I pushed the latest image from pc to the ACR after creating it. Also the admin under access key is enable. Please advise how to resolve this. The log result can be found below.
Please use https://aka.ms/linux-diagnostics to enable logging to see container logs here.
2020-11-02T15:05:03.452Z INFO - Pulling image: myacrqa.azurecr.io/myimage:v1
2020-11-02T15:05:03.462Z ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://ifckpacrqa.azurecr.io/v2/: read tcp IP:17045->IP:443: read: connection reset by peer"}
2020-11-02T15:05:03.462Z ERROR - Pulling docker image myacrqa.azurecr.io/offlinekpqa:v1 failed:
2020-11-02T15:05:03.462Z INFO - Pulling image from Docker hub: myacrqa.azurecr.io/offlinekpqa:v1
2020-11-02T15:05:03.470Z ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://ifckpacrqa.azurecr.io/v2/: read tcp IP:17047->IP:443: read: connection reset by peer"}
2020-11-02T15:05:03.471Z ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
2020-11-02T15:05:08.847Z INFO - Stopping site ifc-kp-ml-qa because it failed during startup.
2020-11-02T15:10:01.023Z INFO - Starting container for site
2020-11-02T15:10:01.024Z INFO - docker run -d -p 8356:8081 --name ifc-kp-func_app_0_22558c6b_msiProxy -e WEBSITE_CORS_ALLOWED_ORIGINS=https://functions.azure.com,https://functions-staging.azure.com,https://functions-next.azure.com,https://storage.z13.web.core.windows.net -e WEBSITE_CORS_SUPPORT_CREDENTIALS=False -e WEBSITES_ENABLE_APP_SERVICE_STORAGE=false -e WEBSITE_SITE_NAME=IFC-KP-ML-QA -e WEBSITE_AUTH_ENABLED=True -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=xxxx -e WEBSITE_INSTANCE_ID=65631c3af46c684539e2d9f55e37247be307daaa00f59cdf3231284117e30b40 appsvc/msitokenservice:2007200210
2020-11-02T15:10:01.025Z INFO - Logging is not enabled for this container.
Please use https://aka.ms/linux-diagnostics to enable logging to see container logs here.
2020-11-02T15:10:04.362Z INFO - Pulling image: ifckpacrqa.azurecr.io/offlinekpqa:v1
2020-11-02T15:10:04.372Z ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://myacrqa.azurecr.io/v2/: read tcp IP:17489->IP:443: read: connection reset by peer"}
2020-11-02T15:10:04.373Z ERROR - Pulling docker image myacrqa.azurecr.io/myimage:v1 failed:
2020-11-02T15:10:04.373Z INFO - Pulling image from Docker hub: myacrqa.azurecr.io/myimage:v1
2020-11-02T15:10:04.398Z ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://ifckpacrqa.azurecr.io/v2/: read tcp 10.168.216.12:17491->52.168.114.2:443: read: connection reset by peer"}
2020-11-02T15:10:04.401Z ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
2020-11-02T15:10:09.853Z INFO - Stopping site ifc-kp-ml-qa because it failed during startup.
2020-11-02T15:15:02.120Z INFO - Starting container for site
2020-11-02T15:15:02.121Z INFO - docker run -d -p 7603:8081 --name ifc-kp-ml-qa_0_969b061e_msiProxy -e WEBSITE_CORS_ALLOWED_ORIGINS=https://functions.azure.com,https://functions-staging.azure.com,https://functions-next.azure.com,https://storage.z13.web.core.windows.net -e WEBSITE_CORS_SUPPORT_CREDENTIALS=False -e WEBSITES_ENABLE_APP_SERVICE_STORAGE=false -e WEBSITE_SITE_NAME=IFC-KP-ML-QA -e WEBSITE_AUTH_ENABLED=True -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=ifc-kp-ml-qa.aseqa.ifc.org -e WEBSITE_INSTANCE_ID=65631c3af46c684539e2d9f55e37247be307daaa00f59cdf3231284117e30b40 appsvc/msitokenservice:2007200210
2020-11-02T15:15:02.122Z INFO - Logging is not enabled for this container.
Please use https://aka.ms/linux-diagnostics to enable logging to see container logs here.
2020-11-02T15:15:04.469Z INFO - Pulling image: myacrqa.azurecr.io/myimage:v1
2020-11-02T15:15:04.479Z ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://myacrqa.azurecr.io/v2/: read tcp IP:17953->IP:443: read: connection reset by peer"}
2020-11-02T15:15:04.479Z ERROR - Pulling docker image myacrqa.azurecr.io/myimage:v1 failed:
2020-11-02T15:15:04.479Z INFO - Pulling image from Docker hub: myacrqa.azurecr.io/myimage:v1
2020-11-02T15:15:04.487Z ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://myacrqa.azurecr.io/v2/: read tcp IP:17955->IP:443: read: connection reset by peer"}
2020-11-02T15:15:04.490Z ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
2020-11-02T15:15:09.926Z INFO - Stopping site ifc-kp-ml-qa because it failed during startup.
To pull the docker images from the ACR or other private registry, you need to set the environment variables like this:
You can set these environment variables in the Azure portal, function settings. Or use the Azure CLI command az functionapp create with the paramteres:
--deployment-container-image-name
--docker-registry-server-password
--docker-registry-server-user
Related
I am new to Jaeger and Kafka,
I am trying to use Kafka as intermediate buffer.
I am using OpenTelemetry to send data to Jaeger-Collector directly using -Dotel.exporter.jaeger.endpoint.
Jaeger-Collector is deployed on Kubernetes and the Kafka is on another network but is accessible. I can confirm that the traces are sent to Jaeger-collector.
On hitting the /metrics of collector and output tells me that spans were written successfully to Kafka.
jaeger_kafka_spans_written_total{status="success"} 21
The Collector logs indicate what topic I am writing to
{"Brokers":["myKafkaBroker......"}},"topic":"tp6"}
I want to get this (Span) data from Kafka Queue to ElasticSearch. To do this I am starting the Jaeger Ingester as follows
docker run -e "SPAN_STORAGE_TYPE=elasticsearch" jaegertracing/jaeger-ingester:1.22 --kafka.consumer.topic=tp6 --kafka.consumer.brokers='myKafkaBroker' --es.tls.skip-host-verify
But the container is stopped after fatal error
{"level":"fatal","ts":1615546463.7784193,"caller":"command-line-arguments/main.go:64","msg":"Failed to init storage factory","error":"failed to create primary Elasticsearch client: health check timeout: Head \"http://127.0.0.1:9200\": dial tcp 127.0.0.1:9200: connect: connection refused: no Elasticsearch node available","stacktrace":"main.main.func1\n\tcommand-line-arguments/main.go:64\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra#v0.0.7/command.go:838\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra#v0.0.7/command.go:943\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra#v0.0.7/command.go:883\nmain.main\n\tcommand-line-arguments/main.go:113\nruntime.main\n\truntime/proc.go:204"}
The elasticsearch and ingester are being run on the same machine using docker. The elasticsearch is running on docker using
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node"ocker.elastic.co/elasticsearch/elasticsearch:7.11.2
I have disabled TLS so that shouldn't be a problem. I am unable to get this to work.
I'm going to setup a local registry by following https://docs.docker.com/registry/deploying/.
docker run -d -p 5000:5000 --restart=always --name reg ubuntu:16.04
When I try to run the following command:
$ docker push localhost:5000/my-ubuntu
I get Error:
Get http://localhost:5000/v2/: dial tcp 127.0.0.1:5000: connect:connection refused
Any idea?
Connection refused usually means that the service you are trying to connect to isn't actually up and running like it should. There could be other reasons as outlined in this question, but essentially, for your case, it simply means that the registry is not up yet.
Wait for the registry container to be created properly before you do anything else - docker run -d -p 5000:5000 --restart=always --name registry registry:2 that creates a local registry from the official docker image.
Make sure that the registry container is up by running docker ps | grep registry, and then proceed further.
More comments about
Kubenetes(K8s) / Minikube
docker / image / registry, container
If you are using Minikube, and want to pull down an image from 127.0.0.1:5000,
then you meet the errors below:
Failed to pull image "127.0.0.1:5000/nginx_operator:latest": rpc error: code = Unknown desc = Error response from daemon: Get http://127.0.0.1:5000/v2/: dial tcp 127.0.0.1:5000: connect: connection refused
Full logs:
$ kubectl describe pod/your_pod
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m29s default-scheduler Successfully assigned tj-blue-whale-05-system/tj-blue-whale-05-controller-manager-6c8f564575-kwxdv to minikube
Normal Pulled 2m25s kubelet Container image "gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0" already present on machine
Normal Created 2m24s kubelet Created container kube-rbac-proxy
Normal Started 2m23s kubelet Started container kube-rbac-proxy
Normal BackOff 62s (x5 over 2m22s) kubelet Back-off pulling image "127.0.0.1:5000/nginx_operator:latest"
Warning Failed 62s (x5 over 2m22s) kubelet Error: ImagePullBackOff
Normal Pulling 48s (x4 over 2m23s) kubelet Pulling image "127.0.0.1:5000/nginx_operator:latest"
Warning Failed 48s (x4 over 2m23s) kubelet Failed to pull image "127.0.0.1:5000/nginx_operator:latest": rpc error: code = Unknown desc = Error response from daemon: Get http://127.0.0.1:5000/v2/: dial tcp 127.0.0.1:5000: connect: connection refused
Warning Failed 48s (x4 over 2m23s) kubelet Error: ErrImagePull
Possible root cause:
The registry must be setup inside the Minikube side instead of your host side.
i.e.
host: registry (127.0.0.1:5000)
minikube: no registry (the K8s could not find your image)
How to check?
Step1: check your Minikube container
$ docker ps -a
CONTAINER ID IMAGE ... STATUS PORTS NAMES
8c6f49491dd6 gcr.io/k8s-minikube/kicbase:v0.0.15-snapshot4 ... Up 15 hours 127.0.0.1:49156->22/tcp, 127.0.0.1:49155->2376/tcp, 127.0.0.1:49154->5000/tcp, 127.0.0.1:49153->8443/tcp minikube
# your Minikube is under running
# host:49154 <--> minikube:5000
# where:
# - port 49154 was allocated randomly by the docker service
# - port 22: for ssh
# - port 2376: for docker service
# - port 5000: for registry (image repository)
# - port 8443: for Kubernetes
Step2: login to your Minikube
$ minikube ssh
docker#minikube:~$ curl 127.0.0.1:5000
curl: (7) Failed to connect to 127.0.0.1 port 5000: Connection refused
# setup
# =====
# You did not setup the registry.
# Let's try to setup it.
docker#minikube:~$ docker run --restart=always -d -p 5000:5000 --name registry registry:2
# test
# ====
# test the registry using the following commands
docker#minikube:~$ curl 127.0.0.1:5000
docker#minikube:~$ curl 127.0.0.1:5000/v2
Moved Permanently.
docker#minikube:~$ curl 127.0.0.1:5000/v2/_catalog
{"repositories":[]}
# it's successful
docker#minikube:~$ exit
logout
Step3: build your image, and push it into the registry of your Minikube
# Let's take nginx as an example. (You can build your own image)
$ docker pull nginx
# modify the repository (the source and the name)
$ docker tag nginx 127.0.0.1:49154/nginx_operator
# check the new repository (source and the name)
$ docker images | grep nginx
REPOSITORY TAG IMAGE ID CREATED SIZE
127.0.0.1:49154/nginx_operator latest ae2feff98a0c 3 weeks ago 133MB
# push the image into the registry of your Minikube
$ docker push 127.0.0.1:49154/nginx_operator
Step4: login to your Minikube again
$ minikube ssh
# check the registry
$ curl 127.0.0.1:5000/v2/_catalog
{"repositories":["nginx_operator"]}
# it's successful
# get the image info
$ curl 127.0.0.1:5000/v2/nginx_operator/manifests/latest
docker#minikube:~$ exit
logout
Customize exposed ports of Minikube
if you would like to use the port 5000 on the host side instead of using 49154 (which was allocated randomly by the docker service)
i.e.
host:5000 <--> minikube:5000
you need to recreate a minikube instance with the flag --ports
# delete the old minikube instance
$ minkube delete
# create a new one (with the docker driver)
$ minikube start --ports=5000:5000 --driver=docker
# or
$ minikube start --ports=127.0.0.1:5000:5000 --driver=docker
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5d1e5b61a3bf gcr.io/k8s-minikube/kicbase:v0.0.15-snapshot4 "/usr/local/bin/entr…" About a minute ago Up About a minute 0.0.0.0:5000->5000/tcp, 127.0.0.1:49162->22/tcp, 127.0.0.1:49161->2376/tcp, 127.0.0.1:49160->5000/tcp, 127.0.0.1:49159->8443/tcp minikube
$ docker port minikube
22/tcp -> 127.0.0.1:49162
2376/tcp -> 127.0.0.1:49161
5000/tcp -> 127.0.0.1:49160
5000/tcp -> 0.0.0.0:5000
8443/tcp -> 127.0.0.1:49159
you can see: 0.0.0.0:5000->5000/tcp
Re-test your registry in the Minikube
# in the host side
$ docker pull nginx
$ docker tag nginx 127.0.0.1:5000/nginx_operator
$ docker ps -a
$ docker push 127.0.0.1:5000/nginx_operator
$ minikube ssh
docker#minikube:~$ curl 127.0.0.1:5000/v2/_catalog
{"repositories":["nginx_operator"]}
# Great!
I've an Openshift Origin cluster running offline on 3 Centos 7 vm. It's working fine, I've a registry where I push my images like this :
docker login -u <username> -e <any_email_address> -p <token_value> <registry_ip>:<port>
Login is successful, then :
oc tag <image-id> <docker-registry-IP>:<port>/<project-name>/<image>
So, for nginx for example :
oc tag 49011ce3b713 172.30.222.111:5000/test/nginx
Then I push it to the internal registry :
docker push 172.30.222.111:5000/test/nginx
And finaly :
oc new-app nginx --name="nginx"
With nginx, everything is working fine, now my problem :
I'm actually wanting to put Zookeeper on it, so : I do the same steps than above, I also install "jboss/base-jdk:7" which is a dependancy of Zookeeper, problem is :
docker push 172.30.222.111:5000/test/jboss/base-jdk:7
Giving :
[root#master 994089]# docker push 172.30.222.111:5000/test/jboss/base-jdk:7
The push refers to a repository [172.30.222.111:5000/test/jboss/base-jdk]
c4c6a9114a05: Layer already exists
3bf2c105669b: Layer already exists
85c6e373d858: Layer already exists
dc1e2dcdc7b6: Layer already exists
Received unexpected HTTP status: 500 Internal Server Error
The problem seems to be the "/" here jboss**/**base-jdk:7
I also tried to push just like this :
docker push 172.30.222.111:5000/test/base-jdk:7
This is working , but Zookeeper is looking for exactly "jboss/base-jdk:7", and not just "base-jdk:7"
Finally, I'm blocked here, when trying this command : oc new-app zookeeper --name="zookeeper" --loglevel=8 --insecure-registry --allow-missing-images
I0628 14:31:54.009713 53407 dockerimagelookup.go:92] checking local Docker daemon for "jboss/base-jdk:7"
I0628 14:31:54.030546 53407 dockerimagelookup.go:380] partial match on "172.30.222.111:5000/test/base-jdk:7" with 0.375000
I0628 14:31:54.030571 53407 dockerimagelookup.go:346] exact match on "jboss/base-jdk:7"
I0628 14:31:54.030578 53407 dockerimagelookup.go:107] Found local docker image match "172.30.222.111:5000/test/base-jdk:7" with score 0.375000
I0628 14:31:54.030589 53407 dockerimagelookup.go:107] Found local docker image match "jboss/base-jdk:7" with score 0.000000
I0628 14:31:54.032799 53407 componentresolvers.go:59] Error from resolver: [can't look up Docker image "jboss/base-jdk:7": Internal error occurred: Get http://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.253.158.90:53: no such host]
I0628 14:31:54.032831 53407 dockerimagelookup.go:169] Added missing image match for jboss/base-jdk:7
F0628 14:31:54.032882 53407 helpers.go:110] error: can't look up Docker image "jboss/base-jdk:7": Internal error occurred: Get http://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.253.158.90:53: no such host
We can see that 172.30.222.111:5000/test/base-jdk:7 is found but it's not exactly what the command is looking for so it doesn't use it...
So, if you have any idea how to solve this ! :)
Resolved by upgrading to Openshift 1.5.1, previous was 1.3.1.
I am using use the ice command line interface for IBM Container Services, and I am seeing a couple of different problems from a couple of different boxes I am testing with. Here is one example:
[root#cds-legacy-monitor ~]# ice --verbose login --org chrisr#ca.ibm.com --space dev --user chrisr#ca.ibm.com --registry registry-ice.ng.bluemix.net
#2015-11-26 01:38:26.092288 - Namespace(api_key=None, api_url=None, cf=False, cloud=False, host=None, local=False, org='chrisr#ca.ibm.com', psswd=None, reg_host='registry-ice.ng.bluemix.net', skip_docker=False, space='dev', subparser_name='login', user='chrisr#ca.ibm.com', verbose=True)
#2015-11-26 01:38:26.092417 - Executing: cf login -u chrisr#ca.ibm.com -o chrisr#ca.ibm.com -s dev -a https://api.ng.bluemix.net
API endpoint: https://api.ng.bluemix.net`
Password>
Authenticating...
OK
Targeted org chrisr#ca.ibm.com
Targeted space dev
API endpoint: https://api.ng.bluemix.net (API version: 2.40.0)
User: chrisr#ca.ibm.com
Org: chrisr#ca.ibm.com
Space: dev
#2015-11-26 01:38:32.186204 - cf exit level: 0
#2015-11-26 01:38:32.186340 - config.json path: /root/.cf/config.json
#2015-11-26 01:38:32.186640 - Bearer: <long string omitted>
#2015-11-26 01:38:32.186697 - cf login succeeded. Can access: https://api-ice.ng.bluemix.net/v3/containers
Authentication with container cloud service at https://api-ice.ng.bluemix.net/v3/containers completed successfully
You can issue commands now to the container service
Proceeding to authenticate with the container cloud registry at registry-ice.ng.bluemix.net
#2015-11-26 01:38:32.187317 - using bearer token
#2015-11-26 01:38:32.187350 - config.json path: /root/.cf/config.json
#2015-11-26 01:38:32.187489 - Bearer: <long pw string omitted>
#2015-11-26 01:38:32.187517 - Org Guid: dae00d7c-1c3d-4bfd-a207-57a35a2fb42b
#2015-11-26 01:38:32.187551 - docker login -u bearer -p '<long pw string omitted>' -e a#b.c registry-ice.ng.bluemix.net
FATA[0012] Error response from daemon: </html>
#2015-11-26 01:38:44.689721 - docker call exit level: 256
docker is not available on your system or is not properly configured
Could not authenticate with cloud registry at registry-ice.ng.bluemix.net
You can still use IBM Containers but will not be able to run local docker containers, push, or pull images
#2015-11-26 01:38:44.689842 - Exit err level = 2**
On the other box, it also fails, but the final error is slightly different.
#2015-11-26 01:44:48.916034 - docker login -u bearer -p '<long pw string omitted>' -e a#b.c registry-ice.ng.bluemix.net
Error response from daemon: Unexpected status code [502] : <html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>
#2015-11-26 01:45:02.582753 - docker call exit level: 256
docker is not available on your system or is not properly configured
Could not authenticate with cloud registry at registry-ice.ng.bluemix.net
You can still use IBM Containers but will not be able to run local docker containers, push, or pull images
#2015-11-26 01:45:02.582868 - Exit err level = 2
Any thoughts on what might be causing these issues?
The errors are referring the same problem, ice isn't finding any docker env locally.
It doesn't prevent working remotely on Bluemix but without a local docker env ice cannot work with local containers
I've successfully logged in bluemix container service via command ice login with following output:
Authentication with container cloud service at https://api-ice.ng.bluemix.net/v1.0/containers completed successfully
You can issue commands now to the container service
Proceeding to authenticate with the container cloud registry at registry-ice.ng.bluemix.net
Login Succeeded
And ice ps -a works as well, but when issuing ice images, it failed with:
$ ice --verbose images
#2015-05-07 13:59:29.221306 - Namespace(cloud=False, local=False, subparser_name='images', verbose=True)
#2015-05-07 13:59:29.221370 - request url: https://api-ice.ng.bluemix.net/v1.0/containers/images/json
#2015-05-07 13:59:30.012412 - Return code: 404 Return reason: NOT FOUND
#2015-05-07 13:59:30.012439 - Req-ID: a382f2f79d54b157
#2015-05-07 13:59:30.012451 - Exit err level = 1
Here's the command line version:
$ ice version
ICE CLI Version : 2.0.1 000 2015-03-26T19:51:27
Notice that ice images works last week.
is there anything changed in the server side?
Try login to ice with this
ice login -a https://api.ng.bluemix.net -H https://api-ice.ng.bluemix.net/v2/containers -R registry-ice.ng.bluemix.net
This is what I get when running ice --verbose images
bash-3.2$ ice --verbose images
#2015-05-08 14:54:49.692386 - Namespace(cloud=False, local=False, subparser_name='images', verbose=True)
#2015-05-08 14:54:49.692455 - request url: https://api-ice.ng.bluemix.net/v2/containers/images/json
#2015-05-08 14:54:49.692466 - using bearer token and space id
#2015-05-08 14:54:49.692482 - config.json path: /Users/stanli/.cf/config.json
It seems that your ice command was pointing to v1 of the api.
-Stan