I'm attempting to use the circleci cli tool to test my build configurations locally but whatever I do I keep getting the error:
$ circleci local execute --job test
Fetching latest build environment...
Docker image digest: sha256:0d6c17bb09da8707eabad913d61a68b28028906d48c176d6bd60ebd36cce222e
====>> Spin up environment
Build-agent version ()
System information:
Server Version: 20.10.20
Storage Driver: overlay2
Backing Filesystem: extfs
Cgroup Driver: systemd
Cgroup Version: 2
Kernel Version: 5.15.74-1-lts
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
Error:
Unexpected environment preparation error: error connecting build-agent to ephemeral network: error connecting build-agent container to ephemeral network: Error response from daemon: invalid name or ID supplied: ""
Step failed
Task failed
Error: Unhandled prepare executor error: error connecting build-agent to ephemeral network: error connecting build-agent container to ephemeral network: Error response from daemon: invalid name or ID supplied: ""
I get this error for any job in any project on my machine indicating that it's something wrong with my system not the project. I can even run these commands fine on other machines. It something wrong with this machine but I have no idea what this error means. "invalid name or ID" to what? What is the thing with name or ID ""?
Update
Checking the docker systemctl logs I can see where circleci is getting the error:
Oct 27 13:36:46 myhost dockerd[1238]: time="2022-10-27T13:36:46.455536188-05:00" level=error msg="Handler for POST /v1.41/networks/a878ce01450ef752f4610bca358d4bf6b590385f1300ddac605b475ed8536985/disconnect returned error: invalid name: "
Oct 27 13:36:46 myhost dockerd[1238]: time="2022-10-27T13:36:46.763284410-05:00" level=info msg="ignoring event" container=5276cac204527a25344183be31abef0592a91ce6c36c2160e16d4d5d59bde7c3 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
so it looks like an error when trying to disconnect from a docker network with no name?
I'm trying to pull a docker image from our insecure private docker registry. But it fails with below given error.
Environment:-
Ubuntu 20.04.4 LTS
Docker version 20.10.16, build aa7e414
$ cat /etc/docker/daemon.json
{
"insecure-registries": ["private-registry:8080"]
}
Problem:-
$ docker pull private-registry:8080/docker/image/linux:1.1
Error response from daemon: received unexpected HTTP status: 502 Bad Gateway
$ sudo systemctl status docker.service
May 31 07:49:58 server-1 dockerd[3596]: time="2022-05-31T07:49:58.200818890+05:30" level=warning msg="Error getting v2 registry: Get \"https://private-registry>
May 31 07:49:58 server-1 dockerd[3596]: time="2022-05-31T07:49:58.200866976+05:30" level=info msg="Attempting next endpoint for pull after error: Get \">
May 31 07:49:58 server-1 dockerd[3596]: time="2022-05-31T07:49:58.208355568+05:30" level=info msg="Attempting next endpoint for pull after error: receiv>
May 31 07:49:58 server-1 dockerd[3596]: time="2022-05-31T07:49:58.211891361+05:30" level=error msg="Handler for POST /v1.41/images/create returned error>
Any help will be appreciated to solve this problem. thanks!
Problem solved after rebooting our docker private registry server.
Docker-CE 19.03.8
Swarm init
Setup: 1 Manager Node nothing more.
We deploy many new stacks per day and sometime i see the following line:
evel=error msg="Failed to allocate network resources for node sdlk0t6pyfb7lxa2ie3w7fdzr" error="could not find network allocator state for network qnkxurc5etd2xrkb53ry0fu59" module=node node.id=yp0u6n9c31yh3xyekondzr4jc
After 2 to 3 days. No new services can be started because there are no free VIPs.
I see the following line in my logs:
level=error msg="Could not parse VIP address while releasing"
level=error msg="error deallocating vip" error="invalid CIDR address: " vip.addr= vip.network=oqcsj99taftdu3b0t3nrgbgy1
level=error msg="Event api.EventUpdateTask: Failed to get service idid0u7vjuxf2itpv8n31da57 for task 6vnc8jdkgxwxqbs3ixly2i6u4 state NEW: could not find service idid0u7vjuxf2itpv8n31da57" module=node ...
level=error msg="Event api.EventUpdateTask: Failed to get service sbjb7nk0wk31c2ayg8x898fhr for task noo21whnbwkyijnqavseirfg0 state NEW: could not find service sbjb7nk0wk31c2ayg8x898fhr" module=node ...
level=error msg="Failed to find network y73pnq85mjpn1pon38pdbtaw2 on node sdlk0t6pyfb7lxa2ie3w7fdzr" module=node node.id=yp0u6n9c31yh3xyekondzr4jc
We tried to investigate this by using the debug mode.
Here are some lines that get to me:
level=debug msg="Remove interface veth84e7185 failed: Link not found"
level=debug msg="Remove interface veth64c3a65 failed: Link not found"
level=debug msg="Remove interface vethf1703f1 failed: Link not found"
level=debug msg="Remove interface vethe069254 failed: Link not found"
level=debug msg="Remove interface veth2b81763 failed: Link not found"
level=debug msg="Remove interface veth0bf3390 failed: Link not found"
level=debug msg="Remove interface veth2ed04cc failed: Link not found"
level=debug msg="Remove interface veth0bc27ef failed: Link not found"
level=debug msg="Remove interface veth444343f failed: Link not found"
level=debug msg="Remove interface veth036acf9 failed: Link not found"
level=debug msg="Remove interface veth62d7977 failed: Link not found"
and
level=debug msg="Request address PoolID:10.0.0.0/24 App: ipam/default/data, ID: GlobalDefault/10.0.0.0/24, DBIndex: 0x0, Bits: 256, Unselected: 60, Sequence: (0xf7dfeeee, 1)->(0xedddddb7, 1)->(0x77777777, 3)->(0x77777775, 1)->(0x77ffffff, 1)->(0xffd55555, 1)->end Curr:233 Serial:true PrefAddress:<
When the UNSELECTED part goes to 0 no new containers can be deployed. They are stuck in the NEW state.
Has anyone expirenced something like this? Or can someone help me?
We believe, that the problem has to do something with the release of the 10.0.0.0/24 (our ingress) addresses.
Did you tried to stop and re- start the docker demon?
sudo service docker stop
sudo service docker start
Also, you may find it useful to have a look at the magnificent documentation on https://dockerswarm.rocks/
I usually use this sequence to update a service
export DOMAIN=xxxx.xxxxx.xxx
docker stack rm $service_name
export NODE_ID=$(docker info -f '{{.Swarm.NodeID}}')
# export environment vars if needed
# update data if needed
docker node update --label-add $service_name.$service_name-data=true $NODE_ID
docker stack deploy -c $service_name.yml $service_name
If you see your container stuck in NEW state, probably your are affected by this problem: https://github.com/moby/moby/issues/37338 reported by cintiadr:
Docker stack fails to allocate IP on an overlay network, and gets stuck in NEW current state #37338
Reproducing it:
Create a swarm cluster (1 manager, 1 worker). I created AWS t2.large Amazon linux instances, installed docker using their docs, version 18.06.1-ce.
# Deploy a new overlay network from a stack (docker-network.yml)
$ ./deploy-network.sh
Deploy 60 identical services attaching to that network - 3 replicas each - from stacks (docker-network.yml)
$ ./deploy-services.sh
You can verify that all services are happily running.
Now let's bring the worker down.
Run:
docker node update --availability drain <node id> && docker node rm --force <node id>
Note: drain is an async operation (something I wasn't aware), so to reproduce this use case you shouldn't wait for the drain to complete
Create a new worker (completely new node/machine), and join the cluster.
You are going to see that very few services are actually able to start. All other will be continuously being rejected due to no IP available.
In past versions (17 I believe), the containers wouldn't be rejected (but rather be stuck in NEW).
How to avoid that problem?
If you drain and patiently wait for all the containers to be terminated before removing the node, it appears that this problem is completely avoided.
I start docker at private cloud host and get next error
sudo docker run hello-world
docker: Error response from daemon: OCI runtime create failed: /var/lib/docker/overlay2/83d55d497417883ea85b63ebe3138e5f5dbc2b3c8b2db663c1a007e7af1757f1/merged
is not an absolute path or is a symlink: unknown.
ERRO[0000] error waiting for container: context canceled
ll shows next
ll /var/lib/docker/overlay2
lrwxrwxrwx 1 root root 20 Jun 19 15:32 /var/lib/docker/overlay2 -> /dev/docker-overlay2/
Are there any way to fix it ?
I met this problem, I restart docker service, solve it
systemctl restart docker
problem was that someone configured symlink for "fat" directory with images to move it to another disc. Instead of properly configuring in /etc/docker/daemon.json
After removing and reinstalling a default machine using Docker Quickstart and VirtualBox any docker pull fails. Restarting docker-machine doesn't help.
For example:
~$ docker pull ubuntu:14.04
Error response from daemon: Get https://registry-1.docker.io/v2/library/ubuntu/manifests/14.04:
Received unexpected HTTP status: 500 Internal Server Error
Likely caused by an error at docker.io (500 should've been a red flag ;) that was intermittent. Next time double check from another machine if possible.
Earlier thoughts:
With the default docker-machine running, regenerating the tls certs:
docker-machine regenerate-certs
fixes the problem, sometimes, for one pull. But has also yielded:
Error response from daemon: Get <url omitted>: Get <url omitted>:
net/http: request canceled (Client.Timeout exceeded while awaiting headers)