I loaded a docker image to my K3 cluster like so:
# ctr -n k8s.io -a /run/k3s/containerd/containerd.sock image import /home/rocky/myawx-v1.0.0.tar
(actually I have tried many different version of the above command)
I can see the image here:
# k3s ctr image ls | grep awx
docker.io/library/myawx:v1.0.0 application/vnd.docker.distribution.manifest.v2+json sha256:7...9 617.3 MiB linux/amd64 io.cri-containerd.image=managed
quay.io/ansible/awx-ee:latest application/vnd.docker.distribution.manifest.v2+json sha256:5...0 613.4 MiB linux/amd64 io.cri-containerd.image=managed
quay.io/ansible/awx-ee#sha256:5...0 application/vnd.docker.distribution.manifest.v2+json sha256:5...0 613.4 MiB linux/amd64 io.cri-containerd.image=managed```
But if I try to pull the image I get this error:
# k3s ctr image pull docker.io/library/myawx:v1.0.0
docker.io/library/myawx:v1.0.0: resolving |--------------------------------------|
elapsed: 0.5 s total: 0.0 B (0.0 B/s)
INFO[0000] trying next host error="pull access denied, repository does not exist or may require authorization: server message:
insufficient_scope: authorization failed" host=registry-1.docker.io
ctr: failed to resolve reference "docker.io/library/myawx:v1.0.0": pull access denied,
repository does not exist or may require authorization: server message:
insufficient_scope: authorization failed
If I try to run the image I get this ...
# k3s ctr run docker.io/library/myawx myawx-run
ctr: image "docker.io/library/myawx": not found
I have no idea what I am doing and I am just grasping at straws. Please help!
Related
I am getting error as error creating overlay mount to /var/lib/containers/storage/overlay/7a617fad39ce9178c810e29aaef4af73647d8e35ae0969483059441c1c4ee9cd/merged
Please find debug info below.
OS
root#cks-master:/vagrant/files# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.6 LTS
Release: 18.04
Codename: bionic
Dockerfile
# cat ch5Dockerfile
FROM bash
CMD ["ping", "killer.sh"]
Build log
root#cks-master:/vagrant/files# podman build -t simple -f ch5Dockerfile .
STEP 1/2: FROM bash
ERRO[0000] error unmounting /var/lib/containers/storage/overlay/7a617fad39ce9178c810e29aaef4af73647d8e35ae0969483059441c1c4ee9cd/merged: invalid argument
Error: error mounting new container: error mounting build container "6c0f88a6da54d713e18283e16521385fff736bc1a1072938fddfc6be4b3d43cc": error creating overlay mount to /var/lib/containers/storage/overlay/7a617fad39ce9178c810e29aaef4af73647d8e35ae0969483059441c1c4ee9cd/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/BNREFG6CRAAHJ7VSYG3EUXV5UO:/var/lib/containers/storage/overlay/l/MDSWZVRVZNCOW75JF32K6D4QQC:/var/lib/containers/storage/overlay/l/4G3NS52LYHWPTKA4FURHLYMAPF,upperdir=/var/lib/containers/storage/overlay/7a617fad39ce9178c810e29aaef4af73647d8e35ae0969483059441c1c4ee9cd/diff,workdir=/var/lib/containers/storage/overlay/7a617fad39ce9178c810e29aaef4af73647d8e35ae0969483059441c1c4ee9cd/work": invalid argument
root#cks-master:/vagrant/files#
podman version
root#cks-master:/vagrant/files# podman version
Version: 3.4.2
API Version: 3.4.2
Go Version: go1.15.2
Built: Thu Jan 1 00:00:00 1970
OS/Arch: linux/amd64
root#cks-master:/vagrant/files#
Issue resolved.
I also posted the issue in podman room at https://app.element.io/#/room/#podman:fedoraproject.org
I am suggested with podman reset(command trace is below), then its complained about storage.conf, I removed that file and did reset again. Then its worked.
I still wonder whats inside of storage.conf causing this issue but I deleted before looking into it. Finally its worked and I am able to continue. Hope it helps.
Note: Post the deleting storage.conf file and podman reset, I tried with docker build as well just to check if docker has any dependency over storage.conf file, but none, docker build also executed successfully( command trace below)
root#cks-master:~# podman system reset -f
A storage.conf file exists at /etc/containers/storage.conf
You should remove this file if you did not modified the configuration.
root#cks-master:~# rm /etc/containers/storage.conf
root#cks-master:~# podman system reset -f
root#cks-master:~# podman build -t simple -f /vagrant/files/ch5Dockerfile .
STEP 1/2: FROM bash
Resolving "bash" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull docker.io/library/bash:latest...
Getting image source signatures
Copying blob 9621f1afde84 done
Copying blob 1dd831616e40 done
Copying blob fd6cd28e0879 done
Copying config 8b332999f6 done
Writing manifest to image destination
Storing signatures
STEP 2/2: CMD ["ping", "killer.sh"]
COMMIT simple
--> cd1407a69ea
Successfully tagged localhost/simple:latest
cd1407a69ea490496d6635700958f2b5fcf2b1d01f8dd218dea0f83187e55872
root#cks-master:~# podman run --name simple simple
PING killer.sh (35.227.196.29): 56 data bytes
64 bytes from 35.227.196.29: seq=0 ttl=42 time=15.689 ms
64 bytes from 35.227.196.29: seq=1 ttl=42 time=14.662 ms
64 bytes from 35.227.196.29: seq=2 ttl=42 time=15.161 ms
^C
--- killer.sh ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 14.662/15.170/15.689 ms
root#cks-master:~# docker build -t simple -f /vagrant/files/ch5Dockerfile .
Sending build context to Docker daemon 3.141MB
Step 1/2 : FROM bash
latest: Pulling from library/bash
9621f1afde84: Pull complete
1dd831616e40: Pull complete
fd6cd28e0879: Pull complete
Digest: sha256:e4624241e953934fc4c396217253d8322ebda53be3b1863cd7795541d168034f
Status: Downloaded newer image for bash:latest
---> 8b332999f684
Step 2/2 : CMD ["ping", "killer.sh"]
---> Running in 306963a83d1c
Removing intermediate container 306963a83d1c
---> 51dee555fd57
Successfully built 51dee555fd57
Successfully tagged simple:latest
root#cks-master:~# ^C
root#cks-master:~#
I've started k3d with k3d create && k3d start.
All pods fail to start with the following error:
Warning FailedCreatePodSandBox 14s (x2 over 31s) kubelet,
k3d-k3s-default-server Failed to create pod sandbox: rpc error: code
= Unknown desc = failed to get sandbox image "docker.io/rancher/pause:3.1": failed to pull image
"docker.io/rancher/pause:3.1": failed to pull and unpack image
"docker.io/rancher/pause:3.1": failed to resolve reference
"docker.io/rancher/pause:3.1": failed to do request: Head
https://registry-1.docker.io/v2/rancher/pause/manifests/3.1: dial tcp:
lookup registry-1.docker.io: Try again
As recommended by a k3d contributor, I've exec'ed into the k3d server container and attempted to pull the image manually:
$ docker exec -it k3d-k3s-default-server sh
/ # ctr image pull docker.io/rancher/pause:3.1
docker.io/rancher/pause:3.1: resolving |--------------------------------------|
elapsed: 4.9 s total: 0.0 B (0.0 B/s)
ctr: failed to resolve reference "docker.io/rancher/pause:3.1": failed to do request: Head https://registry-1.docker.io/v2/rancher/pause/manifests/3.1: dial tcp: lookup registry-1.docker.io: Try again
In the host environment, docker pull docker.io/rancher/pause:3.1 works just fine.
I've seen a number of people resolve the issue by tweaking various DNS settings. But none described how they arrived at their particular solution.
Solving this issue would make me happy. Discovering a general diagnosis strategy would make me even happier.
What hasn't worked
From here:
I got the issue. I had one entry in
/etc/systemd/network/en0.networking Deleted that file, and everything
is fine.
I have no files in /etc/systemd/network/.
I had the same issue with k3s not being able to pull images and solved it by updating my /etc/resolv.conf to be symlinked from /run/systemd/resolve/stub-resolv.conf on the host machine with
ln -sf /etc/resolv.conf /run/systemd/resolve/stub-resolv.conf
I would like to use docker with amazon's elastic file system (EFS). So I mounted the EFS on the EC2 machine and set an efs-folder as the docker root folder in the docker config file. But when I want to create an image I get this error:
Step 1 : FROM ubuntu:14.04
14.04: Pulling from library/ubuntu
862a3e9af0ae: Extracting [==================================================>] 65.7 MB/65.7 MB
6498e51874bf: Download complete
159ebdd1959b: Download complete
0fdbedd3771a: Download complete
7a1f7116d1e3: Download complete
failed to register layer: Error processing tar file(exit status 1): errno 524
In the docker logs I get this error:
Error trying v2 registry: failed to register layer: Error processing tar file(exit status 1): errno 524
Attempting next endpoint for pull after error: failed to register layer: Error processing tar file(exit status 1): errno 524
Any ideas how I can solve this?
I solved it by adding -s devicemapper to the docker daemoncommand as described in an issue: New Docker versions cannot host datastore on NFS share
The private registry was worked well based on docker 1.10.3,but I can not pull/push images after the docker updated to 1.12.0.
I had modified the /etc/sysconfig/docker as:
OPTIONS='--selinux-enabled=true --insecure-registry=myip:5000'
or
OPTIONS='--selinux-enabled=true --insecure-registry myip:5000'
but when I exec pull/push,I got this error:
$ docker pull myip:5000/cadvisor
Using default tag: latest
Error response from daemon: Get https://myip:5000/v1/_ping: http: server gave HTTP response to HTTPS client
when I change back docker to 1.10.3, it still work well as below:
$ docker pull myip:5000/cadvisor
Using default tag: latest
Trying to pull repository myip:5000/cadvisor ...
latest: Pulling from myip:5000/cadvisor
09d0220f4043: Pull complete
a3ed95caeb02: Pull complete
151807d34af9: Pull complete
14cd28dce332: Pull complete
Digest:
sha256:33b6475cd5b7646b3748097af1224de3eee3ba7cf5105524d95c0cf135f59b47
Status: Downloaded newer image for myip/cadvisor:latest
Some relative information are listed below:
docker version
Client:
Version: 1.12.0
API version: 1.24
Go version: go1.6.3
Git commit: 8eab29e
Built:
OS/Arch: linux/amd64
Server:
Version: 1.12.0
API version: 1.24
Go version: go1.6.3
Git commit: 8eab29e
Built:
OS/Arch: linux/amd64
docker info
Containers: 4
Running: 1
Paused: 0
Stopped: 3
Images: 241
Server Version: 1.12.0
Storage Driver: devicemapper
Pool Name: docker-253:0-6809-pool
Pool Blocksize: 65.54 kB
Base Device Size: 107.4 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 5.459 GB
Data Space Total: 107.4 GB
Data Space Available: 34.74 GB
Metadata Space Used: 9.912 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.138 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use '--storage-opt dm.thinpooldev' to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: host overlay null bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 62.39 GiB
Name: server_3
ID: TITS:BL4B:M5FE:CIRO:5SW6:TVIV:HW36:J7OS:WLHF:46T6:2RBA:WCNV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 21
Goroutines: 32
System Time: 2016-08-02T10:33:06.414048675+08:00
EventsListeners: 0
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0/8
docker exec <registry-container> registry -version
registry github.com/docker/distribution v2.2.1
After I restart the docker daemon in debug mode, the daemon logs when reproducing my problem are listed below:
DEBU[0794] Calling POST /v1.24/images/create?fromImage=10.10.10.40%3A5000%2Fcadvisor&tag=latest
DEBU[0794] hostDir: /etc/docker/certs.d/10.10.10.40:5000
DEBU[0794] hostDir: /etc/docker/certs.d/10.10.10.40:5000
DEBU[0794] Trying to pull 10.10.10.40:5000/cadvisor from https://10.10.10.40:5000 v2
WARN[0794] Error getting v2 registry: Get https://10.10.10.40:5000/v2/: http: server gave HTTP response to HTTPS client
ERRO[0794] Attempting next endpoint for pull after error: Get https://10.10.10.40:5000/v2/: http: server gave HTTP response to HTTPS client
DEBU[0794] Trying to pull 10.10.10.40:5000/cadvisor from https://10.10.10.40:5000 v1
DEBU[0794] hostDir: /etc/docker/certs.d/10.10.10.40:5000
DEBU[0794] attempting v1 ping for registry endpoint https://10.10.10.40:5000/v1/
DEBU[0794] Fallback from error: Get https://10.10.10.40:5000/v1/_ping: http: server gave HTTP response to HTTPS client
ERRO[0794] Attempting next endpoint for pull after error: Get https://10.10.10.40:5000/v1/_ping: http: server gave HTTP response to HTTPS client
ERRO[0794] Handler for POST /v1.24/images/create returned error: Get https://10.10.10.40:5000/v1/_ping: http: server gave HTTP response to HTTPS client
DEBU[1201] clean 2 unused exec commands
What's more, I just run a simple command to launch the private registry for test, anything else is by default:
docker run -d -p 5000:5000 --restart=always --name registry -v 'pwd'/data:/var/lib/registry registry:2
No proxy is configured. In summary, it is only a quiet sample environment for test.
I had the same issue.
This helped for me:
Create or modify /etc/docker/daemon.json on the client machine
{ "insecure-registries":["myregistry.example.com:5000"] }
Restart docker daemon
sudo /etc/init.d/docker restart
For Windows users
Add local registry here and apply:
For Mac Users:
Update the docker preferences using the (docker) icon in top bar
Preferences -> Daemon -> Insecure Registry [Click (+) sign] -> add :port
hit "Apply & Restart" button at bottom
I also had same issue and followed below steps:
1. Create file
vi /etc/docker/daemon.json
2. Add below content
{
"insecure-registries":["192.168.1.142:5000"]
}
3.Restart Docker
service docker restart
If you are using Windows and you get this error you need to create a file here: "C:\ProgramData\docker\config\daemon.json"
and do the same as #Bspec mentioned above:
{ "insecure-registries":["myregistry.example.com:5000"] }
Then restart docker using PowerShell commands:
Stop-Service docker
Start-Service docker
modifying "/etc/docker/daemon.json" didn't work for me.
Putting it under "/etc/sysconfig/docker" as below, worked.
INSECURE_REGISTRY="--insecure-registry 192.168.24.1:8787"
In order to push, add the ip to insecure registry on the client side (e.g. for Windows)
To pull, add it to the server side (in this case Ubuntu)
vim /etc/docker/daemon.json
and then restart Docker.
None of the solutions worked on Ubuntu 18.04 so spend some time to find the root cause.
Steps to solve an issue
sudo vi /lib/systemd/system/docker.service
# ExecStart=dockerd .... --insecure-registry=192.168.99.100:5000
sudo systemctl stop docker.service
sudo systemctl daemon-reload
sudo systemctl start docker.service
What was the issue?
I would recommend to check where exactly dockerd options are configured regardless of you Linux distribution with:
sudo find /etc /lib -name 'docker*' | while read -r line; do grep dockerd $line /dev/null; done
first test localy
docker push localhost:5000/<ImageName>
if docker pushed is Done going to another server and do it:
sudo nano /etc/docker/daemon.json
{"insecure-registries" : ["<HostName or IP Address registry server>:5000"]}
Saving and...
and for next
sudo systemctl daemon-reload
sudo service docker restart
nice!
now docker pushing on another server:
docker tag <image id> <HostName or IP Address registry server>:5000/<ImageName>
docker push <HostName or IP Address registry server>:5000/<ImageName>
Enjoy It.
I've created a Docker machine to run TensorFlow:
$ docker-machine create --driver virtualbox tensorflow
$ eval $(docker-machine env tensorflow)
$ docker-machine ls
NAME ACTIVE URL STATE URL SWARM DOCKER ERRORS
tensorflow * virtualbox Running tcp://193.168.99.101:2376 v1.9.1
Then I try to launch a container with the TesnorFlow dev image:
$ docker run -it b.gcr.io/tensorflow/tensorflow:latest-devel
But I get the following printout and error:
Unable to find image 'b.gcr.io/tensorflow/tensorflow:latest-devel' locally
latest-devel: Pulling from tensorflow/tensorflow
d5abe3500fe7: Verifying Checksum
b4906bf7cb96: Download complete
bfd1864b7d78: Verifying Checksum
b5834ff0edff: Download complete
37c70795c272: Download complete
fd1295db72c3: Verifying Checksum
0851dd369c75: Download complete
f090bc83193e: Download complete
3505c3e79922: Download complete
5f7f94f4a89a: Download complete
82d880401459: Download complete
3f8ab9378c1a: Verifying Checksum
cd2e7af6dd36: Verifying Checksum
0fbd1aaa7f93: Download complete
a073e9eadc31: Verifying Checksum
eac8972ce5cd: Download complete
d20000bda44f: Download complete
7abc7a47a494: Download complete
6a90f0a0e005: Download complete
Pulling repository b.gcr.io/tensorflow/tensorflow
74c849ee103e: Error pulling image (latest-devel) from b.gcr.io/tensorflow/tensorflow, Untar re-exec error: exit status 1: output: write /usr/lib/gcc/x86_64-linux-gnu/4.8/libitm.a: no space left on device bitm.a: no space left on device
a82f81f25750: Download complete
b207c06aba70: Download complete
d55e68e6cc9c: Download complete
0aa714ad37e0: Download complete
28f77615a692: Error downloading dependent layers
Error pulling image (latest-devel) from b.gcr.io/tensorflow/tensorflow, Untar re-exec error: exit status 1: output: write /usr/lib/gcc/x86_64-linux-gnu/4.8/libitm.a: no space left on device
I'm not sure what to do here... Will someone explain what this means and how I should proceed?
Extra info: I'm on Mac OSX 10.9,
$ docker info
Containers: 0
Images: 12
Server Version: 1.9.1
Storage Driver: aufs
Root Dir: /mnt/sda1/var/lib/docker/aufs
Backing Filesystem: tmpfs
Dirs: 12
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.13-boot2docker
Operating System: Boot2Docker 1.9.1 (TCL 6.4.1); master : cef800b - Fri Nov 20 19:33:59 UTC 2015
CPUs: 1
Total Memory: 996.2 MiB
Name: tensorflow
Debug mode (server): true
File Descriptors: 11
Goroutines: 19
System Time: 2016-01-24T02:54:29.677797424Z
EventsListeners: 0
Init SHA1:
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
provider=virtualbox
Instead of the default docker create, try this:
docker-machine create --driver virtualbox --virtualbox-disk-size "20000" tensorflow
Then:
eval $(docker-machine env tensorflow)
docker run -it b.gcr.io/tensorflow/tensorflow:latest-devel
Note:
See Docker increase disk space
Make sure you have 20 GB on your physical disk