Normally when I redirect my docker log to a text file, all the content will store in the output file.
This is what I expected outcome
[root#ip-xx-xx-xx-xx ~]$ docker logs aa46e7139ce5 > docker.log
[root#ip-xx-xx-xx-xx ~]$
But sometimes the output will become following and the errors are not included in the docker.log
[root#ip-xx-xx-xx-xx ~]$ docker logs aaasdas9c356 > docker.log
Debugger failed to attach: handshake failed - received >GET / HTTP/1.0 < - expected >JDWP-Handshake<
Debugger failed to attach: recv failed during handshake: Resource temporarily unavailable
Debugger failed to attach: received >< - expected >JDWP-Handshake<
[root#ip-xx-xx-xx-xx ~]$
Just wondering why the output will have different behaviour.
Thanks
You're missing stderr. The output redirect handles file descriptor 1, stdout. For messages sent to stderr on file descriptor 2, you need to redirect that separately. Either to the same location:
docker logs aaasdas9c356 > docker.log 2>&1
Or to a separate file:
docker logs aaasdas9c356 > docker.log 2>docker.err
Related
This is what my commend looks like
kubectl cp /Users/Documents/keycloak-deployment/import/realm-export-sdp.json sdp-steve/keycloak-7458697ddb-tbzp8:/tmp
And I got the error message as below:
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:235: starting container process caused "exec: \"tar\": executable file not found in $PATH"
error: Internal error occurred: error executing command in container: read unix #->/var/run/docker.sock: read: connection reset by peer
Does anyone know how to handle this error? Thank you in advance.
Your container image must have tar binary present for running kubectl cp subcommand. As a result, you are getting the following error:
"exec: \"tar\": executable file not found in $PATH
See the below snippet:
kubectl cp --help
Copy files and directories to and from containers.
Examples:
# !!!Important Note!!!
# Requires that the 'tar' binary is present in your container
# image. If 'tar' is not present, 'kubectl cp' will fail.
#
# For advanced use cases, such as symlinks, wildcard expansion or
# file mode preservation, consider using 'kubectl exec'.
You may check this page showing why tar is needed.
I am trying to install Kubernetes on my CentOS machine, when I intialize the cluster, I have the following error.
I specify that I am behind a corporate proxy. I have already configured it for Docker in the directory: /etc/systemd/system/docker.service.d/http-proxy.conf
Docker work fine.
No matter how hard I look, I can't find a solution to this problem.
Thank you for your help.
# kubeadm init
W1006 14:29:38.432071 7560 version.go:102] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get "https://dl.k8s.io/release/stable-1.txt": x509: certificate signed by unknown authority
W1006 14:29:38.432147 7560 version.go:103] falling back to the local client version: v1.19.2
W1006 14:29:38.432367 7560 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.19.2
[preflight] Running pre-flight checks
[WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
[WARNING HTTPProxy]: Connection to "https://192.168.XXX.XXX" uses proxy "http://proxyxxxxx.xxxx.xxx:xxxx/". If that is not intended, adjust your proxy settings
[WARNING HTTPProxyCIDR]: connection to "10.96.0.0/12" uses proxy "http://proxyxxxxx.xxxx.xxx:xxxx/". This may lead to malfunctional cluster setup. Make sure that Pod and Services IP ranges specified correctly as exceptions in proxy configuration
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.19.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: handshake failure
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.19.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: handshake failure
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.19.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: handshake failure
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.19.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: handshake failure
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: handshake failure
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.4.13-0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: handshake failure
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.7.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: handshake failure
, error: exit status 1
# kubeadm config images pull
W1006 17:33:41.362395 80605 version.go:102] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get "https://dl.k8s.io/release/stable-1.txt": x509: certificate signed by unknown authority
W1006 17:33:41.362454 80605 version.go:103] falling back to the local client version: v1.19.2
W1006 17:33:41.362685 80605 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
failed to pull image "k8s.gcr.io/kube-apiserver:v1.19.2": output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: handshake failure
, error: exit status 1
To see the stack trace of this error execute with --v=5 or higher
Maybe root certificates on your machine are outdated - so it does not consider certificate of k8s.gcr.io as valid one. This message x509: certificate signed by unknown authority hints to it.
Try to update them: yum update ca-certificates || yum reinstall ca-certificates
I just did a dig to k8s.gcr.io, and I added the IP given by the request to /etc/hosts.
# dig k8s.gcr.io
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.2 <<>> k8s.gcr.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44303
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;k8s.gcr.io. IN A
;; ANSWER SECTION:
k8s.gcr.io. 21599 IN CNAME googlecode.l.googleusercontent.com.
googlecode.l.googleusercontent.com. 299 IN A 64.233.168.82
;; Query time: 72 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Nov 24 11:45:37 CST 2020
;; MSG SIZE rcvd: 103
# cat /etc/hosts
64.233.168.82 k8s.gcr.io
And now it works!
# kubeadm config images pull
W1124 11:46:41.297352 50730 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.19.4
[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.19.4
[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.19.4
[config/images] Pulled k8s.gcr.io/kube-proxy:v1.19.4
[config/images] Pulled k8s.gcr.io/pause:3.2
[config/images] Pulled k8s.gcr.io/etcd:3.4.13-0
[config/images] Pulled k8s.gcr.io/coredns:1.7.0
Working also with v1.19.2 - I've got the same error.
It seems to be related to the issue mentioned here (and I think also in here).
I re-install kubeadm on the node and ran the kubeadm init workflow again - it is now working with v1.19.3 and the errors are gone.
All master nodes images are pulled successfully.
Also verified with:
sudo kubeadm config images pull
(*) You can run kubeadm init with --kubernetes-version=X.Y.Z (1.19.3 in our case).
I had the same error. Maybe as others say, it's because of an outdated certificate. I believe it's not required to delete anything.
Simple solution was running one of those two commands, which will reconnect to Container repositories via:
podman login
docker login
Source: podman-login
I had this issue on version version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2" when i tried joining a second control panel.
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:v1.9.3: output: E0923 04:47:51.763983 1598 remote_image.go:242] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image \"k8s.gcr.io/coredns:v1.9.3\": failed to resolve reference \"k8s.gcr.io/coredns:v1.9.3\": k8s.gcr.io/coredns:v1.9.3: not found" image="k8s.gcr.io/coredns:v1.9.3"
time="2022-09-23T04:47:51Z"...
See #99321 it's now k8s.gcr.io/coredns/coredns:v1.9.3 instead of
k8s.gcr.io/coredns:v1.9.3 and i don't now why
by kluevandrew,
refererence: https://github.com/kubernetes/kubernetes/issues/112131
This worked, am using containerd:
crictl pull k8s.gcr.io/coredns/coredns:v1.9.3
ctr --namespace=k8s.io image tag k8s.gcr.io/coredns/coredns:v1.9.3 k8s.gcr.io/coredns:v1.9.3
docker solution:
docker pull k8s.gcr.io/coredns/coredns:v1.9.3
docker tag k8s.gcr.io/coredns/coredns:v1.9.3 k8s.gcr.io/coredns:v1.9.3
Check imageRepository in kubeadm-config configmap (or your kubeadm config file, if You run something like kubeadm init --config=/tmp/kubeadm-config.yml).
In a docker container I want to run k8s.
When I run kubeadm join ... or kubeadm init commands I see sometimes errors like
\"modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could
not open moddep file
'/lib/modules/3.10.0-1062.1.2.el7.x86_64/modules.dep.bin'.
nmodprobe:
FATAL: Module configs not found in directory
/lib/modules/3.10.0-1062.1.2.el7.x86_64",
err: exit status 1
because (I think) my container does not have the expected kernel header files.
I realise that the container reports its kernel based on the host that is running the container; and looking at k8s code I see
// getKernelConfigReader search kernel config file in a predefined list. Once the kernel config
// file is found it will read the configurations into a byte buffer and return. If the kernel
// config file is not found, it will try to load kernel config module and retry again.
func (k *KernelValidator) getKernelConfigReader() (io.Reader, error) {
possibePaths := []string{
"/proc/config.gz",
"/boot/config-" + k.kernelRelease,
"/usr/src/linux-" + k.kernelRelease + "/.config",
"/usr/src/linux/.config",
}
so I am bit confused what is simplest way to run k8s inside a container such that it consistently past this getting the kernel info.
I note that running docker run -it solita/centos-systemd:7 /bin/bash on a macOS host I see :
# uname -r
4.9.184-linuxkit
# ls -l /proc/config.gz
-r--r--r-- 1 root root 23834 Nov 20 16:40 /proc/config.gz
but running exact same on a Ubuntu VM I see :
# uname -r
4.4.0-142-generic
# ls -l /proc/config.gz
ls: cannot access /proc/config.gz
[Weirdly I don't see this FATAL: Module configs not found in directory error every time, but I guess that is a separate question!]
UPDATE 22/November/2019. I see now that k8s DOES run okay in a container. Real problem was weird/misleading logs. I have added an answer to clarify.
I do not believe that is possible given the nature of containers.
You should instead test your app in a docker container then deploy that image to k8s either in the cloud or locally using minikube.
Another solution is to run it under kind which uses docker driver instead of VirtualBox
https://kind.sigs.k8s.io/docs/user/quick-start/
It seems the FATAL error part was a bit misleading.
It was badly formatted by my test environment (all on one line.
When k8s was failing I saw the FATAL and assumed (incorrectly) that was root cause.
When I format the logs nicely I see ...
kubeadm join 172.17.0.2:6443 --token 21e8ab.1e1666a25fd37338 --discovery-token-unsafe-skip-ca-verification --experimental-control-plane --ignore-preflight-errors=all --node-name 172.17.0.3
[preflight] Running pre-flight checks
[WARNING FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 4.4.0-142-generic
DOCKER_VERSION: 18.09.3
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUACCT: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.3. Latest validated version: 18.06
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-142-generic/modules.dep.bin'\nmodprobe: FATAL: Module configs not found in directory /lib/modules/4.4.0-142-generic\n", err: exit status 1
[discovery] Trying to connect to API Server "172.17.0.2:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://172.17.0.2:6443"
[discovery] Failed to request cluster info, will try again: [the server was unable to return a response in the time allotted, but may still be processing the request (get configmaps cluster-info)]
There are other errors later, which I originally though were a side-effect of the nasty looking FATAL error e.g. .... "[util/etcd] Attempt timed out"]} but I now think root cause is Etcd part times out sometimes.
Adding this answer in case someone else puzzled like I was.
I've tried to live migrate a wildfly-container to another host like described here. The example with the np container works well. When I replace the example with a simple jboss/wildfly container, I just received this error when criu tries to restore the container on the other host :
Error response from daemon: Cannot restore container <CONTAINER-ID>: criu failed: type NOTIFY errno 0
Error: failed to restore one or more containers
Because I didn't found a solution to this error, I've compiled the linux kernel like described on the criu website and here.
After that sudo criu check prints:
Warn (criu/libnetlink.c:54): ERROR -2 reported by netlink
Warn (criu/libnetlink.c:54): ERROR -2 reported by netlink
Warn (criu/sockets.c:711): The current kernel doesn't support packet_diag
Warn (criu/libnetlink.c:54): ERROR -2 reported by netlink
Warn (criu/sockets.c:721): The current kernel doesn't support netlink_diag
Info prctl: PR_SET_MM_MAP_SIZE is not supported
Looks good.
criu --version
Version: 2.11
docker --version
Docker version 1.6.2, build 7c8fca2
Checkpoint/Restore for an example shell script example worked very well. But when I want to checkpoint a container
docker run -d --name looper busybox /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'
with
criu dump -t $PID --images-dir /tmp/looper
I receive this output
Error (criu/sockets.c:132): Diag module missing (-2)
Error (criu/sockets.c:132): Diag module missing (-2)
Error (criu/sockets.c:132): Diag module missing (-2)
Error (criu/mount.c:701): mnt: 87:./etc/hosts doesn't have a proper root mount
Error (criu/cr-dump.c:1641): Dumping FAILED.`
I can't find some solutions with these errors. Is there any known solution to live migrate a wildfly-container?
Thanks in advance
I'm trying to use Packer to build a docker image of the webapp I'm working on. Whenever I run packer build, when it gets to the step that it runs the runit recipe, I would get Build 'docker' errored: Error executing Chef: Non-zero exit status: 137
I looked into 137, and found out this is the exit code commonly associated with a kill -9. In most cases this would imply that the system is running critically low on memory, and the system is attempting to compensate.
I tried to find the smallest possible reproduction, and I came up with this packer configuration:
{
"builders":[{
"type": "docker",
"pull": false,
"image": "silkstart/basic_server",
"export_path": "image.tar",
"run_command":[
"-d",
"-i",
"-t",
"--memory-reservation",
"1G",
"{{.Image}}",
"/bin/bash"
]
}],
"provisioners":[
{
"type": "chef-solo",
"cookbook_paths": ["cookbooks", "vendor/cookbooks"],
"data_bags_path": "data_bags",
"roles_path": "roles",
"environments_path": "environments",
"run_list": [
"recipe[runit]"
]
}
],
"post-processors": [
{
"type": "docker-import",
"repository": "silkstart/docker_test",
"tag": "0.1"
}
]
}
When I run packer build on this configuration, this is my output:
TMPDIR=/opt/shared packer build packer_files/docker_test.json
docker output will be in this color.
==> docker: Creating a temporary directory for sharing data...
==> docker: Starting docker container...
docker: Run command: docker run -v /opt/shared/packer-docker484290992:/packer-files -d -i -t --memory-reservation 1G silkstart/basic_server /bin/bash
docker: Container ID: 1f87b0cf1fe71f07b580ae6b18415a79c23a1a32a40f5f0366be90f160977a50
==> docker: Provisioning with chef-solo
docker: Installing Chef...
docker: % Total % Received % Xferd Average Speed Time Time Time Current
docker: Dload Upload Total Spent Left Speed
docker: 100 20022 100 20022 0 0 45092 0 --:--:-- --:--:-- --:--:-- 45196
docker: Getting information for chef stable for ubuntu...
docker: downloading https://omnitruck-direct.chef.io/stable/chef/metadata?v=&p=ubuntu&pv=14.04&m=x86_64
docker: to file /tmp/install.sh.23/metadata.txt
docker: trying curl...
docker: url https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/14.04/x86_64/chef_12.6.0-1_amd64.deb
docker: md5 5cfc19d5a036b3f7860716bc9795a85e
docker: sha256 e0b42748daf55b5dab815a8ace1de06385db98e29a27ca916cb44f375ef65453
docker: version 12.6.0downloaded metadata file looks valid...
docker: downloading https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/14.04/x86_64/chef_12.6.0-1_amd64.deb
docker: to file /tmp/install.sh.23/chef_12.6.0-1_amd64.deb
docker: trying curl...
docker: Comparing checksum with sha256sum...
docker:
docker: WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING
docker:
docker: You are installing an omnibus package without a version pin. If you are installing
docker: on production servers via an automated process this is DANGEROUS and you will
docker: be upgraded without warning on new releases, even to new major releases.
docker: Letting the version float is only appropriate in desktop, test, development or
docker: CI/CD environments.
docker:
docker: WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING
docker:
docker: Installing chef
docker: installing with dpkg...
docker: Selecting previously unselected package chef.
docker: (Reading database ... 17195 files and directories currently installed.)
docker: Preparing to unpack .../chef_12.6.0-1_amd64.deb ...
docker: Unpacking chef (12.6.0-1) ...
docker: Setting up chef (12.6.0-1) ...
docker: Thank you for installing Chef!
docker: Creating directory: /tmp/packer-chef-solo
docker: Creating directory: /tmp/packer-chef-solo/cookbooks-0
docker: Creating directory: /tmp/packer-chef-solo/cookbooks-1
docker: Creating directory: /tmp/packer-chef-solo/roles
docker: Creating directory: /tmp/packer-chef-solo/data_bags
docker: Creating directory: /tmp/packer-chef-solo/environments
docker: Creating configuration file 'solo.rb'
docker: Creating JSON attribute file
docker: Executing Chef: sudo chef-solo --no-color -c /tmp/packer-chef-solo/solo.rb -j /tmp/packer-chef-solo/node.json
docker: [2016-01-29T06:42:48+00:00] INFO: Forking chef instance to converge...
docker: [2016-01-29T06:42:48+00:00] INFO: *** Chef 12.6.0 ***
docker: [2016-01-29T06:42:48+00:00] INFO: Chef-client pid: 207
docker: [2016-01-29T06:42:50+00:00] INFO: Setting the run_list to ["recipe[runit]"] from CLI options
docker: [2016-01-29T06:42:50+00:00] INFO: Run List is [recipe[runit]]
docker: [2016-01-29T06:42:50+00:00] INFO: Run List expands to [runit]
docker: [2016-01-29T06:42:50+00:00] INFO: Starting Chef Run for 1f87b0cf1fe7
docker: [2016-01-29T06:42:50+00:00] INFO: Running start handlers
docker: [2016-01-29T06:42:50+00:00] INFO: Start handlers complete.
docker: [2016-01-29T06:42:52+00:00] INFO: Processing service[runit] action nothing (runit::default line 20)
docker: [2016-01-29T06:42:52+00:00] INFO: Processing execute[start-runsvdir] action nothing (runit::default line 24)
docker: [2016-01-29T06:42:52+00:00] INFO: Processing execute[runit-hup-init] action nothing (runit::default line 33)
docker: [2016-01-29T06:42:52+00:00] INFO: Processing apt_package[runit] action install (runit::default line 64)
docker: [2016-01-29T06:42:55+00:00] INFO: Processing cookbook_file[/var/chef/cache/preseed/runit/runit-2.1.1-6.2ubuntu3.seed] action create (dynamically defined)
docker: [2016-01-29T06:42:55+00:00] INFO: cookbook_file[/var/chef/cache/preseed/runit/runit-2.1.1-6.2ubuntu3.seed] created file /var/chef/cache/preseed/runit/runit-2.1.1-6.2ubuntu3.seed
docker: [2016-01-29T06:42:55+00:00] INFO: cookbook_file[/var/chef/cache/preseed/runit/runit-2.1.1-6.2ubuntu3.seed] updated file contents /var/chef/cache/preseed/runit/runit-2.1.1-6.2ubuntu3.seed
docker: [2016-01-29T06:42:55+00:00] INFO: apt_package[runit] pre-seeding package installation instructions
==> docker: Killing the container: 1f87b0cf1fe71f07b580ae6b18415a79c23a1a32a40f5f0366be90f160977a50
Build 'docker' errored: Error executing Chef: Non-zero exit status: 137
I'm not entirely sure what is causing the code 137, and any help would be appreciated.
Update 1
I'm including a gist of the full debug output from Chef. It's much more verbose, mainly due it would seem to all of the attempts Ohai makes to get information.
https://gist.github.com/jrstarke/4c5f3b432aaee70c7f77
No references in here seem to suggest an out of memory error, at least on the docker host.
After much much digging, I found the problem. The underlying problem, and the solution were both found on an issue in cloudfoundry-incubator/garden-linux.
Apparently as part of the setup process one of the post init scripts for runit executes a kill -s HUP 1. Why I'm not entirely sure, but as they noted there, doing a trap '' HUP right before my apt-get install runit and a trap HUP afterwards totally solved my problem.
Check the OOM log on the host machine. Also you can use the execute_command configuration value to turn the log level to debug.
This answer seemed to work for me: https://stackoverflow.com/a/42398166/2878244
I had to increase the memory resources assigned to docker by going to the Docker Tab > Preferences > Advanced