Docker: Jenkins container can't access internet on QNAP device - jenkins

I try to get the Docker image (1.651.3 or latest version) running on my QNAP NAS using the internal ContainerStation.
Just using the default settings without setting any parameters or binding any resource, I can't access the internet. I already tried the NAT or Host network mode, but this would make no difference.
Stacktrace:
Oct 02, 2016 1:55:07 PM javax.jmdns.impl.HostInfo newHostInfo
WARNING: Could not intialize the host network interface on nullbecause of an error: 5929616b9f0b: 5929616b9f0b: unknown error
java.net.UnknownHostException: 5929616b9f0b: 5929616b9f0b: unknown error
at java.net.InetAddress.getLocalHost(InetAddress.java:1505)
at javax.jmdns.impl.HostInfo.newHostInfo(HostInfo.java:75)
at javax.jmdns.impl.JmDNSImpl.<init>(JmDNSImpl.java:407)
at javax.jmdns.JmDNS.create(JmDNS.java:60)
at hudson.DNSMultiCast$1.call(DNSMultiCast.java:32)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: 5929616b9f0b: unknown error
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getLocalHost(InetAddress.java:1500)
... 9 more
Using bash:
$ ping google.de
ping: unknown host
Any idea what might be the problem? Any other docker image can access the internet, only this image has these problems.

I recently experienced the same problem and in my case it turned out to be caused by the ACL lists on the QNAP, preventing the jenkins user in the container from reading the /etc/hosts and /etc/resolv.conf files.
Here's how I fixed it. Perhaps it can be of some help to you as well:
ssh to the QNAP
$ ssh admin#<your IP or domain name here>
Manually create a jenkins user on the QNAP with UID 1000 (the same
UID as the jenkins user in the container)
$ useradd -u 1000 -M -s /bin/false jenkins
Log into the QNAP web interface
Navigate to Control panel -> Users
Click the Edit Shared Folder Permissions icon for the jenkins user
Tick the RW checkbox (Read/Write access) for the Container folder
and click Apply
Start your Jenkins container
Disclaimer: I am no sysadmin so I don't know whether this approach would cause any security issues on your system. You may want to look into that before giving outside access to your Jenkins web interface... :)

Related

How to use vpnkit with minikube on mac

There are many question around this topic, but not the specific info I am after.
Host OS is Mac, and recently had to uninstall Docker Desktop due to their licensing change. So instead we have moved to minikube, and it is all working great with VirtualBox driver.
But ideally we would like to use the hyperkit driver, as it requires less resources than virtualbox, and is (anecdotally) faster. This also all works great until we connect to our VPN (using cisco anyconnect) and then all outbound networking from within the minikube VM stops working. e.g.
k8> minikube ssh "traceroute 8.8.8.8"
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 46 byte packets
1 host.minikube.internal (192.168.64.1) 0.154 ms 0.181 ms 0.151 ms
2 * * *
Everything else is is fine, inbound networking via ingress is all good. And maven-docker-plugin is happily creating images with the minikube docker daemon. Just nothing outbound.
So figured I'd try to work with VPNKit as I have read it is meant to address this issue. But cannot find a lot of detailed documentation, and so am struggling.
We have tried starting VPNKit with minimal config:
vpnkit --ethernet /tmp/vpkit-ethernet.socket --debug
And then attempt to start minikube, but it fails:
k8> minikube delete
🔥 Deleting "minikube" in hyperkit ...
💀 Removed all traces of the "minikube" cluster.
k8> minikube start --driver=hyperkit --hyperkit-vpnkit-sock=/tmp/vpnkit-ethernet.socket
😄 minikube v1.25.1 on Darwin 10.15.7
✨ Using the hyperkit driver based on user configuration
👍 Starting control plane node minikube in cluster minikube
🔥 Creating hyperkit VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🔥 Deleting "minikube" in hyperkit ...
🤦 StartHost failed, but will try again: creating host: create: Error creating machine: Error in driver during machine creation: hyperkit crashed! command line:
hyperkit loglevel=3 console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes random.trust_cpu=on hw_rng_model=virtio base host=minikube
🔥 Creating hyperkit VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
😿 Failed to start hyperkit VM. Running "minikube delete" may fix it: creating host: create: Error creating machine: Error in driver during machine creation: hyperkit crashed! command line:
hyperkit loglevel=3 console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes random.trust_cpu=on hw_rng_model=virtio base host=minikube
❌ Exiting due to PR_HYPERKIT_CRASHED: Failed to start host: creating host: create: Error creating machine: Error in driver during machine creation: hyperkit crashed! command line:
hyperkit loglevel=3 console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes random.trust_cpu=on hw_rng_model=virtio base host=minikube
💡 Suggestion: Hyperkit is broken. Upgrade to the latest hyperkit version and/or Docker for Desktop. Alternatively, you may choose an alternate --driver
🍿 Related issues:
▪ https://github.com/kubernetes/minikube/issues/6079
▪ https://github.com/kubernetes/minikube/issues/5780
And in the vpnkit log we see:
time="2022-02-14T06:07:57Z" level=debug msg="usernet: accepted vmnet connection"
time="2022-02-14T06:07:57Z" level=warning msg="Uwt: Pipe.listen: rejected ethernet connection: EOF"
time="2022-02-14T06:08:07Z" level=debug msg="usernet: accepted vmnet connection"
time="2022-02-14T06:08:07Z" level=warning msg="Uwt: Pipe.listen: rejected ethernet connection: EOF"
So kind of implies something is not right with how I started vpnkit. Have played with IP args to ensure it all matches, but does not help.
My guess is that the --ethernet=path arg is not the right type of socket. I have seen there is also --vsock-path=path but specifying this does not appear to create the socket file like --ethernet=path does. Do I have to create this some other way?
Or are there other config options I need to mess with. e.g. I thought --gateway-forwards=path could help, but can find no documentation on file format or contents.
So, I guess two main questions:
Is what we are trying even possible? Is it the the right way to go about it? Or is it much more complicated than simply running the vpnkit command?
If we are on the right track, does anyone have experience with this, and know how to set up the socket for minikube+vpnkit+hyperkit? What args, config, or other setup is required?
And just to note: --hyperkit-vpnkit-sock=auto is not an option for us, as we do not have docker installed, and so the docker socket file does not exist.
And just in case its a version issue:
k8> minikube version
minikube version: v1.25.1
commit: 3e64b11ed75e56e4898ea85f96b2e4af0301f43d
k8> vpnkit --version
854498c13b1884d4a48d84f3569eb34681af2126
k8> hyperkit -v
hyperkit: 0.20200908
Homepage: https://github.com/docker/hyperkit
License: BSD

Can I run k8s master INSIDE a docker container? Getting errors about k8s looking for host's kernel details

In a docker container I want to run k8s.
When I run kubeadm join ... or kubeadm init commands I see sometimes errors like
\"modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could
not open moddep file
'/lib/modules/3.10.0-1062.1.2.el7.x86_64/modules.dep.bin'.
nmodprobe:
FATAL: Module configs not found in directory
/lib/modules/3.10.0-1062.1.2.el7.x86_64",
err: exit status 1
because (I think) my container does not have the expected kernel header files.
I realise that the container reports its kernel based on the host that is running the container; and looking at k8s code I see
// getKernelConfigReader search kernel config file in a predefined list. Once the kernel config
// file is found it will read the configurations into a byte buffer and return. If the kernel
// config file is not found, it will try to load kernel config module and retry again.
func (k *KernelValidator) getKernelConfigReader() (io.Reader, error) {
possibePaths := []string{
"/proc/config.gz",
"/boot/config-" + k.kernelRelease,
"/usr/src/linux-" + k.kernelRelease + "/.config",
"/usr/src/linux/.config",
}
so I am bit confused what is simplest way to run k8s inside a container such that it consistently past this getting the kernel info.
I note that running docker run -it solita/centos-systemd:7 /bin/bash on a macOS host I see :
# uname -r
4.9.184-linuxkit
# ls -l /proc/config.gz
-r--r--r-- 1 root root 23834 Nov 20 16:40 /proc/config.gz
but running exact same on a Ubuntu VM I see :
# uname -r
4.4.0-142-generic
# ls -l /proc/config.gz
ls: cannot access /proc/config.gz
[Weirdly I don't see this FATAL: Module configs not found in directory error every time, but I guess that is a separate question!]
UPDATE 22/November/2019. I see now that k8s DOES run okay in a container. Real problem was weird/misleading logs. I have added an answer to clarify.
I do not believe that is possible given the nature of containers.
You should instead test your app in a docker container then deploy that image to k8s either in the cloud or locally using minikube.
Another solution is to run it under kind which uses docker driver instead of VirtualBox
https://kind.sigs.k8s.io/docs/user/quick-start/
It seems the FATAL error part was a bit misleading.
It was badly formatted by my test environment (all on one line.
When k8s was failing I saw the FATAL and assumed (incorrectly) that was root cause.
When I format the logs nicely I see ...
kubeadm join 172.17.0.2:6443 --token 21e8ab.1e1666a25fd37338 --discovery-token-unsafe-skip-ca-verification --experimental-control-plane --ignore-preflight-errors=all --node-name 172.17.0.3
[preflight] Running pre-flight checks
[WARNING FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 4.4.0-142-generic
DOCKER_VERSION: 18.09.3
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUACCT: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.3. Latest validated version: 18.06
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-142-generic/modules.dep.bin'\nmodprobe: FATAL: Module configs not found in directory /lib/modules/4.4.0-142-generic\n", err: exit status 1
[discovery] Trying to connect to API Server "172.17.0.2:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://172.17.0.2:6443"
[discovery] Failed to request cluster info, will try again: [the server was unable to return a response in the time allotted, but may still be processing the request (get configmaps cluster-info)]
There are other errors later, which I originally though were a side-effect of the nasty looking FATAL error e.g. .... "[util/etcd] Attempt timed out"]} but I now think root cause is Etcd part times out sometimes.
Adding this answer in case someone else puzzled like I was.

Jenkins Master-Slave: Key exchange was not finished, connection is closed

I want to connect a slave to Master-Jenkins, but when trying to connect i'm getting following Error:
[05/02/18 15:26:59] [SSH] Opening SSH connection to <IP>
Key exchange was not finished, connection is closed.
java.io.IOException: There was a problem while connecting to <IP>:22
at com.trilead.ssh2.Connection.connect(Connection.java:818)
at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1324)
at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:831)
at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:820)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Key exchange was not finished, connection is closed.
at com.trilead.ssh2.transport.KexManager.getOrWaitForConnectionInfo(KexManager.java:93)
at com.trilead.ssh2.transport.TransportManager.getConnectionInfo(TransportManager.java:230)
at com.trilead.ssh2.Connection.connect(Connection.java:770)
... 7 more
Caused by: java.io.IOException: Cannot negotiate, proposals do not match.
at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:405)
at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
... 1 more
[05/02/18 15:26:59] Launch failed - cleaning up connection
[05/02/18 15:26:59] [SSH] Connection closed.
Configuration for Node:
- Start-Method: Start Slave over SSH
- Hostname: is the IP
- Access Data: user i created for SSH Access - > public key is in authorized keys on Slave Node
If i am on my Master as user "jenkins" and do a ssh jenkins#<IP> i can login wihtout problems (public key is on slave).
Why it doesn't work for "UI-Jenkins".
Jenkins-Version: 1.658
Node: Ubuntu 14.04
SSH-Slave Plugin: 1.26
That "solved" the issue:
"Workaround is by commenting out MACs and KexAlgorithm line in /etc/ssh/sshd_config of Jenkins Slave and restarting the sshd (service ssh restart on Ubuntu)
UPDATE: the issue has been resolved as of 2017-04-29 "
Jenkins master fails to connect to the slave over SSH
Thought I'd throw my experience in this thread: my environment had a Windows master and mixed Windows and Linux agents. One Windows agent refused to connect to master, even after Master pushed 'jenkins-agent' and the other supporting files to the agent.
This agent had 6 different versions of the JDK and JRE installed. I uninstalled all of them, reinstalled only the latest JDK we needed, and set JAVA_HOME. This fixed the connectivity issue.
Execute this command on destination node.
sudo -i su -c 'sed -i -e "s/MACs /MACs hmac-sha1,/g" /etc/ssh/sshd_config; service sshd restart'
Just recently run into this issue with docker
Find the Java Path
/home/jenkins # which java
/opt/java/openjdk/bin/java
Export the Java Path. In this case I am using the docker-compose
...
exp_agent:
image: jenkins/ssh-agent:alpine
restart: always
environment:
JENKINS_AGENT_SSH_PUBKEY: $ENV_JENKINS_AGENT_SSH_PUBKEY
JAVA_HOME: $ENV_JAVA_HOME
container_name: jenkins-ssh-agent
ports:
- 22:22
networks:
- jenkins
...
The master still complains about the path of Java as /opt/java/openjdk/bin/java is not among the expected paths
...
[12/04/21 11:44:07] [SSH] Checking java version of /usr/bin/java
...
Create a symbolic link between the java path and one of the expected paths in the docker container (This could be automated in a Dockerfile)
ln -s /opt/java/openjdk/bin/java /usr/bin/java

Why Docker can't start or starting forever?

I have installed docker on Windows 10, run it as administrator.
It's still in the process of launching (starting) after half an hour.
Log looks like:
Version: 17.07.0-ce-win26 (13125)
Channel: edge
Sha1: 7c2cb3783c478f82e7a09cfbd5933d7b587c9c1e
Started on: 2017/09/06 09:17:16.119
Resources: C:\Program Files\Docker\Docker\Resources
OS: Windows 10 Pro
Edition: Professional
Id: 1607
Build: 14393
BuildLabName: 14393.1593.amd64fre.rs1_release.170731-1934
...
[09:18:19.619][DockerDaemonChecker][Error ] Docker daemon is not running
[09:18:19.658][NamedPipeServer][Error ] Unable to execute Start: error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.31/containers/json: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.
в Docker.Backend.DockerDaemonChecker.Check(Func`1 isDaemonProcessStillRunning) в C:\gopath\src\github.com\docker\pinata\win\src\Docker.Backend\DockerDaemonChecker.cs:line 63
в Docker.Core.Pipe.NamedPipeServer.<>c__DisplayClass9_0.<Register>b__0(Object[] parameters) в C:\gopath\src\github.com\docker\pinata\win\src\Docker.Core\pipe\NamedPipeServer.cs:line 47
в Docker.Core.Pipe.NamedPipeServer.RunAction(String action, Object[] parameters) в C:\gopath\src\github.com\docker\pinata\win\src\Docker.Core\pipe\NamedPipeServer.cs:line 145
[09:18:19.674][NamedPipeClient][Error ] Unable to send Start: error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.31/containers/json: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.
[09:18:19.674][Notifications ][Error ] error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.31/containers/json: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.
[09:18:19.705][CrashReport ][Info ] Preparing package to send with the diagnostics
I am trying to reboot, shut down PC, virtualization is enable in bios.
Could you give me some advise to solve the problem?
I had a similar problem some weeks ago with a new user of our systems.
In my case i forgot to add the user to the docker-users group.
Further problems/solutions that i already had:
not enough memory, docker consumes some of it
Crashed containers
Firewall or other tools that block even local TCP access
Bad docker config in %ProgramData%\Docker\config\daemon.json, you could for example try to set it to an empty json file ({})

Spring Cloud Samples Eureka - Docker - Use of underscore in link

I may have encountered an interesting anomaly with the use of Spring Cloud, Eureka and Docker. I am not sure if I have uncovered an issue or if the behavior is expected, but here is the gist.
I start first with eureka running in a named docker container. Next, I launch a docker client with ClientDiscoveryEnabled. The docker client container is using the docker "link" parameter to gain hostname accessibility into the eureka container. The yaml file has an entry for connecting to Eureka that is property driven:
defaultZone: http://user:${eureka.password}#${host.name}:8761/eureka/
Everything works just great, unless I attempt to use an underscore in my container name. If I use an underscore to name my container, the client container cannot resolve this name completely using Eureka registration. If I remove the underscore, everything works fine. Perhaps I missed something and this is expected, but I have not seen any mention of this "feature".
My client is based from the Spring-Cloud-Samples feign-eureka project. Below is the scenario...
This will work and the client will register:
sudo docker run -d -p=8761:8761 --name foobar chrisccoy/microsvcdemoeureka
sudo docker run -d -p=7311:7311 --name democlnt --link foobar:foobar chrisccoy/microsvcdemoclnt java -jar /opt/tst/ms_clnt.jar --host.name=foobar
The following will not work! Eureka will boot, the client will boot, but cannot register:
sudo docker run -d -p=8761:8761 --name foo_bar chrisccoy/microsvcdemoeureka
sudo docker run -d -p=7311:7311 --name democlnt --link foo_bar:foo_bar chrisccoy/microsvcdemoclnt java -jar /opt/tst/ms_clnt.jar --host.name=foo_bar
Below is the log entry and subsequent exception:
2015-02-25 18:51:27.762 ERROR 1 --- [pool-4-thread-1] com.netflix.discovery.DiscoveryClient : Can't get a response from http://user:password#foo_bar:8761/eureka/apps/HELLOCLIENT/172.17.0.11:HelloClient:7311
Can't contact any eureka nodes - possibly a security group issue?
com.sun.jersey.api.client.ClientHandlerException: java.lang.IllegalArgumentException: Host name may not be null
at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:184)
at com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:120)
at com.netflix.discovery.EurekaIdentityHeaderFilter.handle(EurekaIdentityHeaderFilter.java:28)
at com.sun.jersey.api.client.Client.handle(Client.java:648)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:680)
at com.sun.jersey.api.client.WebResource.put(WebResource.java:211)
at com.netflix.discovery.DiscoveryClient.makeRemoteCall(DiscoveryClient.java:1097)
at com.netflix.discovery.DiscoveryClient.makeRemoteCall(DiscoveryClient.java:1060)
at com.netflix.discovery.DiscoveryClient.access$500(DiscoveryClient.java:105)
at com.netflix.discovery.DiscoveryClient$HeartbeatThread.run(DiscoveryClient.java:1583)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Host name may not be null
at org.apache.http.util.Args.notBlank(Args.java:65)
at org.apache.http.HttpHost.<init>(HttpHost.java:81)
at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.getHost(ApacheHttpClient4Handler.java:190)
at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:170)
... 14 common frames omitted
I am able to ping "foo_bar" from within a container running /bin/bash without issue.
sudo docker run -i -t --link foo_bar:foo_bar chrisccoy/microsvcdemoclnt /bin/bash
root#0175222c11bb:~# ping foo_bar
PING foo_bar (172.17.0.12) 56(84) bytes of data.
64 bytes from foo_bar (172.17.0.12): icmp_seq=1 ttl=64 time=0.137 ms
I am not sure where the disconnect is coming from. Or maybe it is a feature that I am unaware.
Any ideas?
Looks like java.net.URI doesn't understand underscore in a domain name. See this gist: https://gist.github.com/spencergibb/ced5199c80f7a6c89499 and this http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6587184

Resources