I am using Jenkins Docker plugin to launch slaves dynamically on the docker host. Jenkins master runs on the same host. When building, it picks a random port on the docker-host and cannot connect to it. It launches docker containers. Here are the logs. Doesn't say more than this. How to debug the issue?
INFO: Started container ID 54fe5780ca820a6e2b7cae40610cfc3229dbf889b0c66d6e34a39b74e13aaec7 for node slave-0000w16w841rc from image: jenkinsubuntu
Feb 28, 2019 9:35:54 PM com.nirima.jenkins.plugins.docker.utils.PortUtils$ConnectionCheckSSH execute
INFO: SSH port is open on 127.0.0.1:10007
[02/28/19 21:35:54] SSH Launch of slave-0000w16w841rc on 127.0.0.1 failed in 29 ms
Feb 28, 2019 9:36:03 PM hudson.slaves.NodeProvisioner$2 run
INFO: Image of jenkinsubuntu provisioning successfully completed. We have now 7 computer(s) `
Didn't find an answer. But ended-up selecting the connect method to attach docker containerin Docker template for docker plugin which in-turn made it work.
Although you have posted an alternative way but here what you should do and be aware about as you asked about how to debug the issue
You need an image that contains SSHD and the plugin's documentation advises to use jenkins/ssh-slave as a base for your custom image.
According to the log you provided it seems that you use a custom image called jenkinsubuntu assuming that image based on jenkins/ssh-slave then you need to make sure that you didn't override the original entrypoint as described in the documentation:
Avoid overriding the docker command, as SSH Launcher relies on it.
You can use an Entrypoint to run some side service inside your build agent container before the agent runtime starts and establish a connexion. Just ensure your entrypoint eventually run the passed command :
exec "$#"
Lastly make sure that the host where the ssh will takes place contains a private key that matches the public key that was injected inside the container.
If the first and the second point was taken care of then here is what to you should do to debug the ssh problem.
You can login try to login to the same container using ssh while its running.
If the previous step worked then check the log either using docker logs or by checking /var/log, note that you may need to change the LogLevel under /etc/ssh/sshd_config to VERBOSE to make all the details of ssh login attempts saved in /var/log/auth.log file where you can identify the issue
Related
I downloaded Docker using Docker Desktop for Apple M1 chips. I can run containers, the integration with VsCode works okay but I can't integrate it with Intellij IDEA Ultimate. It keeps giving this error.
But I can run my containers and create images from the terminal, I can also see the containers and images in Docker Desktop too. What could be the reason behind this? I also tried to check whether var/run/docker.sock is existing and it really isn't, there is no such file as that.
I also tried the same steps on my second computer and the exact same thing happened. Steps to reproduce: 1- Download Intellij IDEA Ultimate, open a repo that uses docker 2- Download Docker Desktop for Mac M1 3- Try to add Docker service to Intellij
I didn't do anything else because I think Docker Desktop is enough to configure everything on Mac. I am trying to run an FT on intellij and I get the error
[main] ERROR o.t.d.DockerClientProviderStrategy - Could not find a valid Docker environment. Please check configuration. Attempted configurations were:
[main] ERROR o.t.d.DockerClientProviderStrategy - UnixSocketClientProviderStrategy: failed with exception InvalidConfigurationException (Could not find unix domain socket). Root cause NoSuchFileException (/var/run/docker.sock)
[main] ERROR o.t.d.DockerClientProviderStrategy - DockerMachineClientProviderStrategy: failed with exception ShellCommandException (Exception when executing docker-machine status ). Root cause InvalidExitValueException (Unexpected exit value: 1, allowed exit values: [0], executed command [docker-machine, status, ], output was 122 bytes:
Docker machine "" does not exist. Use "docker-machine ls" to list machines. Use "docker-machine create" to add a new one.)
[main] ERROR o.t.d.DockerClientProviderStrategy - As no valid configuration was found, execution cannot continue
I've been trying everything for the last 2 days but I can't seem to find a solution.
EDITED 2022-10-31
As per the latest release notes for Docker Desktop (4.13.1), there is no need to create the symlink anymore, citing notes:
Added back the /var/run/docker.sock symlink on Mac by default, to increase compatibility with tooling like tilt and docker-py. Fixes docker/for-mac#6529.
The official fix now is to UPGRADE your Docker Desktop installation.
For the Docker Desktop (4.13.0) version:
By default Docker will not create the /var/run/docker.sock symlink on the host and use the docker-desktop CLI context instead. (see: https://docs.docker.com/desktop/release-notes/)
That will prevent IntelliJ from finding Docker using the default context.
You can see the current contexts in your machine by running docker context ls, which should produce an output like:
NAME TYPE DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT ORCHESTRATOR
default moby Current DOCKER_HOST based configuration unix:///var/run/docker.sock https://kubernetes.docker.internal:6443 (default) swarm
desktop-linux * moby unix:///Users/<USER>/.docker/run/docker.sock
As a workaround that will allow IntelliJ to connect to Docker you can use the TCP Socket checkbox and put in the Engine API URL the value that appears under DOCKER ENDPOINT in the active context.
The case for this example will be: unix:///Users/<USER>/.docker/run/docker.sock
Then IntelliJ will be able to connect to Docker Desktop.
**Hacky option**
Another way to make IntelliJ (and other components that rely on the default config) to find Docker will be to manually create a symlink to the new DOCKER ENDPOINT by running:
sudo ln -svf /Users/<USER>/.docker/run/docker.sock /var/run/docker.sock
In that way all the components looking for Docker under /var/run/docker.sock will find it.
I have an ECS Fargate service running the jetbrains/teamcity-agent image. This is connected to my TeamCity Host which is running on an EC2 instance(windows).
When I check whether the agent is capable of running docker commands, it shows the following errors:
Unmet requirements:
docker.server.osType contains linux
docker.server.version exists
Under Agent Parameters -> Configuration Parameters, I can see the docker version and the dockerCompose.version properly. Is there a setting that I am missing?
If you are trying to access a docker socket in fargate, Fargate does not support running docker commands, there is a proposed ticket for this feature.
the issue with "docker.server.osType" not showing up usually means
that the docker command run from the agent cannot connect with the
docker daemon running. This is usually due to a lack of permissions,
as docker by default only allows connections from root and users of
the group docker
Teamcity-Unmet-requirements-docker-server-osType-contains-linux
I was facing similar issues got them fixed by adding "build agent" user in "docker" group and restarted/rebooted the server.
Where build agent user ==> Means the user with which your TeamCity services are running.
Command to add a user to group
#chmod -a -G docker <userasperyourrequirement>
Command to reboot the server:
#init 6
I am trying to configure jenkins slave as docker container, have enabled docker API and connections works fine to the API
Have added the configuration for docker template and docker cloud but it seems that my job does not starts
I can see container getting created on my docker node but the job does not start
Docker cloud configuration image
docker template image
One thing to note is that when i run the container specifically on the docker node and then try to ssh using the same credentials that i am using in jenkins i can ssh into the container.
This message of "Jenkins doesn't have label XXXX" is rather misleading and unhelpful.
You think the problem is something you did wrong in your configuration and when you find out what happen it is nothing to do with jenkins or how you set up the docker plugin.
I run into the same problem than you, and the problem was the docker installation I was using.
The steps I followed to fix it were:
(I was using CENTOS7,jenkins 2.1.38, docker version 1.13.1)
1) Go to the logs of your jenkins (centos logs are /var/log/jenkins.log)
2) Looking into the logs you are going to find out the problem. For instance for me was this:
com.github.dockerjava.api.exception.NotFoundException: {"message":"driver failed programming external connectivity on endpoint happy_heyrovsky (cbfa0d43f8c89d2531323249468503be11e9dd603597a870530d28540c662695): exec: \"docker-proxy\": executable file not found in $PATH"}
As you see the problem is that docker it is not able to find docker-proxy ¿how to fix this?
Go to /usr/libexec/docker and you will see docker-proxy-current. so what you have to do is create a link:
sudo ln -s docker-proxy-current docker-proxy
Tha´s all. After doing this change I execute my build on jenkins and it works.
I'm playing around now with docker 1.12, created a service and noticed there is a stage of "preparing" when I ran "docker service tasks xxx".
I can only guess that on this stage the images are being pulled or updated.
My question is: how can I see the logs for this stage? Or more generally: how can I see the logs for docker service tasks?
I have been using docker-machine for emulating different "hosts" in my development environment.
This is what I did to figure out what was going on during this "Preparing" phase for my services:
docker service ps <serviceName>
You should see the nodes (machines) where your service was scheduled to run. Here you'll see the "Preparing" message.
Use docker-machine ssh to connect to a particular machine:
docker-machine ssh <nameOfNode/Machine>
Your prompt will change. You are now inside another machine.
Inside this other machine do this:
tail -f /var/log/docker.log
You'll see the "daemon" log for that machine.
There you'll see if that particular daemon is doing the "pull" or what's is doing as part of the service preparation.
In my case, I found something like this:
time="2016-09-05T19:04:07.881790998Z" level=debug msg="pull progress map[progress:[===========================================> ] 112.4 MB/130.2 MB status:Downloading
Which made me realise that it was just downloading some images from my docker account.
Your assumption (about pulling during preparation) is correct.
There is no log command yet for tasks, but you could certainly connect to that daemon and do docker logs in the regular way.
I am trying to run Tomcat in a Docker container with limited success. After I tried various things, I wanted to "reset" without completely deleting everything. I did stop and remove the virtual machine from the Virtualbox console. I then tried docker-machine create and docker-machine restart. My question is, if things reach a state in which the application appears to be hanging, what is the best procedure for starting from scratch that does not involve, for example, actually rebuilding the Docker container?
EDIT: All I am now asking is, given that "docker version" returns Client information but when it reaches the Server information I get the "An error occurred trying to connect" message, is what now needs to be done? What is it not connecting to? I tried with apparent success "docker-machine restart" but got no further with "docker version" after that.
First, don't delete the boot2docker VM itself (created by docker-machine)
If you want to reset, you might have to delete the container and image (quickly rebuilt with a docker build). But you can stay in the same docker-based boot2docker VM. No need for deletion.
Retrying a docker container session simply involve killing/removing the current container, and doing a new docker run.
Then, don't forget check what is not working: does a docker ps -a shows your container running? Can you access Tomcat from the boot2docker Linux host? From your actual OS host?
Based on that diagnostic and the exact content of your Dockerfile, you will be able to debug the issue.
The main issue might come from the fact docker command are executed from outside the VM.
That works only if the commands from docker-machine env <machine-name> are set.
See docker-machine env:
For cmd.exe:
$ docker-machine.exe env --shell cmd dev
set DOCKER_TLS_VERIFY=1
set DOCKER_HOST=tcp://192.168.99.101:2376
set DOCKER_CERT_PATH=C:\Users\captain\.docker\machine\machines\dev
set DOCKER_MACHINE_NAME=dev
# Run this command to configure your shell: copy and paste the above values into your command prompt.
(replace "dev" by the name of your docker machine here, probably "default")
But it is also perfectly fine to make all docker command from within the VM. No "env" to set.
Everything is on the VM (images, Dockerfile which can be on the Windows host as well, as long as it is under C:\Users\<yourLogin>, since that folder is automatically mounted as /c/Users/<yourLogin>)