I have a cloudwatch agent installed in EC2 instance and also a docker image on the instance.
From the EC2 instance, I could successfully send out logs to endpoint(0.0.0.0:25888) to cloudwatch. But when I get into the docker image using docker exec -it <container id> bash, I tried to publish same logs from inside container but it failed with following error:
2861 2022-07-21 00:11:18,686 ERROR (10.0.1.124,1385:MainThread) aws_embedded_metrics.sinks.tcp_client: Failed to connect to the socket. [Errno 111] Connection refused
2862 2022-07-21 00:11:18,686 INFO (10.0.1.124,1385:MainThread) aws_embedded_metrics.sinks.agent_sink: Parsed agent endpoint (tcp) 0.0.0.0:25888
Wondering if anyone knows the root cause here or any debugging clue? Thanks in advance!
I run into this as well. My solution (workaround?) was to:
Make sure the cloudwatch agent is listening on udp://0.0.0.0:25888 and not 127.0.0.1 (the default). The CWAgent docs I have seen don't have any examples on how to achieve this.
Once inside the container, use the Docker host IP to send messages. For me this was export AWS_EMF_AGENT_ENDPOINT=udp://172.17.0.1:25888, as I was using aws-embedded-metrics-python. YMMV depending on the underlying library that you use.
Related
I'm using Jenkins on Docker on my local Mac Machine.
And I'm running another Docker on ubuntu VirtualBox. So now, there are 2 docker machines. one is on my mac machine and one is on my Ubuntu VirtualBox machine. I'm running Jenkins on Mac Docker. Now in the Jenkins pipeline, I want to build an image on my ubuntu machine.
I've configured Jenkins docker cloud and in the docker host URL, it is connected to the ubuntu docker-machine.
But while building a new image, I'm getting the error. Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
I've tried even adding ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock
at /lib/systemd/system/docker.service
WHen i check ps -aux,
Can someone please help me out?
help is appreciated.
First personally if I had a setup like that I would not bother connecting to the remote docker and would just install a Jenkins agent on the ubuntu machine and make it talk to the Jenkins master.
But if you want to do it they way you have it set up right now we a Jenkins talking from inside out one docker host into another docker host I suggest looking into the following:
Your Jenkins master and the ubuntu machine a very isolated they might as well just be on different machines not even in the same room. Unix domain sockets, the ones that are identified by unix://* are made for communicating within a single local OS kernel, trying to bridge them into remote machine will lead to disaster.
So the only way Jenkins could communicate to the remote host is via a remote protocol like TCP. Most of the time when you install docker with the default settings it doesn't even listen to TCP at all, mostly for security reasons.
First thing you should do is to configure a docker inside of the ubuntu machine to listen on TCP port and accept connections from remote hosts. You can use netstat -nat to see if anything is listening on TCP 4243. When things are configured correctly you see the line that stats with 0.0.0.0:4243 or something like that in the output of the nestat
Second you need to make sure your the firewalls/iptables/netfilter configuration on the Ubuntu host lets in connections from outside. A good test to try is to telnet <ubuntu-ip> 4243 from a terminal session on your Mac.
Then you need to make sure you that docker networking is configured correctly so that connections from the inside of the container that is running Jenkins end up on your ubuntu box. To test you need to exec -it into your jenkins container and repeat the telnet test. On modern linuxes telnet is usually not installed, so you can use curl -vvv which will always end up with an error, so just look at the verbose output to see if the error because things cannot communicate (timeout, connection reset etc) or the error occurs because your curl tried to talk HTTP to docker and got gibberish response. In the later case you can consider things to be set up correctly.
Finally you need to tell Jenkins Docker to communicate to the remote docker via TCP. Usually that is given on the command line to your docker run, docker ps, docker exec
I've configured it by defining the slave label in my Jenkins Pipeline.
Jenkins agents run on a variety of different environments such as physical machines, virtual machines, Kubernetes clusters, and Docker images.
In your Jenkins Pipeline or In your JenkinsFile, you've to set the agent accordingly to what you're using either using Docker image or any virtual machine.
Also Thank you so much #Vlad, all the things you told me, were really helpful.
I have two docker container one with Jenkins and one is remote container. I want to Run a Jenkins job on remote container.
For that I have given the private keys in credentials, but still connection is not successful.
I am able to ping remote container using Jenkins container also I am able to ssh to remote container using Jenkins container.
See the screenshot below.
How are you connecting with remote host ?? Can you please share the ssh command ?
Can you please share output of below command from your Jenkins container :
cat /etc/hosts
There might be possibility that Jenkins container is able to connect using IP but not using the host_name. Try updating the the /etc/hosts file with remote container host name and then connect.
eg: 172.0.0.1 remote_host local_host
Also If you have used port forwarding, then you can simply connect with base server IP and different ports assigned to those containers.
I'm trying to access the contents of remote docker containers on a linux server from my windows machine by forwarding the remote docker socket over SSH. However, the localhost port I am forwarding it to just gives me {"message":"page not found"}, and the remote docker containers are not detected.
Ideally, I want to access the contents of the remote containers in VSCode. I've been following their walkthrough (https://code.visualstudio.com/docs/remote/containers-advanced#_option-2-connect-using-an-ssh-tunnel) on how to connect to a remote docker container. Per their walkthrough, I've also made sure to set "AllowStreamLocalForwarding yes" on the server's sshd_config
C:\Users\me> ssh -nNT -L localhost:23750:/var/run/docker.sock user#remote_server
goto http://localhost:23750/:
{"message":"page not found"}
Following the VSCode instructions:
In settings,
"docker.host":"tcp://localhost:23750"
however, clicking "Remote-Containers: Attach to a Running Container" only lists my local containers, to the server ones
I've not gotten any error messages from following these steps; however, I still don't have any access to the remote docker containers
Solved - I was entering the docker.host setting wrong on vscode
(I had entered
"docker.host":""tcp://localhost:23750""
instead of
"docker.host":"tcp://localhost:23750"
)
Sorry for spam
I have a redis running on my localhost and the docker container which trying connect to that redis, but it cannot do that. I receive this error in terminal:
AbortError: Stream connection ended and command aborted. It might have been processed.
One way to solve that is to use redis in another container and to link it to my existing container. But I need to have redis running on real host.
I'm trying to access Docker remote API from within a container because I need to start other containers.
The host address is 172.19.0.1, so I'm using http://172.19.0.1:2375/images/json to get the list of images (from host, http://localhost:2375/images/json works as expected.
The connection is refused, I guess because Docker (for Windows) listens on 127.0.0.1 and not on 0.0.0.0.
I've tried to change configuration (both from UI and daemon.json) adding the entry:
"hosts": ["tcp://0.0.0.0:2375"]
but the daemon fails to start. How can I access the api?
You can set DOCKER_OPTS in windows as below and try. In Windows, Docker runs inside a VM. So, you have to ssh into the VM and make the changes.
DOCKER_OPTS='-H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock'
Check if it works for you.
Update :- To ssh into the VM (assuming default is the VM name you have created using Docker toolbox), enter the following command in the Docker Quickstart Terminal,
docker-machine ssh default
You can find more details here.
You could link the host's /var/run/docker.sock within the container where you need it. This way, you don't expose the Docker Remote API via an open port.
Be aware that it does provide root-like access to docker.
-v /var/run/docker.sock:/var/run/docker.sock
You should use "tcp://host.docker.internal:2375" to connect to host machine from container. Please make sure that you can ping the "host.docker.internal" address
https://github.com/docker/for-win/issues/1976