I have several containers that are running on my Centos7 VM and I would like to retrieve their CPU and Memory usage using the following command:
echo -e "GET /containers/(container_name)/stats HTTP/1.0\r\n" | \
nc -U /var/run/docker.sock
However, I just receive the following message without any statistics:
HTTP/1.0 200 OK
Server: Docker/1.10.3 (linux)
Date: Sun, 22 Jan 2017 15:53:49 GMT
Content-Type: text/plain; charset=utf-8
The "containers/(container_name)/top" command works fine.
Can you please help me to understand why I don't receive this container's statistics?
Command to use get the stats of the container:
curl -X GET http://127.0.0.1:6000/containers/<container_id>/stats
The stats will be displayed for every second.
Stats can be fetched only for running containers.
Refer this :
how to configure docker daemon port.
Related
My architecture is as follows:
2 docker-dev-1 and docker-dev-2 nodes in a docker-dev VPC
2 docker-internal-1 and docker-internal-2 nodes in a docker-internal VPC
The firewall allows tcp:2377, 7946, udp:4789, 7946, esp as documented here
All of them are masters in order to facilitate testing for the moment. Docker version is 20.10.16. All the instances are exactly the same (packages, configuration...).
Currently I have a flask/jinja application running on docker-dev-X.
To connect to the database, the app passes by a reverse proxy which redirects the streams that arrives on port 3306 (MySQL) to a Cloud SQL instance of the docker-internal VPC.
The flask application is exposed via a reverse proxy that listens on port 8082.
Here is the docker daemon.json configuration:
{
"mtu": 1454,
"no-new-privileges": true
}
Everything works fine when I have only one docker-dev. However, as soon as I add the docker-dev-2 node, all streams with a large output passing through docker-dev-2 do not work.
Let me explain:
On docker-dev-1 :
dev#docker-dev-1:~$ curl localhost:8082/health
Ok
# With a heavier page
dev#docker-dev-1:~$ curl localhost:8082/auth/login
<!DOCTYPE html>
<html lang="en_GB">
<head>
... # Lots of HTMLs
</html>
No problem everything is working fine.
On docker-dev-2 :
dev#docker-dev-2:~$ curl localhost:8082/health
Ok
dev#docker-dev-2:~$ curl -I localhost:8082/health
HTTP/1.1 200 OK
Server: nginx
Date: Wed, 18 May 2022 12:34:57 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 2
...
# With a heavier page
devdocker-dev-2:~$ curl localhost:8082/auth/login
^C # Timeout
# Same curl but shows only header
dev#docker-dev-2:~$ curl -I localhost:8082/auth/login
HTTP/1.1 200 OK
Server: nginx
Date: Wed, 18 May 2022 10:43:57 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 222967 # Long Content-Length
Connection: keep-alive
...
As you can see, when I try to curl the /health --> No problem
When I try to curl /auth/login --> The request timeout, I have no answer
When I try to curl /auth/login to show only headers --> The request works
In a container, everything is working fine, on docker-dev-1 and on docker-dev-2 :
dev#docker-dev-2:~$ docker run -it --rm --name debug --network jinja_flask_network nicolaka/netshoot bash
bash-5.1# curl reverse_proxy_nginx/health:8082
Ok
bash-5.1# curl reverse_proxy_nginx:8082/auth/login
<!DOCTYPE html>
<html lang="en_GB">
<head>
... # Lots of HTMLs
</html>
So the problem doesn't seem to be in docker network.
The problem seems to be when the request output is too long.
I already reduced MTU to 1454 a few months ago to resolve a problem... (Seems to be the same problem but in docker network).
So, when the request is on docker-dev-1 --> No problem, the website is loading normally, But when the request is on docker-dev-2 --> Infinite loading results in a timeout.
I hope I was clear in my explanation, do you have any idea ?
I have a test which starts a Docker container, performs the verification (which is talking to the Apache httpd in the Docker container), and then stops the Docker container.
When I run this test locally, this test runs just fine. But when it runs on hosted VSTS, thus a hosted build agent, it cannot connect to the Apache httpd in the Docker container.
This is the .vsts-ci.yml file:
queue: Hosted Linux Preview
steps:
- script: |
./test.sh
This is the test.sh shell script to reproduce the problem:
#!/bin/bash
set -e
set -o pipefail
function tearDown {
docker stop test-apache
docker rm test-apache
}
trap tearDown EXIT
docker run -d --name test-apache -p 8083:80 httpd
sleep 10
curl -D - http://localhost:8083/
When I run this test locally, the output that I get is:
$ ./test.sh
469d50447ebc01775d94e8bed65b8310f4d9c7689ad41b2da8111fd57f27cb38
HTTP/1.1 200 OK
Date: Tue, 04 Sep 2018 12:00:17 GMT
Server: Apache/2.4.34 (Unix)
Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
ETag: "2d-432a5e4a73a80"
Accept-Ranges: bytes
Content-Length: 45
Content-Type: text/html
<html><body><h1>It works!</h1></body></html>
test-apache
test-apache
This output is exactly as I expect.
But when I run this test on VSTS, the output that I get is (irrelevant parts replaced with …).
2018-09-04T12:01:23.7909911Z ##[section]Starting: CmdLine
2018-09-04T12:01:23.8044456Z ==============================================================================
2018-09-04T12:01:23.8061703Z Task : Command Line
2018-09-04T12:01:23.8077837Z Description : Run a command line script using cmd.exe on Windows and bash on macOS and Linux.
2018-09-04T12:01:23.8095370Z Version : 2.136.0
2018-09-04T12:01:23.8111699Z Author : Microsoft Corporation
2018-09-04T12:01:23.8128664Z Help : [More Information](https://go.microsoft.com/fwlink/?LinkID=613735)
2018-09-04T12:01:23.8146694Z ==============================================================================
2018-09-04T12:01:26.3345330Z Generating script.
2018-09-04T12:01:26.3392080Z Script contents:
2018-09-04T12:01:26.3409635Z ./test.sh
2018-09-04T12:01:26.3574923Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/02476800-8a7e-4e22-8715-c3f706e3679f.sh
2018-09-04T12:01:27.7054918Z Unable to find image 'httpd:latest' locally
2018-09-04T12:01:30.5555851Z latest: Pulling from library/httpd
2018-09-04T12:01:31.4312351Z d660b1f15b9b: Pulling fs layer
[…]
2018-09-04T12:01:49.1468474Z e86a7f31d4e7506d34e3b854c2a55646eaa4dcc731edc711af2cc934c44da2f9
2018-09-04T12:02:00.2563446Z % Total % Received % Xferd Average Speed Time Time Time Current
2018-09-04T12:02:00.2583211Z Dload Upload Total Spent Left Speed
2018-09-04T12:02:00.2595905Z
2018-09-04T12:02:00.2613320Z 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 8083: Connection refused
2018-09-04T12:02:00.7027822Z test-apache
2018-09-04T12:02:00.7642313Z test-apache
2018-09-04T12:02:00.7826541Z ##[error]Bash exited with code '7'.
2018-09-04T12:02:00.7989841Z ##[section]Finishing: CmdLine
The key thing is this:
curl: (7) Failed to connect to localhost port 8083: Connection refused
10 seconds should be enough for apache to start.
Why can curl not communicate with Apache on its port 8083?
P.S.:
I know that a hard-coded port like this is rubbish and that I should use an ephemeral port instead. I wanted to get it running first wirth a hard-coded port, because that's simpler than using an ephemeral port, and then switch to an ephemeral port as soon as the hard-coded port works. And in case the hard-coded port doesn't work because the port is unavailable, the error should look different, in that case, docker run should fail because the port can't be allocated.
Update:
Just to be sure, I've rerun the test with sleep 100 instead of sleep 10. The results are unchanged, curl cannot connect to localhost port 8083.
Update 2:
When extending the script to execute docker logs, docker logs shows that Apache is running as expected.
When extending the script to execute docker ps, it shows the following output:
2018-09-05T00:02:24.1310783Z CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2018-09-05T00:02:24.1336263Z 3f59aa014216 httpd "httpd-foreground" About a minute ago Up About a minute 0.0.0.0:8083->80/tcp test-apache
2018-09-05T00:02:24.1357782Z 850bda64f847 microsoft/vsts-agent:ubuntu-16.04-docker-17.12.0-ce-standard "/home/vsts/agents/2…" 2 minutes ago Up 2 minutes musing_booth
The problem is that the VSTS build agent runs in a Docker container. When the Docker container for Apache is started, it runs on the same level as the VSTS build agent Docker container, not nested inside the VSTS build agent Docker container.
There are two possible solutions:
Replacing localhost with the ip address of the docker host, keeping the port number 8083
Replacing localhost with the ip address of the docker container, changing the host port number 8083 to the container port number 80.
Access via the Docker Host
In this case, the solution is to replace localhost with the ip address of the docker host. The following shell snippet can do that:
host=localhost
if grep '^1:name=systemd:/docker/' /proc/1/cgroup
then
apt-get update
apt-get install net-tools
host=$(route -n | grep '^0.0.0.0' | sed -e 's/^0.0.0.0\s*//' -e 's/ .*//')
fi
curl -D - http://$host:8083/
The if grep '^1:name=systemd:/docker/' /proc/1/cgroup inspects whether the script is running inside a Docker container. If so, it installs net-tools to get access to the route command, and then parses the default gw from the route command to get the ip address of the host. Note that this only works if the container's network default gw actually is the host.
Direct Access to the Docker Container
After launching the docker container, its ip addresses can be obtained with the following command:
docker container inspect --format '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}' <container-id>
Replace <container-id> with your container id or name.
So, in this case, it would be (assuming that the first ip address is okay):
ips=($(docker container inspect --format '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}' nuance-apache))
host=${ips[0]}
curl http://$host/
Hello everybody and happy new year,
I have some issues connecting to my bluemix container,
I followed this IBM's Bluemix guide
to learn how to pull a docker image to bluemix image repository, which works.
Then I executed the command to open port 9080 server side (the vm one is 5000 according to my Dockerfile)
PS: ((I have tried with -P instead of -p 9080 or -p 9080:5000 but none works in fixing this issue))
cf ic -v run -d -p 9080 --name testdebug registry.ng.bluemix.net/datainjector/esolom python testGUI.py
after a "cf ic ps" I obtain:
CONTAINER ID IMAGE
8457d4bb-247 registry.ng.bluemix.net/datainjector/esolom:latest
COMMAND PORTS NAMES
"python testGUI.py " 9080/tcp testdebug
the debug command (executed while running the image) reports me this:
DEMANDE : [2017-01-12T11:57:42+01:00]
POST /UAALoginServerWAR/oauth/token HTTP/1.1
Host: login.ng.bluemix.net
Accept: application/json
Authorization: [DONNEES PRIVEES MASQUEES]
Connection: close
Content-Type: application/x-www-form-urlencoded
User-Agent: go-cli 6.22.2+a95e24c / darwin
grant_type=refresh_token&refresh_token=eyJhbGciOiJIUzI1NiJ9.eyJqdGkiOiJmMTE5ZTI2MC0zZGE4LTQ5NzctOTI4OS05YjY1ZDcwMmM2OWQtciIsInN1YiI6ImZkMWVmM2Q3LTI2OTQtNDQ4Ni1iNjY2LWRmNTVjY2M4MzVmOCIsInNjb3BlIjpbIm9wZW5pZCIsInVhYS51c2VyIiwiY2xvdWRfY29udHJvbGxlci5yZWFkIiwicGFzc3dvcmQud3JpdGUiLCJjbG91ZF9jb250cm9sbGVyLndyaXRlIl0sImlhdCI6MTQ4NDIxMTE1NSwiZXhwIjoxNDg2ODAzMTU1LCJjaWQiOiJjZiIsImNsaWVudF9pZCI6ImNmIiwiaXNzIjoiaHR0cHM6Ly91YWEubmcuYmx1ZW1peC5uZXQvb2F1dGgvdG9rZW4iLCJ6aWQiOiJ1YWEiLCJncmFudF90eXBlIjoicGFzc3dvcmQiLCJ1c2VyX25hbWUiOiJlbW1hbnVlbC5zb2xvbUBmci5pYm0uY29tIiwib3JpZ2luIjoidWFhIiwidXNlcl9pZCI6ImZkMWVmM2Q3LTI2OTQtNDQ4Ni1iNjY2LWRmNTVjY2M4MzVmOCIsInJldl9zaWciOiI2MWNkZjM4MiIsImF1ZCI6WyJjZiIsIm9wZW5pZCIsInVhYSIsImNsb3VkX2NvbnRyb2xsZXIiLCJwYXNzd29yZCJdfQ.PIQlkKPDwxfa0c6951pO52qcAzggfPGrsCMuFl4V-eY&scope=
REPONSE : [2017-01-12T11:57:42+01:00]
HTTP/1.1 200 OK
Connection: close
Transfer-Encoding: chunked
Cache-Control: no-cache, no-store, max-age=0, must-revalidate,no-store
Content-Security-Policy: default-src 'self' www.ibm.com 'unsafe-inline';
Content-Type: application/json;charset=UTF-8
Date: Thu, 12 Jan 2017 10:57:42 GMT
Expires: 0
Pragma: no-cache,no-cache
Server: Apache-Coyote/1.1
Strict-Transport-Security: max-age=2592000 ; includeSubDomains
X-Archived-Client-Ip: 169.54.180.83
X-Backside-Transport: OK OK,OK OK
X-Client-Ip: 169.54.180.83,91.151.65.169
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Global-Transaction-Id: 3429006079
X-Powered-By: Servlet/3.1
X-Vcap-Request-Id: 8990fd56-827c-4956-696d-497922464ac0,04094b44-0ace- 4891-6ec0-c4855fd481f7
X-Xss-Protection: 1; mode=block
6f6
{"access_token":"[DONNEES PRIVEES MASQUEES]","token_type":"[DONNEES PRIVEES MASQUEES]","refresh_token":"[DONNEES PRIVEES MASQUEES]","expires_in":1209599,"scope":"cloud_controller.read password.write cloud_controller.write openid uaa.user","jti":"9619d1dd-995f-41b4-8a8a-825af8397ccb"}
0
ae4b3e08-4ba6-47d9-bf1f-30654af7fcfc
Next I bind an IP I requested with:
cf ic ip request // 169.46.18.243 was given
cf ic ip bind 169.46.18.243 testdebug
OK
The IP address was bound successfully.
And the "cf ic ps" command gave me this:
CONTAINER ID IMAGE
1afd6916-718 registry.ng.bluemix.net/datainjector/esolom:latest
COMMAND CREATED STATUS
"python testGUI.py " 5 minutes ago Running 5 minutes ago
PORTS NAMES
169.46.18.243:9080->9080/tcp testdebug
Associated logs:
cf ic logs -ft testdebug
2017-01-12T12:45:54.531512850Z /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:334: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
32017-01-12T12:45:54.531555759Z SNIMissingWarning
�2017-01-12T12:45:54.531568916Z /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:132: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
92017-01-12T12:45:54.531576871Z InsecurePlatformWarning
�2017-01-12T12:45:54.836875400Z /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:132: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
92017-01-12T12:45:54.836895214Z InsecurePlatformWarning
�2017-01-12T12:45:54.884966459Z WebSocket transport not available. Install eventlet or gevent and gevent-websocket for improved performance.
Y2017-01-12T12:45:54.889150620Z * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
So, is the fact that the same ports are attributed to host and VM are responsible for "connection: close" in the debug log? or are those two different problems?
Is "connection: close" explains that I can't connect to the web app?
Do you have an idea of how to fix this? (something to fix in the image? or an additional option in the CLI?)
Thank you for reading, your commitment and for your answers!
PS:
Clues: I'm looking for modifying the Dockerfile, I read that I have to had 2 instructions in order to integrate my docker image to Bluemix, a sleep command so that I'm sure the container is up before calling him, and apparently the "ENV PORT 3000" instruction, here is my Dockerfile, don't hesitate to review it, a simple error easily happens.
FROM ubuntu:14.04
RUN apt-get update && apt-get -y install python2.7
RUN apt-get -y install python-pip
RUN pip install Flask
RUN pip install ibmiotf
RUN pip install requests
RUN pip install flask-socketio
RUN pip install cloudant
ENV PORT=3000
EXPOSE 3000
ADD ./SIARA /opt/SIARA/
WORKDIR /opt/SIARA/
CMD (sleep 60)
CMD ["python", "testGUI.py"]
changed the Dockerfile for:
CMD sleep 60 && python testGUI.py
after seeing the #gile comment.
Even if it didn't works immediately, after 3 hours it works surprisingly well!
EDIT: seems it was just a lucky shot, I can't access to the container again, my idea is that it worked because my dockerHub rebuilt the image after commiting the changes on the Dockerfile because it worked just after I copied the images from dockerhub repo to bluemix private repo.
Does it highlight an issue? Because I don't understand why it didn't worked from again...
Pretty straightforward:
christian#christian:~/development$ docker -v
Docker version 1.6.2, build 7c8fca2
I ran these instructions to start docker.
docker run --detach --name neo4j --publish 7474:7474 \
--volume $HOME/neo4j/data:/data neo4j
Nothing exciting here; this should all just work.
But, http://localhost:7474 doesn't respond. When I jump into the container, it seems to respond just fine (see debug session). What did I miss?
christian#christian:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2d9e0d5d2f73 neo4j:latest "/docker-entrypoint. 15 minutes ago Up 15 minutes 7473/tcp, 0.0.0.0:7474->7474/tcp neo4j
christian#christian:~$ curl http://localhost:7474
^C
christian#christian:~$ time curl http://localhost:7474
^C
real 0m33.353s
user 0m0.008s
sys 0m0.000s
christian#christian:~$ docker exec -it 2d9e0d5d2f7389ed8b7c91d923af4a664471a93f805deb491b20fe14d389a3d2 /bin/bash
root#2d9e0d5d2f73:/var/lib/neo4j# curl http://localhost:7474
{
"management" : "http://localhost:7474/db/manage/",
"data" : "http://localhost:7474/db/data/"
}root#2d9e0d5d2f73:/var/lib/neo4j# exit
christian#christian:~$ docker logs 2d9e0d5d2f7389ed8b7c91d923af4a664471a93f805deb491b20fe14d389a3d2
Starting Neo4j Server console-mode...
/var/lib/neo4j/data/log was missing, recreating...
2016-03-07 17:37:22.878+0000 INFO No SSL certificate found, generating a self-signed certificate..
2016-03-07 17:37:25.276+0000 INFO Successfully started database
2016-03-07 17:37:25.302+0000 INFO Starting HTTP on port 7474 (4 threads available)
2016-03-07 17:37:25.462+0000 INFO Enabling HTTPS on port 7473
2016-03-07 17:37:25.531+0000 INFO Mounting static content at /webadmin
2016-03-07 17:37:25.579+0000 INFO Mounting static content at /browser
2016-03-07 17:37:26.384+0000 INFO Remote interface ready and available at http://0.0.0.0:7474/
I can't reproduce this. Docker 1.8.2. & 1.10.0 is OK with your case:
docker run --detach --name neo4j --publish 7474:7474 neo4j
curl -i 127.0.0.1:7474
HTTP/1.1 200 OK
Date: Tue, 08 Mar 2016 16:45:46 GMT
Content-Type: application/json; charset=UTF-8
Access-Control-Allow-Origin: *
Content-Length: 100
Server: Jetty(9.2.4.v20141103)
{
"management" : "http://127.0.0.1:7474/db/manage/",
"data" : "http://127.0.0.1:7474/db/data/"
}
Try upgrade Docker and check netfilter rules for forwarding.
Instead of making the request to localhost you'll want to use the docker-machine VM ip address, which you can determine with this command:
docker-machine inspect default | grep IPAddress
or
curl -i http://$(docker-machine ip default):7474/
The default IP address is 192.168.99.100
OK, basically I removed the volume mount in the args to docker and it works. Ultimately, I don't want an out-of-container mount anyways. Thank you #LoadAverage for cluing me in. It's still not 'right' but for my purposes I don't care.
christian#christian:~/development$ docker run --detach --name neo4j --publish 7474:7474 neo4j
6c94527816057f8ca1e325c8f9fa7b441b4a5d26682f72d42ad17614d9251170
christian#christian:~/development$ curl http://127.0.0.1:7474
{
"management" : "http://127.0.0.1:7474/db/manage/",
"data" : "http://127.0.0.1:7474/db/data/"
}
christian#christian:~/development$
So here is the situation, I have a container running built with this dockerfile:
FROM python:2-onbuild
EXPOSE 8888
CMD [ "nohup", "mock-server", "--dir=/usr/src/app", "&" ]
I run it with this command:
docker build -t mock_server .
docker run -d -p 8888:8888 --name mocky mock_server
I am using it on a mac so boot2docker is going and I hit it from the boot2docker ip on 8888. I tried boot2docker ssh and hitting the container from there. I ran docker exec -it mocky bash and ps aux shows:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.9 113316 18576 ? Ss 15:16 0:00 /usr/local/bin/python2 /usr/local/bin/mock-server --dir=/usr/src/app &
root 5 1.6 0.1 21916 3440 ? Ss 17:52 0:00 bash
root 9 0.0 0.1 19180 2404 ? R+ 17:53 0:00 ps aux
When I cURL it:
curl -I -XGET localhost:8888/__manage
HTTP/1.1 200 OK
Content-Length: 183108
Set-Cookie: flash_msg_success=; expires=Thu, 04 Sep 2014 17:54:58 GMT; Path=/
Set-Cookie: flash_msg_error=; expires=Thu, 04 Sep 2014 17:54:58 GMT; Path=/
Server: TornadoServer/4.2.1
Etag: "efdb5b362491b8e4b8347b97ccafeca02db8d27d"
Date: Fri, 04 Sep 2015 17:54:58 GMT
Content-Type: text/html; charset=UTF-8
So I the app is running inside the container but I can't get anything from outside it. What can be done here?
First guess is the python program is explicitly binding to the loopback IP address 127.0.0.1 which disallows any remote connections. Check the docs for that python mock tornado server for something like --bind=0.0.0.0 and adjust accordingly.
You can confirm if this is the case by doing a docker exec and in the container running netstat -ntlp | grep 8888 and seeing which IP is bound. If it's 127.0.0.1, that confirms that is indeed the problem.
Docker runs on top of an OS and docker machine has its own ip address. One possible reason why the port is not accessible is that you are using localhost which is trying to hit 127.0.0.1: but your docker machine might be running another ip address hence by replacing the ip address your curl should work.
$ docker-machine ip default
This should give you docker machine's ip address replace it with localhost.