uWSGI - unresponsive after reloading workers - uwsgi

I'm trying to let uWSGI refresh its workers, because they are leaking a bit of memory. However, after reloading the workers it's no longer responsive. Am I forgetting something?
uwsgi --http-socket 0.0.0.0:8000 --wsgi-file entry.py --processes 3 --master --req-logger file:/log/reqlog --logger file:/log/errlog --harakiri 15 --max-requests 3
max-requests 3 is to test the reloading:
mapped 291072 bytes (284 KB) for 3 cores
*** Operational MODE: preforking ***
2018-02-01 13:31:04,416 root [INFO] Starting
WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x17d9dc0 pid: 1 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 1)
spawned uWSGI worker 1 (pid: 11, cores: 1)
spawned uWSGI worker 2 (pid: 12, cores: 1)
spawned uWSGI worker 3 (pid: 13, cores: 1)
flask#4078cdd3df37:/app$ curl localhost:8000
{"message": "ok"}
flask#4078cdd3df37:/app$ curl localhost:8000
{"message": "ok"}
flask#4078cdd3df37:/app$ curl localhost:8000
{"message": "ok"}
flask#4078cdd3df37:/app$ curl localhost:8000
{"message": "ok"}
flask#4078cdd3df37:/app$ curl localhost:8000
{"message": "ok"}
...The work of process 11 is done. Seeya!
flask#4078cdd3df37:/app$ curl localhost:8000
worker 1 killed successfully (pid: 11)
Respawned uWSGI worker 1 (new pid: 33)
{"message": "ok"}
flask#4078cdd3df37:/app$ curl localhost:8000
{"message": "ok"}
flask#4078cdd3df37:/app$ curl localhost:8000
{"message": "ok"}
...The work of process 13 is done. Seeya!
flask#4078cdd3df37:/app$ curl localhost:8000
{"message": "ok"}
...The work of process 12 is done. Seeya!
flask#4078cdd3df37:/app$ curl localhost:8000 --max-time 10
worker 3 killed successfully (pid: 13)
Respawned uWSGI worker 3 (new pid: 40)
worker 2 killed successfully (pid: 12)
Respawned uWSGI worker 2 (new pid: 43)
curl: (28) Operation timed out after 10001 milliseconds with 0 bytes received
flask#4078cdd3df37:/app$ curl localhost:8000 --max-time 10
curl: (28) Operation timed out after 10001 milliseconds with 0 bytes received
Eg uWSGI is no longer responding (connection is alive forever, unless i use curl --max-time). How does uWSGI communicate internally? How does the master process know how to reach the workers? I think something is going wrong there.

I ran into this this same issue. It appears that when master flag is unset, this issue goes away. For those using emperor, the 'master flag' is the one in the vassal configuration.

Related

Python Error when testing uWSGI in basic way

I'm just testing uWSGI by following the quick-start guide in official web site. But, I am facing a problem.
This is what I did. It's exactly same as the steps in quick guide.
$ uwsgi --http :9090 --wsgi-file foobar.py
*** Starting uWSGI 2.0.19.1 (64bit) on [Sat Mar 12 09:28:41 2022] ***
compiled with version: Clang 11.0.0 on 18 January 2021 21:53:23
os: Darwin-21.3.0 Darwin Kernel Version 21.3.0: Wed Jan 5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_X86_64
nodename: mac-brian.local
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 16
current working directory: /Users/brian/Documents/project/uwsgi_test
detected binary path: /opt/anaconda3/envs/price_analysis/bin/uwsgi
*** WARNING: you are running uWSGI without its master process manager ***
your processes number limit is 5568
your memory page size is 4096 bytes
detected max file descriptor number: 256
lock engine: OSX spinlocks
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on :9090 fd 4
spawned uWSGI http 1 (pid: 10349)
uwsgi socket 0 bound to TCP address 127.0.0.1:59738 (port auto-assigned) fd 3
Python version: 3.9.1 | packaged by conda-forge | (default, Jan 10 2021, 02:52:42) [Clang 11.0.0 ]
Fatal Python error: init_import_site: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
File "/opt/anaconda3/envs/price_analysis/lib/python3.9/site.py", line 73, in <module>
import os
File "/opt/anaconda3/envs/price_analysis/lib/python3.9/os.py", line 29, in <module>
from _collections_abc import _check_methods
File "/opt/anaconda3/envs/price_analysis/lib/python3.9/_collections_abc.py", line 416, in <module>
class _CallableGenericAlias(GenericAlias):
TypeError: type 'types.GenericAlias' is not an acceptable base type
And this is my foobar.py:
def application(env, start_response):
start_response('200 OK', [('Content-Type','text/html')])
return [b"Hello World"]
After this I tried to connect to the http://localhost:9090 in this way.
curl -v 127.0.0.1:9090
These are the responses.
curl -v 127.0.0.1:9090
* Trying 127.0.0.1:9090...
* Connected to 127.0.0.1 (127.0.0.1) port 9090 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:9090
> User-Agent: curl/7.77.0
> Accept: */*
>
[uwsgi-http] unable to connect() to node "127.0.0.1:59738" (0 retries): Connection refused
[uwsgi-http] unable to connect() to node "127.0.0.1:59738" (1 retries): Connection refused
[uwsgi-http] unable to connect() to node "127.0.0.1:59738" (2 retries): Connection refused
[uwsgi-http] unable to connect() to node "127.0.0.1:59738" (3 retries): Connection refused
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server
I expected Hello World but I get empty messages from the server.
How can I solve this?

How to solve 504 HTTP Code in Scrapy-Splash

I use scrapy for a long time but now I need to use scrapy-splash for some reason.
I ran
docker run -it -p 8050:8050 --rm --name spider -v /etc/splash/proxy-profiles:/etc/splash/proxy-profiles scrapinghub/splash --max-timeout 3600
In /etc/splash/proxy-profiles/default.ini I write corporate proxy
Log:
[user#vir ~]$ sudo docker run -it -p 8050:8050 --rm --name spider -v /etc/splash/proxy-profiles:/etc/splash/proxy-profiles scrapinghub/splash --max-timeout 3600
2020-10-30 10:36:19+0000 [-] Log opened.
2020-10-30 10:36:19.324303 [-] Xvfb is started: ['Xvfb', ':1087332563', '-screen', '0', '1024x768x24', '-nolisten', 'tcp']
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-splash'
2020-10-30 10:36:19.496380 [-] Splash version: 3.5
2020-10-30 10:36:19.593826 [-] Qt 5.14.1, PyQt 5.14.2, WebKit 602.1, Chromium 77.0.3865.129, sip 4.19.22, Twisted 19.7.0, Lua 5.2
2020-10-30 10:36:19.594291 [-] Python 3.6.9 (default, Jul 17 2020, 12:50:27) [GCC 8.4.0]
2020-10-30 10:36:19.594536 [-] Open files limit: 1048576
2020-10-30 10:36:19.594911 [-] Can't bump open files limit
2020-10-30 10:36:19.627214 [-] proxy profiles support is enabled, proxy profiles path: /etc/splash/proxy-profiles
2020-10-30 10:36:19.627645 [-] memory cache: enabled, private mode: enabled, js cross-domain access: disabled
2020-10-30 10:36:19.880659 [-] verbosity=1, slots=20, argument_cache_max_entries=500, max-timeout=3600.0
2020-10-30 10:36:19.881199 [-] Web UI: enabled, Lua: enabled (sandbox: enabled), Webkit: enabled, Chromium: enabled
2020-10-30 10:36:19.882106 [-] Site starting on 8050
2020-10-30 10:36:19.882406 [-] Starting factory <twisted.web.server.Site object at 0x7fda701815f8>
2020-10-30 10:36:19.883106 [-] Server listening on http://0.0.0.0:8050
To find out my docker ip: ip addr show docker0 | grep -Po 'inet \K[\d.]+'
It is 172.17.0.1
I try
curl 'http://172.17.0.1:8050/render.html?url=https://www.google.com/' --noproxy "*"
This --noproxy "*" is due to a company works behind a proxy.
And I get the 200 response in Docker Log. It says that proxy I set in /etc/splash/proxy-profiles/default.ini works correct.
2020-10-30 13:15:25.586864 [-] "172.17.0.1" - - [30/Oct/2020:13:15:25 +0000] "GET /render.html?url=https://www.google.ru HTTP/1.1" 200 199991 "-" "curl/7.29.0"
2020-10-30 13:18:22.103537 [events] {"path": "/render.html", "rendertime": 0.7088587284088135, "maxrss": 214808, "load": [0.02, 0.13, 0.14], "fds": 57, "active": 0, "qsize": 0, "_id": 140576160225768, "method": "GET", "timestamp": 1604063902, "user-agent": "curl/7.29.0", "args": {"url": "https://www.google.ru", "uid": 140576160225768}, "status_code": 200, "client_ip": "172.17.0.1"}
2020-10-30 13:18:22.103956 [-] "172.17.0.1" - - [30/Oct/2020:13:18:21 +0000] "GET /render.html?url=https://www.google.ru HTTP/1.1" 200 199128 "-" "curl/7.29.0"
Great! That's what I wanted
But when I try to implement ScrapyRequest and run Scrapy with code below
yield SplashRequest(
args={
'wait': 0.5,
'timeout': 10,
'proxy': '<hidden for security reasons>',
},
splash_url="http://172.17.0.1:8050",
url="http://www.google.ru/",
endpoint='render.html',
method="GET",
callback=self.parse_splash,
headers=headers,
dont_filter=True,
)
I got error
2020-10-30 16:45:37 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-10-30 16:45:37 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.google.ru/ via http://172.17.0.1:8050/render.html> (failed 1 times): 504 Gateway Time-out
2020-10-30 16:46:37 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-10-30 16:46:39 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.google.ru/ via http://172.17.0.1:8050/render.html> (failed 2 times): 504 Gateway Time-out
2020-10-30 16:47:27 [scrapy.crawler] INFO: Received SIGINT, shutting down gracefully. Send again to force
2020-10-30 16:47:27 [scrapy.core.engine] INFO: Closing spider (shutdown)
2020-10-30 16:47:37 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-10-30 16:47:40 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET http://www.google.ru/ via http://172.17.0.1:8050/render.html> (failed 3 times): 504 Gateway Time-out
2020-10-30 16:47:40 [scrapy.core.engine] DEBUG: Crawled (504) <GET http://www.google.ru/ via http://172.17.0.1:8050/render.html> (referer: None)
Important note!
When I use just requests - it works!
When I use Scrapy requests - it works!
When I use ScrapySplash requests - it doesnt work!
Here is overriden settings of my Scrapy crawler:
custom_settings = {
'ITEM_PIPELINES': {
'spidermon.pipelines.DotcomPipeline': 400
},
'DUPEFILTER_DEBUG': True,
'HTTPERROR_ALLOWED_CODES': [503, 504],
'LOG_ENABLED': True,
'DOWNLOAD_DELAY': 0.5,
'DOWNLOADER_MIDDLEWARES': {
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': None,
},
'SPIDER_MIDDLEWARES': {
'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
},
'SPLASH_URL': "http://172.17.0.1:8050",
'DUPEFILTER_CLASS': 'scrapy_splash.SplashAwareDupeFilter',
'HTTPCACHE_STORAGE': 'scrapy_splash.SplashAwareFSCacheStorage',
}

How to run cassandra docker container from nomad?

I want to run a cassandra container from a nomad job. It seems to start, but after a few seconds it dies (it seems to be killed by nomad itself).
If I run the container from the command line, with:
docker run --name some-cassandra -p 9042:9042 -d cassandra:3.0
the container starts flawlessly. But if I create a nomad job like this:
job "cassandra" {
datacenters = ["dc1"]
type = "service"
update {
max_parallel = 1
min_healthy_time = "10s"
healthy_deadline = "5m"
progress_deadline = "10m"
auto_revert = false
canary = 0
}
migrate {
max_parallel = 1
health_check = "checks"
min_healthy_time = "10s"
healthy_deadline = "5m"
}
group "cassandra" {
restart {
attempts = 2
interval = "240s"
delay = "120s"
mode = "delay"
}
task "cassandra" {
driver = "docker"
config {
image = "cassandra:3.0"
network_mode = "bridge"
port_map {
cql = 9042
}
}
resources {
memory = 2048
cpu = 800
network {
port "cql" {}
}
}
env {
CASSANDRA_LISTEN_ADDRESS = "${NOMAD_IP_cql}"
}
service {
name = "cassandra"
tags = ["global", "cassandra"]
port = "cql"
}
}
}
}
Then it will never start. The nomad's web interface shows nothing in the stdout logs of the created allocation, and the stdin stream only shows Killed.
I know that as this is happening, docker containers are created, and removed after a few seconds. I cannot read the logs of these containers, for when I try (with docker logs <container_id>), all I get is:
Error response from daemon: configured logging driver does not support reading
And the allocation overview shows this message:
12/06/18 14:16:04 Terminated Exit Code: 137, Exit Message: "Docker container exited with non-zero exit code: 137"
According to docker:
If there is no database initialized when the container starts, then a
default database will be created. While this is the expected behavior,
this means that it will not accept incoming connections until such
initialization completes. This may cause issues when using automation
tools, such as docker-compose, which start several containers
simultaneously.
But I doubt this is the source of my problem, because I've increased the restart stanza values with no effect, and because the task fails after just a few seconds.
Not long ago I experience a somewhat similar problem, with a kafka container, that -it turns out- was not happy because it wanted more memory. But in this case I've provided generous values for memory and for CPU in the resources stanza, and it doesn't seem to make any difference.
My host OS is Arch Linux, with kernel 4.19.4-arch1-1-ARCH. I'm running consul as a systemd service, and the nomad agent with this command line:
sudo nomad agent -dev
What can I possibly be missing? Any help and/or pointers will be appreciated.
Update (2018-12-06 16:26 GMT): by reading in detail the output of the nomad agent, I get that some valuable information can be read at the host's /tmp directory. A snippet of that output:
2018/12/06 16:03:03 [DEBUG] memberlist: TCP connection from=127.0.0.1:45792
2018/12/06 16:03:03.180586 [DEBUG] driver.docker: docker pull cassandra:latest succeeded
2018-12-06T16:03:03.184Z [DEBUG] plugin: starting plugin: path=/usr/bin/nomad args="[/usr/bin/nomad executor {"LogFile":"/tmp/NomadClient073551030/1c315bf2-688c-2c7b-8d6f-f71fec1254f3/cassandra/executor.out","LogLevel":"DEBUG"}]"
2018-12-06T16:03:03.185Z [DEBUG] plugin: waiting for RPC address: path=/usr/bin/nomad
2018-12-06T16:03:03.235Z [DEBUG] plugin.nomad: plugin address: timestamp=2018-12-06T16:03:03.235Z address=/tmp/plugin681788273 network=unix
2018/12/06 16:03:03.253166 [DEBUG] driver.docker: Setting default logging options to syslog and unix:///tmp/plugin559865372
2018/12/06 16:03:03.253196 [DEBUG] driver.docker: Using config for logging: {Type:syslog ConfigRaw:[] Config:map[syslog-address:unix:///tmp/plugin559865372]}
2018/12/06 16:03:03.253206 [DEBUG] driver.docker: using 2147483648 bytes memory for cassandra
2018/12/06 16:03:03.253217 [DEBUG] driver.docker: using 800 cpu shares for cassandra
2018/12/06 16:03:03.253237 [DEBUG] driver.docker: binding directories []string{"/tmp/NomadClient073551030/1c315bf2-688c-2c7b-8d6f-f71fec1254f3/alloc:/alloc", "/tmp/NomadClient073551030/1c315bf2-688c-2c7b-8d6f-f71fec1254f3/cassandra/local:/local", "/tmp/NomadClient073551030/1c315bf2-688c-2c7b-8d6f-f71fec1254f3/cassandra/secrets:/secrets"} for cassandra
2018/12/06 16:03:03.253282 [DEBUG] driver.docker: allocated port 127.0.0.1:29073 -> 9042 (mapped)
2018/12/06 16:03:03.253296 [DEBUG] driver.docker: exposed port 9042
2018/12/06 16:03:03.253320 [DEBUG] driver.docker: setting container name to: cassandra-1c315bf2-688c-2c7b-8d6f-f71fec1254f3
2018/12/06 16:03:03.361162 [INFO] driver.docker: created container 29b0764bd2de69bda6450ebb1a55ffd2cbb4dc3002f961cb5db71b323d611199
2018/12/06 16:03:03.754476 [INFO] driver.docker: started container 29b0764bd2de69bda6450ebb1a55ffd2cbb4dc3002f961cb5db71b323d611199
2018/12/06 16:03:03.757642 [DEBUG] consul.sync: registered 1 services, 0 checks; deregistered 0 services, 0 checks
2018/12/06 16:03:03.765001 [DEBUG] client: error fetching stats of task cassandra: stats collection hasn't started yet
2018/12/06 16:03:03.894514 [DEBUG] client: updated allocations at index 371 (total 2) (pulled 0) (filtered 2)
2018/12/06 16:03:03.894584 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 2)
2018/12/06 16:03:05.190647 [DEBUG] driver.docker: error collecting stats from container 29b0764bd2de69bda6450ebb1a55ffd2cbb4dc3002f961cb5db71b323d611199: io: read/write on closed pipe
2018-12-06T16:03:09.191Z [DEBUG] plugin.nomad: 2018/12/06 16:03:09 [ERR] plugin: plugin server: accept unix /tmp/plugin681788273: use of closed network connection
2018-12-06T16:03:09.194Z [DEBUG] plugin: plugin process exited: path=/usr/bin/nomad
2018/12/06 16:03:09.223734 [INFO] client: task "cassandra" for alloc "1c315bf2-688c-2c7b-8d6f-f71fec1254f3" failed: Wait returned exit code 137, signal 0, and error Docker container exited with non-zero exit code: 137
2018/12/06 16:03:09.223802 [INFO] client: Restarting task "cassandra" for alloc "1c315bf2-688c-2c7b-8d6f-f71fec1254f3" in 2m7.683274502s
2018/12/06 16:03:09.230053 [DEBUG] consul.sync: registered 0 services, 0 checks; deregistered 1 services, 0 checks
2018/12/06 16:03:09.233507 [DEBUG] consul.sync: registered 0 services, 0 checks; deregistered 0 services, 0 checks
2018/12/06 16:03:09.296185 [DEBUG] client: updated allocations at index 372 (total 2) (pulled 0) (filtered 2)
2018/12/06 16:03:09.296313 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 2)
2018/12/06 16:03:11.541901 [DEBUG] http: Request GET /v1/agent/health?type=client (452.678µs)
But the contents of /tmp/NomadClient.../<alloc_id>/... is deceptively simple:
[root#singularity 1c315bf2-688c-2c7b-8d6f-f71fec1254f3]# ls -lR
.:
total 0
drwxrwxrwx 5 nobody nobody 100 Dec 6 15:52 alloc
drwxrwxrwx 5 nobody nobody 120 Dec 6 15:53 cassandra
./alloc:
total 0
drwxrwxrwx 2 nobody nobody 40 Dec 6 15:52 data
drwxrwxrwx 2 nobody nobody 80 Dec 6 15:53 logs
drwxrwxrwx 2 nobody nobody 40 Dec 6 15:52 tmp
./alloc/data:
total 0
./alloc/logs:
total 0
-rw-r--r-- 1 root root 0 Dec 6 15:53 cassandra.stderr.0
-rw-r--r-- 1 root root 0 Dec 6 15:53 cassandra.stdout.0
./alloc/tmp:
total 0
./cassandra:
total 4
-rw-r--r-- 1 root root 1248 Dec 6 16:19 executor.out
drwxrwxrwx 2 nobody nobody 40 Dec 6 15:52 local
drwxrwxrwx 2 nobody nobody 60 Dec 6 15:52 secrets
drwxrwxrwt 2 nobody nobody 40 Dec 6 15:52 tmp
./cassandra/local:
total 0
./cassandra/secrets:
total 0
./cassandra/tmp:
total 0
Both cassandra.stdout.0 and cassandra.stderr.0 are empty, and the full contents of the executor.out file is:
2018/12/06 15:53:22.822072 [DEBUG] syslog-server: launching syslog server on addr: /tmp/plugin278120866
2018/12/06 15:55:53.009611 [DEBUG] syslog-server: launching syslog server on addr: /tmp/plugin242312234
2018/12/06 15:58:29.135309 [DEBUG] syslog-server: launching syslog server on addr: /tmp/plugin226242288
2018/12/06 16:00:53.942271 [DEBUG] syslog-server: launching syslog server on addr: /tmp/plugin373025133
2018/12/06 16:03:03.252389 [DEBUG] syslog-server: launching syslog server on addr: /tmp/plugin559865372
2018/12/06 16:05:19.656317 [DEBUG] syslog-server: launching syslog server on addr: /tmp/plugin090082811
2018/12/06 16:07:28.468809 [DEBUG] syslog-server: launching syslog server on addr: /tmp/plugin383954837
2018/12/06 16:09:54.068604 [DEBUG] syslog-server: launching syslog server on addr: /tmp/plugin412544225
2018/12/06 16:12:10.085157 [DEBUG] syslog-server: launching syslog server on addr: /tmp/plugin279043152
2018/12/06 16:14:48.255653 [DEBUG] syslog-server: launching syslog server on addr: /tmp/plugin209533710
2018/12/06 16:17:23.735550 [DEBUG] syslog-server: launching syslog server on addr: /tmp/plugin168184243
2018/12/06 16:19:40.232181 [DEBUG] syslog-server: launching syslog server on addr: /tmp/plugin839254781
2018/12/06 16:22:13.485457 [DEBUG] syslog-server: launching syslog server on addr: /tmp/plugin406142133
2018/12/06 16:24:24.869274 [DEBUG] syslog-server: launching syslog server on addr: /tmp/plugin964077792
Update (2018-12-06 16:40 GMT): since it's apparent that logging to syslog is desirable for the agent, I've setup and launched a local syslog server, to no avail. And the syslog server receive no message whatsoever.
Problem solved. Its nature is twofold:
Nomad's docker driver is (very efficiently) encapsulating the
behaviour of the containers, making them at times very silent.
Cassandra is very demanding of resources. Much more than I
originally thought. I was convinced that 4 GB RAM would be enough for
it run comfortably, but as it turns out it needs (at least in my
environment) 6 GB.
Disclaimer: I'm actually using now bitnami/cassandra instead of cassandra, because I believe their images are of very high quality, secure and configurable by means of environment variables. This discovery I made using bitnami's image, and I haven't tested how the original one reacts to having this amount of memory.
As to why it doesn't fail when running the container directly from docker's CLI, I think that's because there is no specification of limits when running it that way. Docker simply takes as much memory as it needs for its containers, so if eventually host's memory is insufficient for all containers, the realisation will come much later (and possibly painfully). So this early failure should be a welcome benefit of an orchestration platform as nomad. If there is any complain on my part is that finding what the problem was took so long because of the lack of visibility of the container!

Using Docker with Sensu

i am a Novice in Docker and wanted to use Sensu for monitoring containers. I have set up a Sensu server and Sensu client ( where my Docker containers are running ) using the below material:
Click [here] (http://devopscube.com/monitor-docker-containers-guide/)
I get the Sensu client information in Uchiwa Dashboard while running the below command:
docker run -d --name sensu-client --privileged \
-v $PWD/load-docker-metrics.sh:/etc/sensu/plugins/load-docker-metrics.sh \
-v /var/run/docker.sock:/var/run/docker.sock \
usman/sensu-client SENSU_SERVER_IP RABIT_MQ_USER RABIT_MQ_PASSWORD CLIENT_NAME CLIENT_IP
However, when i try to fire a new container from the same host machine , i do not get the information of the client in Uchiwa Dashboard.
It would be great if anyone have used Sensu with Docker to monitor Docker containers can guide on the same.
Thanks for the time.
Please logs of the sensu-client
'Supervisord is running as root and it is searching '
2017-01-09 04:11:47,210 CRIT Supervisor running as root (no user in config file)
2017-01-09 04:11:47,212 INFO supervisord started with pid 12
2017-01-09 04:11:48,214 INFO spawned: 'sensu-client' with pid 15
2017-01-09 04:11:49,524 INFO success: sensu-client entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-01-09 04:11:49,530 INFO exited: sensu-client (exit status 0; expected)
[ec2-user#ip-172-31-0-89 sensu-client]$ sudo su
[root#ip-172-31-0-89 sensu-client]# docker logs sensu-client
/usr/lib/python2.6/site-packages/supervisor-3.1.3-py2.6.egg/supervisor/options.py:296: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (
including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
'Supervisord is running as root and it is searching '
2017-01-09 04:11:47,210 CRIT Supervisor running as root (no user in config file)
2017-01-09 04:11:47,212 INFO supervisord started with pid 12
2017-01-09 04:11:48,214 INFO spawned: 'sensu-client' with pid 15
2017-01-09 04:11:49,524 INFO success: sensu-client entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-01-09 04:11:49,530 INFO exited: sensu-client (exit status 0; expected)

uWSGI timeout during basic test in tutorial

I feel so dumb admitting this, but I am struggling on the uWSGI tutorial for Django here
My problem is after making a test.py file as described in the tutorial, and running the command:
uwsgi --http :8000 --wsgi-file test.py
I go to port :8000 on the IP adress for my VPS and the connection times out. I have been playing around with nginx and have been able to get the "Welcome to nginx" screen to successfully show itself. The output on my terminal after starting uwsgi with the above command is:
--wsgi-file test.py
*** Starting uWSGI 1.9.17.1 (64bit) on [Thu Oct 10 20:58:40 2013] ***
compiled with version: 4.6.3 on 10 October 2013 20:17:02
os: Linux-3.9.3-x86_64-linode33 #1 SMP Mon May 20 10:22:57 EDT 2013
nodename: Name
machine: x86_64
clock source: unix
detected number of CPU cores: 8
current working directory: /usr/local/uwsgi-tutorail/mytest
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** WARNING: you are running uWSGI without its master process manager ***
your processes number limit is 7883
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on :8000 fd 4
spawned uWSGI http 1 (pid: 18638)
uwsgi socket 0 bound to TCP address 127.0.0.1:52306 (port auto-assigned) fd 3
Python version: 2.7.3 (default, Sep 26 2013, 20:13:52) [GCC 4.6.3]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x26599f0
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 72792 bytes (71 KB) for 1 cores
*** Operational MODE: single process ***
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x26599f0 pid: 18637 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 18637, cores: 1)
I am a complete newb at uwsgi, any help would be greatly appreciated.
Not an elegant solution but I was able to "fix" the problem by rebuilding my VPS and starting from scratch.

Resources