On my Virtual Server running Debian, I have impression, that there is wrongly configured memory, even though my provider claims everything works correctly.
Even with 3GB of RAM, I keep running out of memory, even though top commands claims it still has enough memory.
Is there a way to test, that the free memory is actually usable? For instance, if I had 1,5 GB of memory, I would like to create a block of 1 GB and see that everything still works correctly.
Thanks,
Which applications are you using? There must be a reason that you are running out of memory.
Try command free:
$ free -m
total used free shared buffers cached
Mem: 3022 2973 48 0 235 1948
-/+ buffers/cache: 790 2232
Swap: 3907 0 3907
This will show you something like the above (that is a 3 GB machine of my own).
Always check the system log of your machine if you have memory issues.
# tail /var/log/syslog
Related
It happened twice now that all my containers stopped all of a sudden.
I would suspect, my VPS test server restarted somehow. Might be RAM issues maybe.
$ free -m
total used free shared buff/cache available
Mem: 987 572 67 59 347 175
Swap: 1975 551 1424
I can not seem to find a similar question on here. Most of them are concerned with how to stop containers.
Furthermore, what would be the best implementation to get notified, when a container stops?
Edit:
All processes had this status:
Exited (255)
We're running Docker containers of NiFi 1.6.0 in production and have to come across a memory leak.
Once started, the app runs just fine, however, after a period of 4-5 days, the memory consumption on the host keeps on increasing. When checked in the NiFi cluster UI, the JVM heap size used hardly around 30% but the memory on the OS level goes to 80-90%.
On running the docker starts command, we found that the NiFi docker container is consuming the memory.
After collecting the JMX metrics, we found that the RSS memory keeps growing. What could be the potential cause of this? In the JVM tab of cluster dialog, young GC also seems to be happening in a timely manner with old GC counts shown as 0.
How do we go about identifying in what's causing the RSS memory to grow?
You need to replicate that in a non-docker environment, because with docker, memory is known to raise.
As I explained in "Difference between Resident Set Size (RSS) and Java total committed memory (NMT) for a JVM running in Docker container", docker has some bugs (like issue 10824 and issue 15020) which prevent an accurate report of the memory consumed by a Java process within a Docker container.
That is why a plugin like signalfx/docker-collectd-plugin mentions (two weeks ago) in its PR -- Pull Request -- 35 to "deduct the cache figure from the memory usage percentage metric":
Currently the calculation for memory usage of a container/cgroup being returned to SignalFX includes the Linux page cache.
This is generally considered to be incorrect, and may lead people to chase phantom memory leaks in their application.
For a demonstration on why the current calculation is incorrect, you can run the following to see how I/O usage influences the overall memory usage in a cgroup:
docker run --rm -ti alpine
cat /sys/fs/cgroup/memory/memory.stat
cat /sys/fs/cgroup/memory/memory.usage_in_bytes
dd if=/dev/zero of=/tmp/myfile bs=1M count=100
cat /sys/fs/cgroup/memory/memory.stat
cat /sys/fs/cgroup/memory/memory.usage_in_bytes
You should see that the usage_in_bytes value rises by 100MB just from creating a 100MB file. That file hasn't been loaded into anonymous memory by an application, but because it's now in the page cache, the container memory usage is appearing to be higher.
Deducting the cache figure in memory.stat from the usage_in_bytes shows that the genuine use of anonymous memory hasn't risen.
The signalFX metric now differs from what is seen when you run docker stats which uses the calculation I have here.
It seems like knowing the page cache use for a container could be useful (though I am struggling to think of when), but knowing it as part of an overall percentage usage of the cgroup isn't useful, since it then disguises your actual RSS memory use.
In a garbage collected application with a max heap size as large, or larger than the cgroup memory limit (e.g the -Xmx parameter for java, or .NET core in server mode), the tendency will be for the percentage to get close to 100% and then just hover there, assuming the runtime can see the cgroup memory limit properly.
If you are using the Smart Agent, I would recommend using the docker-container-stats monitor (to which I will make the same modification to exclude cache memory).
Yes, NiFi docker has memory issues, shoots up after a while & restarts on its own. On the other hand, the non-docker works absolutely fine.
Details:
Docker:
Run it with 3gb Heap size & immediately after the start up it consumes around 2gb. Run some processors, the machine's fan runs heavily & it restarts after a while.
Non-Docker:
Run it with 3gb Heap size & it takes 900mb & runs smoothly. (jconsole)
I have a C program running in an alipine docker container. The image size is 10M on both OSX and on ubuntu.
On OSX, when I run this image, using the 'docker stats' I see it uses 1M RAM and so in the docker compose file I allocate a max of 5M within my swarm.
However, on Ubuntu 16.04.4 LTS the image is also 10M but when running it uses about 9M RAM, and I have to increase the max allocated memory in my compose file.
Why is there a such a difference in RAM usage between OSX and Ubuntu?
Even though we have different OSs, I would have thought once you are running inside a framework, then you would behave similarly on different machines. So I would have thought there should be comparable memory usage.
Update:
Thanks for the comments. So 'stats' may be inaccurate, and there are differences so best to baseline on linux. As an aside, but I think interesting, the reason for asking this question is to understand the 'under the hood' in order to tune my setup for a large number of deployed programs. Originally, when I tested I tried to allocate the smallest amount of maximum RAM on ubuntu, this resulted in a lot of disk thrashing something I didn't see or hear on my Macbook, (no hard disks!).
Some numbers which are completely my setup but also I think are interesting.
1000 docker containers, 1 C program each, 20M RAM MAX per container, Server load of 98, Server runs 4K processes in total, [1000 C programs total]
20 docker containers, 100 C programs each, 200M RAM MAX per container, Server load of 5 to 50, Server runs 2.3K processes in total, [2000 C programs total].
This all points at give your docker images a good amount of MAX RAM and it is nicer to your server to have fewer docker containers running.
So I have a simple Go web app I deployed as a Docker container. I am running a t2.small instance on AWS with CoreOS AMI.
The container is very small, only using about 10MB of memory according to docker stat:
CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O
8e230506e99a 0.00% 11.11 MB / 2.101 GB 0.53% 49.01 MB / 16.39 MB 1.622 MB / 0 B
However the CoreOS instance seems to be using a lot of memory:
$ free
total used free shared buffers cached
Mem: 2051772 1686012 365760 25388 253096 1031836
-/+ buffers/cache: 401080 1650692
Swap: 0 0 0
As you can see it's using almost 1.7GB of memory of its 2GB total memory with only about 300MB left. And this seems to be slowly getting worse.
I've had the instance running for about 3 days now and the free memory started at around 400MB after fresh launch and starting a single Docker container.
Is this something I should worry about? Or is CoreOS supposed to use so so much memory when my little Go app in a container only uses tiny 10MB.
Because a lot of that memory usage is buffers and cache. The better indicator is your application from Docker (which is likely close if it is a small Go app) and the OS total usage minux buffers and cache on the second line (which is closer to 400 MB used).
See https://unix.stackexchange.com/a/152301/6515 for a decent explanation.
I have a tutum node and it's always near full memory consumption. If I upgrade the node to more memory, it's again full.
To be more specific: Once I started with a node and, say, 6 services I can't add another one because the node is always full. I have to restart from scratch or buy another node:
> watch -n 5 free
total used free shared buffers cached
Mem: 4048280 3902288 145992 22796 310708 1334052
-/+ buffers/cache: 2257528 1790752
Swap: 0 0 0
The interesting thing is, that tutum itself does not know about that and thinks the node has more than 50% of his memory:
What may be the reason for that? There is nothing else running on the node, it's been completly provisioned via tutum.
The underlying cloud provider is digital ocean, docker version is 1.8.3.