unbuntu 16.04, ram 1gb, on aws instance
I had to run old instance of elasticsearch so I wanted use a docker image of elasticsearch 5.3.3 version. by seeing the stackoverflow multiple links with same title, i have modified my installation of docker image based elasticsearch as below
sudo docker run -p 9200:9200 -p 9300:9300 -d -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" -e "xpack.security.enabled=false" --restart=unless-stopped --name careerassistant-elastic docker.elastic.co/elasticsearch/elasticsearch:5.3.3
the installation is finished and have problem with accessing the elasticsearch, though I had mutliple modifications as above command, I couldnt resolve the issue. when on
sudo docker ps
the status is still --> restarting(1) 48 seconds ago
when i was checking the log of the docker i couldnt understand anything as i am new to docker and its utilization
> docker logs --tail 50 --follow --timestamps careerassistant-elastic
i got the following output
2020-05-04T09:36:00.552415247Z CmaTotal: 0 kB
2020-05-04T09:36:00.552418314Z CmaFree: 0 kB
2020-05-04T09:36:00.552421364Z HugePages_Total: 0
2020-05-04T09:36:00.552424343Z HugePages_Free: 0
2020-05-04T09:36:00.552427401Z HugePages_Rsvd: 0
2020-05-04T09:36:00.552430358Z HugePages_Surp: 0
2020-05-04T09:36:00.552433336Z Hugepagesize: 2048 kB
2020-05-04T09:36:00.552436334Z DirectMap4k: 67584 kB
2020-05-04T09:36:00.552439415Z DirectMap2M: 980992 kB
2020-05-04T09:36:00.552442390Z
2020-05-04T09:36:00.552445460Z
2020-05-04T09:36:00.552448777Z CPU:total 1 (initial active 1) (1 cores per cpu, 1 threads per core) family 6 model 63 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, lzcnt, tsc, bmi1, bmi2
2020-05-04T09:36:00.552452312Z
2020-05-04T09:36:00.552455227Z /proc/cpuinfo:
2020-05-04T09:36:00.552458471Z processor : 0
2020-05-04T09:36:00.552461695Z vendor_id : GenuineIntel
2020-05-04T09:36:00.552464872Z cpu family : 6
2020-05-04T09:36:00.552467992Z model : 63
2020-05-04T09:36:00.552471311Z model name : Intel(R) Xeon(R) CPU E5-2676 v3 # 2.40GHz
2020-05-04T09:36:00.552474616Z stepping : 2
2020-05-04T09:36:00.552477715Z microcode : 0x43
2020-05-04T09:36:00.552480781Z cpu MHz : 2400.040
2020-05-04T09:36:00.552483934Z cache size : 30720 KB
2020-05-04T09:36:00.552486978Z physical id : 0
2020-05-04T09:36:00.552490023Z siblings : 1
2020-05-04T09:36:00.552493103Z core id : 0
2020-05-04T09:36:00.552496146Z cpu cores : 1
2020-05-04T09:36:00.552511390Z apicid : 0
2020-05-04T09:36:00.552515457Z initial apicid : 0
2020-05-04T09:36:00.552518523Z fpu : yes
2020-05-04T09:36:00.552521677Z fpu_exception : yes
2020-05-04T09:36:00.552524702Z cpuid level : 13
2020-05-04T09:36:00.552527802Z wp : yes
2020-05-04T09:36:00.552531691Z flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single kaiser fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt
2020-05-04T09:36:00.552535638Z bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit
2020-05-04T09:36:00.552538954Z bogomips : 4800.08
2020-05-04T09:36:00.552545171Z clflush size : 64
2020-05-04T09:36:00.552548419Z cache_alignment : 64
2020-05-04T09:36:00.552551514Z address sizes : 46 bits physical, 48 bits virtual
2020-05-04T09:36:00.552554916Z power management:
2020-05-04T09:36:00.552558030Z
2020-05-04T09:36:00.552561090Z
2020-05-04T09:36:00.552564141Z
2020-05-04T09:36:00.552567135Z Memory: 4k page, physical 1014424k(76792k free), swap 0k(0k free)
2020-05-04T09:36:00.552570458Z
2020-05-04T09:36:00.552573441Z vm_info: OpenJDK 64-Bit Server VM (25.131-b11) for linux-amd64 JRE (1.8.0_131-b11), built on Jun 16 2017 13:51:29 by "buildozer" with gcc 6.3.0
2020-05-04T09:36:00.552576947Z
2020-05-04T09:36:00.552579894Z time: Mon May 4 09:36:00 2020
2020-05-04T09:36:00.552582956Z elapsed time: 0 seconds (0d 0h 0m 0s)
2020-05-04T09:36:00.552586052Z
can someone help me out to figure what could be the problem for docker status to be restarting ?
I run my docker container on AWS ec2 t2.small which has 2 GB RAM as t2.micro memory(1GB) isn't enough for running the Elasticsearch container, so it should be fine for you as well until you have configured a lot more things. Looked into your logs but I don't see any error, hence difficult to debug without your docker-file.
Below is my docker-compose file for running Elasticsearch 7.6 in a docker container in AWS t2.small instance, let me know if it doesn't work for you and happy to help further.
version: '2.2'
services:
#Elasticsearch Docker Images: https://www.docker.elastic.co/
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.6.0
container_name: elasticsearch
environment:
- xpack.security.enabled=false
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
cap_add:
- IPC_LOCK
volumes:
- elasticsearch-data:/usr/share/elasticsearch/data
ports:
- 9200:9200
- 9300:9300
volumes:
elasticsearch-data:
driver: local
You can run it using docker-compose up -d -e "discovery.type=single-node" command. and refer my this Elasticsearch docker container in non-prod mode to eliminate vm.max_map_count=262144 requirement answer if you face any memory-related issue like vm.max_map_count=262144 requirement
Related
I've run mongodb service via docker-compose like this:
version: '2'
services:
mongo:
image: mongo
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: example
mem_limit: 4GB
If I run docker stats I can see 4 GB allocated:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
cf3ccbd17464 michal_mongo_1 0.64% 165.2MiB / 4GiB 4.03% 10.9kB / 4.35kB 0B / 483kB 35
But I run this command I get RAM from my laptop which is 32 GB:
~$ docker exec michal_mongo_1 free -g
total used free shared buff/cache available
Mem: 31 4 17 0 9 24
Swap: 1 0 1
How does mem_limit affect the memory size then?
free (and other utilities like top) will not report correct numbers inside a memory-constraint container because it gathers its information from /proc/meminfo which is not namespaced.
If you want the actual limit, you must use the entries populated by cgroup pseudo-filesystem under /sys/fs/cgroup.
For example:
docker run --rm -i --memory=128m busybox cat /sys/fs/cgroup/memory/memory.limit_in_bytes
The real-time usage information is available under /sys/fs/cgroup/memory/memory.stat.
You will probably need the resident-set-size (rss), for example (inside the container):
grep -E -e '^rss\s+' /sys/fs/cgroup/memory/memory.stat
For a more in-depth explanation, see also this article
Docker is running in privileged mode.
I want to know if this behavior is expected.
I am running DPDK based application in container.
My server has total 128G memory, I have limited container memory resource to 4G.
which I can see in docker stats.
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS [0/18152]
4deda4634b22 my_docker 38.12% 1.455GiB / 4GiB 36.37% 1.53kB / 0B 1.94GB / 755MB 69
I am seeing that even after docker memory is constraint to 4G.
application is able to allocate 32G huge pages memory along with other non huge page memory.
Is it expected?
Does docker run --memory option work only with non-huge page memory?
root#server# docker exec -ti my_docker bash
root#4deda4634b22:/#
root#4deda4634b22:/# ps aux |grep riot
root 893 17.2 0.0 68345740 105260 pts/0 Sl 05:54 1:02 /app/riot <<<<<< application.
root#4deda4634b22:/# cat /proc/meminfo |grep -i huge
AnonHugePages: 909312 kB
ShmemHugePages: 0 kB
**HugePages_Total: 32**
**HugePages_Free: 0**
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 1048576 kB
root#4deda4634b22:/# ls -rlt /mnt/huge/* | wc -l
32
I normally pass the access for huge page and vfio devices via docker run -it --privileged -v /sys/bus/pci/drivers:/sys/bus/pci/drivers -v /sys/kernel/mm/hugepages:/sys/kernel/mm/hugepages -v /sys/devices/system/node:/sys/devices/system/node -v /dev:/dev.
It looks like you are missing the same.
I'm running Kubernetes/Docker on Google Container Optimized OS on a GCE instance. When I run docker info it says
$ docker info
Containers: 116
Running: 97
Paused: 0
Stopped: 19
Images: 8
Server Version: 1.11.2
Storage Driver: overlay
Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 4.4.21+
Operating System: Container-Optimized OS from Google
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 14.67 GiB
Name: REDACTED
ID: REDACTED
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
The last line says that there is no swap limit support. I'm having trouble figuring out how to enable swap limit support. I found instructions for Ubuntu/Debian here.
My problem is that my docker containers get OOMKilled as soon as they reach their memory limit instead of trying swapping. I want the containers to use swap as a buffer instead of dying immediately.
Container-Optimized OS (COS) is actually configured with swap disabled completely. You can verify this via running cat /proc/meminfo | grep SwapTotal in a COS VM, which will say that it is configured to 0 kB.
I'm not sure whether it's a good idea to enable swap in your environment, as it may cause more problems (e.g. disk IO starvation/slowdown, kernel hung) if you are using swap frequently.
But if you wanna try it out, these commands might help you (run all of them as root):
cos-swap / # sysctl vm.disk_based_swap=1
vm.disk_based_swap = 1
cos-swap / # fallocate -l 1G /var/swapfile
cos-swap / # chmod 600 /var/swapfile
cos-swap / # mkswap /var/swapfile
Setting up swapspace version 1, size = 1024 MiB (1073737728 bytes)
no label, UUID=406d3dfc-3780-44bf-8add-d19a24fdbbbb
cos-swap / # swapon /var/swapfile
cos-swap / # cat /proc/meminfo | grep Swap
SwapCached: 0 kB
SwapTotal: 1048572 kB
SwapFree: 1048572 kB
I'm running my rethinkdb container in Kubernetes cluster. Below is what I notice:
Running top in the host which is CoreOS, rethinkdb process takes about 3Gb:
$ top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
981 root 20 0 53.9m 34.5m 20.9m S 15.6 0.4 1153:34 hyperkube
51139 root 20 0 4109.3m 3.179g 22.5m S 15.0 41.8 217:43.56 rethinkdb
579 root 20 0 707.5m 76.1m 19.3m S 2.3 1.0 268:33.55 kubelet
But running docker stats to check the rethinkdb container, it takes about 7Gb!
$ docker ps | grep rethinkdb
eb9e6b83d6b8 rethinkdb:2.1.5 "rethinkdb --bind al 3 days ago Up 3 days k8s_rethinkdb-3.746aa_rethinkdb-rc-3-eiyt7_default_560121bb-82af-11e5-9c05-00155d070266_661dfae4
$ docker stats eb9e6b83d6b8
CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O
eb9e6b83d6b8 4.96% 6.992 GB/8.169 GB 85.59% 0 B/0 B
$ free -m
total used free shared buffers cached
Mem: 7790 7709 81 0 71 3505
-/+ buffers/cache: 4132 3657
Swap: 0 0 0
Can someone explain why the container is taking a lot more memory than the rethinkdb process itself?
I'm running docker v1.7.1, CoreOS v773.1.0, kernel 4.1.5
In top command, your are looking at physical memory amount. in stats command, this also include the disk cached ram, so it's always bigger than the physical amount of ram. When you really need more RAM, the disk cached will be released for the application to use.
In deed, the memmory usage is pulled via cgroup memory.usage_in_bytes, you can access it in /sys/fs/cgroup/memory/docker/long_container_id/memory.usage_in_bytes. And acording to linux doc https://www.kernel.org/doc/Documentation/cgroups/memory.txt section 5.5:
5.5 usage_in_bytes
For efficiency, as other kernel components, memory cgroup uses some
optimization to avoid unnecessary cacheline false sharing.
usage_in_bytes is affected by the method and doesn't show 'exact'
value of memory (and swap) usage, it's a fuzz value for efficient
access. (Of course, when necessary, it's synchronized.) If you want to
know more exact memory usage, you should use RSS+CACHE(+SWAP) value in
memory.stat(see 5.2).
Ok, let's start over after a bunch of investigation. Here is what we know:
Ubuntu host
15.04 on i7 3820 (quad 3.6) and Samsung 850 pro SSD 512gb SATA 6
$ docker info
Containers: 2
Images: 101
Storage Driver: overlay
Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.19.0-28-generic
Operating System: Ubuntu 15.04
CPUs: 8
Total Memory: 15.61 GiB
Name: camacho
ID: ZOYN:QGDO:UGMJ:TDDM:WEEM:ZEHJ:4OKB:V5WR:RGCL:NOKG:F5W5:SDEL
WARNING: No swap limit support
OSX host
10.10.5 on i7 (quad 2.7) and Apple SSD 512gb SD512E SATA 6 (2+ years old)
$ docker info
Containers: 3
Images: 185
Storage Driver: aufs
Root Dir: /mnt/sda1/var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 191
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.0.9-boot2docker
Operating System: Boot2Docker 1.8.1 (TCL 6.3); master : 7f12e95 - Thu Aug 13 03:24:56 UTC 2015
CPUs: 8
Total Memory: 3.858 GiB
Name: dinghy
ID: PNNP:PI3E:CRUK:27RI:IPHW:HROF:NQA2:XKV6:VGCZ:WT7B:BZ7R:USWD
Debug mode (server): true
File Descriptors: 21
Goroutines: 54
System Time: 2015-09-24T19:16:01.715069994Z
EventsListeners: 1
Init SHA1:
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
provider=virtualbox
Observations
dockerized rspec running 2x+ faster on OSX
sysbench io results are terrible on Ubuntu
dockerized iozone results are as expected (ubuntu slightly faster)
hardware check complete, ubuntu host is on a SATA 6 cable and port
iozone results
docker run -it threadx/docker-ubuntu-iozone
$ iozone -R -l 5 -u 5 -r 4k -s 100m -F /home/f1 /home/f2 /home/f3 /home/f4 /home/f5 | tee -a /tmp/iozone_results.txt &
Summary
I'm dockerizing our test process because we need stability and concurrency. We've done a bunch of work to limit test times, and a 2x increase in times is a terrible step backwards.
Sysbench shows the ubuntu host performing 4x worse than OSX, and I have no understanding of that. dd and iozone tests show the ubuntu host to be performing as expected.
Question
Why is my rspec performance worse on the ubuntu host which has faster resources? Where should I investigate? Is sysbench io test an indicator or an anomaly? What are sources of bad performance in dockerized ubuntu?
The difference is PCIe vs SATA 6 interfaces. The latest generation of Macbook Pro PCIe is even faster.