I am running docker on my
windows 10 home
machine. So it is the older version of docker not the hyper v version.
I have setup a sql server docker container however when I run it it exits with the error
Exited (1)
When I look at the logs it says
sqlservr: This program requires a machine with at least 2000 megabytes
of memory. /opt/mssql/bin/sqlservr: This program requires a machine
with at least 2000 megabytes of memory.
However I have 8Gb of memory on my machine and I have at least 3.5 Gb free when running docker. I have tried using the --memory flag to allocate over 2Gb for the container (as the docs state that it needs 2Gb for the sql server image) but it still exits...
Does anyone know what is potentially the issue?
I had the same issue and I got it solved by using this docker image
https://hub.docker.com/r/justin2004/mssql_server_tiny
this is my docker-compose file:
services:
db:
image: justin2004/mssql_server_tiny
environment:
- ACCEPT_EULA=Y
- SA_PASSWORD=#P1ssword#
ports:
- '1433:1433'
expose:
- 1433
create .wslconfig file in your user folder
[wsl2]
memory=4GB # Limits VM memory in WSL 2 up to GB
processors=2 # Makes the WSL 2 VM use two virtual processors
Related
I am running Docker Enterprise Preview Edition on Windows Server 2019 and have managed to pull and run the docker-compose.yml file below. However shortly afterwards the container shuts down and when I run the command docker-compose logs it shows me the insufficient memory issue below:
Docker-compose file
version: '3.7'
services:
elasticsearch:
container_name: elasticsearch
# image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
image: docker.elastic.co/elasticsearch/elasticsearch:7.17.1
deploy:
resources:
limits:
cpus: 0.25
memory: 4096m
ports:
- 9200:9200
volumes:
- C:\DockerContainers\Elasticsearch\data:/usr/share/elasticsearch/data
- C:\DockerContainers\Elasticsearch\config\certs:/usr/share/elasticsearch/config/certs
environment:
- xpack.monitoring.enabled=true
- xpack.watcher.enabled=true
- ES_JAVA_OPTS=-Xms512m -Xmx512m
- discovery.type=single-node
# networks:
# - elastic
kibana:
container_name: kibana
# image: docker.elastic.co/kibana/kibana:7.9.2
image: docker.elastic.co/kibana/kibana:7.17.1
deploy:
resources:
limits:
cpus: 0.25
memory: 4096m
ports:
- 5601:5601
volumes:
- C:\DockerContainers\Elasticsearch\Kibana\config\certs:/usr/share/kibana/config/certs
depends_on:
- elasticsearch
# networks:
# - elastic
# networks:
# elastic:
# driver: nat
Docker logs
elasticsearch | # There is insufficient memory for the Java Runtime Environment to continue.
elasticsearch | # Native memory allocation (mmap) failed to map 65536 bytes for committing reserved memory.
elasticsearch | # An error report file with more information is saved as:
elasticsearch | # logs/hs_err_pid7.log
I read on the elasticsearch Docker guideline that it needs at least 4GB RAM. I have included the RAM limit in the docker compose yml file but it doesn't seem to take effect. Does anyone know how to set the memory usage for Docker which is running on Windows Server 2019?
I ran into this same issue trying to start Docker using almost exactly the same configuration as you, but doing so from Windows 10 instead of Windows Server 2019. I suspect the issue isn't the memory configuration for the Elastic containers, but rather the Docker host itself needs to be tweaked.
I'm not sure how to go about increasing memory for the Docker host when running Docker Enterprise Preview Edition, but for something like Docker Desktop, this can be done by changing the memory limit afforded to it by adjusting the Settings -> Advanced -> Memory slider. Docker Desktop may need to be restarted afterwards for this change to take effect.
Similarly, if you are running a Docker machine, like I am, it may be necessary to recreate it, but with more than the default 1GB that is allotted to new Docker machine instances. Something like this:
docker-machine create -d vmwareworkstation --vmwareworkstation-memory-size 8192 default
Swap in whatever vm type makes sense for you (e.g., VirtualBox) and remember that you need to run docker-machine env | Invoke-Expression in each console window in which you intend to run Docker commands.
I saw definite improvement after giving the Docker host more breathing room, as the memory-related errors disappeared. However, the containers still failed to start due to the following error:
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
This is a known issue (see https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#_set_vm_max_map_count_to_at_least_262144).
Again, if you're using Docker Desktop with WSL2 installed, you can run this:
wsl -d docker-desktop sysctl -w vm.max_map_count=262144
sysctl -w vm.max_map_count=262144
For a Docker machine, you'll need to ssh into it and set the vm.max_map_count directly:
docker-machine env | Invoke-Expression
docker-machine ssh default
sudo sysctl -w vm.max_map_count=262144
The above change to the Docker machine will last during your current session but will be lost after reboot. You can make it stick by:
adding sysctl -w vm.max_map_count=262144 to the /var/lib/boot2docker/bootlocal.sh file. Note this file may not exist before your changes.
running chmod +x /var/lib/boot2docker/bootlocal.sh
exit ssh and restart the Docker machine via docker-machine restart default.
to confirm the change, run docker-machine ssh default sudo sysctl vm.max_map_count. You should see it set to 262144.
After these changes, I was able to bring the elastic containers up. You can smoke test this by issuing a GET request for http://localhost:9200 in either Postman or curl. If you're using Docker machine, the ports you've set up are accessible to the machine, but not to your Windows box. To be able to connect to port 9200, you'll need to set up port forwarding via the docker-machine ssh -f -N -L 9200:localhost:9200 command.
Also, consider adding the following to your Docker compose file:
environment:
- bootstrap.memory_lock=true
ulimits:
memlock:
soft: -1
hard: -1
Hope this helps!
This is a late response but still if it's useful for anyone out there.
In fact in windows server, Docker desktop is the best option for running Docker. If you use Docker enterprise edition in windows server 2019, it has some restrictions in memory (RAM) allocation to container if you have not purchased the license keys to operate. Max to max it will allocate 9.7Mb of memory to container. You can confirm this by using below command
Docker stats <container id>
Here you cannot see the MEM_USAGE/LIMIT column at all in the output like Docker desktop.
So when your container request memory more than 9.7Mb, it will go down. Also you cannot use the deploy option in compose file to reserve memory for specific container in case of enterprise edition in windows server 2019. It will throw error 'memory reservation not supported'. However mem_limit will accept by the compose file but again it will not allocate the mentioned memory.
Note - The advantage of Docker enterprise edition is that it will always run in the background even if the user log off the server. But for Docker desktop, it will stop running upon user log off.
On Docker for Windows, I have a simple SQL Server container based on microsoft/mssql-server-windows-developer that is launched with docker-compose up via a simple docker-compose.yaml file.
Is there a way to allocate more than 1GB of memory to this container? I can do it when running the image directly or when I build my image with -m 4GB, but I can't figure out how to do this when using Docker Compose. This container needs more than 1GB of RAM to run properly and all of my research has revealed nothing helpful thus far.
I've looked into the resources configuration option, but that only applies when running under Docker Swarm, which I don't need.
In docker compose version 2.* you could use the mem_limit option as below
version: '2.4'
services:
my-svc:
image: microsoft/mssql-server-windows-developer
mem_limit: 4G
In docker compose version 3 it is replaced by the resources options which requires docker swarm.
version: '3'
services:
my-svc:
image: microsoft/mssql-server-windows-developer
deploy:
resources:
limits:
memory: 4G
There is a compatibility flag that can be used to translate the deploy section to equivalent version 2 parameters when running docker-compose --compatibility up. However this is not recommended for production deployments
From documentation
docker-compose 1.20.0 introduces a new --compatibility flag designed
to help developers transition to version 3 more easily. When enabled,
docker-compose reads the deploy section of each service’s definition
and attempts to translate it into the equivalent version 2 parameter.
Currently, the following deploy keys are translated:
resources
limits and memory reservations
replicas
restart_policy
condition and max_attempts All other keys are ignored and produce a
warning if present. You can review the configuration that will be used
to deploy by using the --compatibility flag with the config command.
We recommend against using --compatibility mode in production. Because the resulting configuration is only an approximate using non-Swarm mode properties, it may produce unexpected results.
Looking for options to set resources on non swarm mode containers?
The options described here are specific to the deploy key and swarm mode. If you want to set resource constraints on non swarm deployments, use Compose file format version 2 CPU, memory, and other resource options. If you have further questions, refer to the discussion on the GitHub issue docker/compose/4513.
You can use the docker-compose file on version 2 instead of version 3. You can use mem_limit (available on version 2) to set the memory limit. So you can use a docker-compose file like this:
version: "2.4"
services:
sql-server:
image: microsoft/mssql-server-windows-developer
environment:
- ACCEPT_EULA=Y
- SA_PASSWORD=t3st&Pa55word
mem_limit: 4GB
You can check the memory limit using docker stats.
Was also out for setting this up via docker-compose. Had a hard time figuring out why sql server worked on a new machine but not any longer on my older one. Finally recalled I had tuned the size down able to allocate in Docker Desktop. Utilizing this you find it through the settings button, Resources/Advanced. Setting Memory to 2GB resolved the issue for me.
I have cypress tests running without any issues in local.
But when I run them in a docker container, it is failing with "out of memory" error. logs - https://pastebin.com/0TEYnfqq
I saw a suggestion in this issue(cypress-io/cypress#350) to use --ipc=host but the issue keeps occurring.
During the tests are running, I see RAM usage of docker container is around 1.6GB Max, but the VM on which the docker is running has around 6GB free.
I ultimately want to run these tests in AWS Fargate, any idea what is the equivalent of --ipc=host in fargate?
Any help is much appreciated. Thank you.
"Out of memory" error you see because chrome under docker has 64MB restricted memory by default which sometimes is not enough. It has nothing about RAM. And when you run tests locally you dont have this restriction and whats why your tests are running smooth locally.
To increase this restriction run docker with 2 additional params
docker run -it --ipc=host --shm-size=1024M
docker compose
version: "3"
services:
name:
image: image_name
environment:
- SERVER_URL=http://server:8111
- AGENT_NAME=docker-agent-1
- DOCKER_IN_DOCKER=start
privileged: true
container_name: docker_agent_1
ipc: host
shm_size: 1024M
example of docker settings in TeamCity CI
Surely using Cypress configuration numTestsKeptInMemory could help reduce the memory consumption.
The default is "50" kept tests results (include heavy DOM snapshots); reduce it to 5 and see if that helps.
First, I do not know whether this issue is with Kafka or with Docker … I am a rookie regarding both topics. But I assume that it is more a Docker than a Kafka problem (in fact it will be my problem not really understanding one or the other …).
I installed Docker on a Raspberry 4 and created Docker images for Kafka and for Zookeeper; I had to create them by myself because 64-bit Raspi was not supported by any of the existing images (at least I could not find anyone). But I got them working.
Next I implemented the Kafka Streams example (Wordcount) from the Kafka documentation; it runs fine, counting the words in all the texts you push into it, keeping the numbers from all previous runs. That is somehow expected; at least it is described that way in that documentation.
So after some test runs I wanted to reset the whole thing.
I thought the easiest way to get there is to shut down the docker containers, delete the mounted folders on the host and start over.
But that does not work: the word counters are still there! Meaning the word count did not start from 0 …
Ok, next turn: not only removing the containers, but rebuild the images, too! Both, Zookeeper and Kafka, of course!
No difference! The word count from all the previous runs were retained.
Using docker system prune --volumes made no difference also …
From my limited understanding of Docker, I assumed that any runtime data is stored in the container, or in the mounted folders (volumes). So when I delete the containers and the folders on the Docker host that were mounted by the containers, I expect that any status would have gone.
Obviously not … so I missed something important here, most probably with Docker.
The docker-compose file I used:
version: '3'
services:
zookeeper:
image: tquadrat/zookeeper:latest
ports:
- "2181:2181"
- "2888:2888"
- "3888:3888"
- "8080:8080"
volumes:
- /data/zookeeper/config:/config
- /data/zookeeper/data:/data
- /data/zookeeper/datalog:/datalog
- /data/zookeeper/logs:/logs
environment:
ZOO_SERVERS: "server.1=zookeeper:2888:3888;2181"
restart: always
kafka:
image: tquadrat/kafka:latest
depends_on:
- zookeeper
ports:
- "9091:9091"
volumes:
- /data/kafka/config:/config
- /data/kafka/logs:/logs
environment:
KAFKA_LISTENERS: "INTERNAL://kafka:29091,EXTERNAL://:9091"
KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka:29091,EXTERNAL://TCON-PI4003:9091"
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT"
KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_DELETE_TOPIC_ENABLE: "true"
restart: always
The script file I used to clear out the mounted folders:
#!/bin/sh
set -eux
DATA="/data"
KAFKA_DATA="$DATA/kafka"
ZOOKEEPER_DATA="$DATA/zookeeper"
sudo rm -R "$KAFKA_DATA"
sudo rm -R "$ZOOKEEPER_DATA"
mkdir -p "$KAFKA_DATA/config" "$KAFKA_DATA/logs"
mkdir -p "$ZOOKEEPER_DATA/config" "$ZOOKEEPER_DATA/data" "$ZOOKEEPER_DATA/datalog" "$ZOOKEEPER_DATA/logs"
Any ideas?
Kafka Streams stores its own state under the "state.dir" config on the Host machine its running on. In Apache Kafka libraries, this is under /tmp. First check if you have overridden that property in your code.
As far as Docker goes, try without volumes first.
Using docker system prune --volumes made no difference also …
That would clean unattached volumes made with docker volume create or volumes: in Compose, not host-mounted directories.
As I assumed right from the beginning, the problem was mainly my lack of knowledge.
The behaviour I observed is not related to a magical data store for Docker that survives all attempts to kill it; it is not related to Docker at all.
I use those Docker images to run Zookeeper and the Kafka server on it. Then I switched back to my workstation machine, wrote that code (the "Wordcount" sample) that implements a Kafka Stream processor. When I started that in my IDE, it was executed on my local machine, accessing Kafka over the network.
My assumption was that any state was stored on the Kafka server, so that dumping that should reset the whole thing; as that does not work, I dumped also the Zookeeper, and as this was to no avail also, I removed nearly everything …
After some hints here I found that Kafka Streams processors maintain their own local state in a filesystem folder that is configured through state.dir (StreamsConfig.STATE_DIR_CONFIG) – see Configuring a Streams Application. This means that a Kafka Streams processor maintains its own local state independent from any Kafka server, and – as in my case when it runs on my local machine – also outside/unrelated to any Docker container …
According to the documentation, the default location should be /var/lib/kafka-streams, but this is not writeable in my environment – no idea where the Stream processor put its state instead.
After setting the configuration value state.dir for my Streams processor explicitly to a folder in my home directory, I could see that state on my disk, and after removing that, the word count started over with one.
A deeper look into the documentation for Kafka Streams revealed that I could have got the same with a call to KafkaStream.cleanup() before starting or after closing the stream processor (no removing files on the filesystem required).
I'm starting a new project with Symfony 3 and I want to use Docker for the development environment. We will work on this project with a dozen developers so I want to have an easy install process.
Here's my docker-compose.yml
version: '2'
services:
db:
image: mysql
ports:
- "3307:3306"
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_DATABASE: mydb
MYSQL_USER: root
MYSQL_PASSWORD: root
php:
build: ./php-fpm
expose:
- "9001"
volumes:
- .:/var/www/project
- ./var/logs:/var/www/project/app/logs
links:
- db
nginx:
build: ./nginx
ports:
- "8001:80"
links:
- php
volumes_from:
- php
volumes:
- ./var/logs/nginx/:/var/log/nginx
I installed the recent Docker for Mac application (beta). The big issue is that my symfony app is very very slow (a simple page takes more than 5 seconds). The same app with MAMP is much faster (500ms max). Is this a know issue of Docker ? How can I debug it ?
This is a known issue. Your local file system is being mounted in the Docker for Mac linux VM with osxfs, there is some additional latency when reading and writing these mounted files. For small applications this isn't too noticeable, but for larger applications that could read thousands of files on a single request it is can slow things down significantly.
Sorry for the late answer but you could install Docker CE Edge, because it supports cache mode.
Download Docker-Edge (waiting for the stable version of docker that will support cached mode)
Add the following line to your docker-compose.yml file
Blockquote
php:
volumes:
- ${SYMFONY_APP_PATH}:/var/www/symfony:cached
Replace ${SYMFONY_APP_PATH} by your own path.
Actually I'm using docker to run projects locally. To run Docker faster I used the below setup:
MAC OSX:
Docker Toolbox
Install normaly the dmg file.
Open your terminal and type:
`$ docker-machine create --driver virtualbox default `
`$ docker-machine env default`
`eval "$(docker-machine env default)"`
Now you have the docker-machine up and running, any docker-compose, docker command will run "inside the machine".
In our case "Symfony" is a large application. The docker-machine file system is under osxfs, so the application will be very slow.
docker-machine-nfs
Install with:
curl -s https://raw.githubusercontent.com/adlogix/docker-machine-nfs/master/docker-machine-nfs.sh | sudo tee /usr/local/bin/docker-machine-nfs > /dev/null && \ sudo chmod +x /usr/local/bin/docker-machine-nfs
Running
It will be necessary to type the root password
$ docker-machine-nfs default
Now your docker-machine is running under the nfs file system.
The speed will be regular.
Mapping your docker-machine to localhost
Regulary the docker container will run under 192.168.99.100:9000
Running on terminal:
$ vboxmanage modifyvm default --natpf1 "default-map,tcp,,9000,,9000'
You can access from localhost:9000
It's possible to get performance with Docker for Mac almost as fast as native shared volumes with Linux by using Mutagen. A benchmark is available here.
I created a full example for a Symfony project, it can be used for any type of project in any language.
I had a similar problem. In my case I was running a python script within a docker container and it was really slow. The way I solved this is using the "old" docker-toolbox.
It's not ideal, but worked for me
I have a detailed solution to this problem in my answer here, docker on OSX slow volumes, please check it out.
I got it where there is no slow downs and no extra software to install.
Known issue
This is known issue https://forums.docker.com/t/file-access-in-mounted-volumes-extremely-slow-cpu-bound/8076.
I won't recommend https://www.docker.com/products/docker-toolbox if you have https://www.docker.com/docker-mac.
Docker for Mac does not use VirtualBox, but rather HyperKit, a
lightweight macOS virtualization solution built on top of
Hypervisor.framework in macOS 10.10 Yosemite and higher.
https://docs.docker.com/docker-for-mac/docker-toolbox/#the-docker-for-mac-environment
My workaround
I have created workaround which may help you. I use http://docker-sync.io/ for my symfony project. Before using docker-sync page was loading 30 sec, now it's below 1 sec - https://github.com/Arkowsky/docker_symfony