I have cypress tests running without any issues in local.
But when I run them in a docker container, it is failing with "out of memory" error. logs - https://pastebin.com/0TEYnfqq
I saw a suggestion in this issue(cypress-io/cypress#350) to use --ipc=host but the issue keeps occurring.
During the tests are running, I see RAM usage of docker container is around 1.6GB Max, but the VM on which the docker is running has around 6GB free.
I ultimately want to run these tests in AWS Fargate, any idea what is the equivalent of --ipc=host in fargate?
Any help is much appreciated. Thank you.
"Out of memory" error you see because chrome under docker has 64MB restricted memory by default which sometimes is not enough. It has nothing about RAM. And when you run tests locally you dont have this restriction and whats why your tests are running smooth locally.
To increase this restriction run docker with 2 additional params
docker run -it --ipc=host --shm-size=1024M
docker compose
version: "3"
services:
name:
image: image_name
environment:
- SERVER_URL=http://server:8111
- AGENT_NAME=docker-agent-1
- DOCKER_IN_DOCKER=start
privileged: true
container_name: docker_agent_1
ipc: host
shm_size: 1024M
example of docker settings in TeamCity CI
Surely using Cypress configuration numTestsKeptInMemory could help reduce the memory consumption.
The default is "50" kept tests results (include heavy DOM snapshots); reduce it to 5 and see if that helps.
Related
I am running integration tests in Azure Pipelines. I spin up two Docker containers. One container holds my test project and another container has the Postgres database.
When I run the docker compose on my local machine, the tests run successfully and take about 6 minutes.
When I run the same docker containers in the pipeline, the job doesn't finish. The job is canceled because of the 60 min limit.
The job running on agent Hosted Agent ran longer than the maximum time of 60 minutes
I do not see any helpful data in the logs.
What tools/logs can I use to diagnose this issue?
It might have to do with RAM or CPU allocation.
Is there a way to do docker stats to see how many resources are allocated to docker containers?
Also, I have multiple test projects and I'm testing them (in the pipeline) one at a time. There are projects that succeeded with this setup. So this approach works, however, when it fails as described, there isn't a way forward to troubleshoot.
The pipeline:
pool:
vmImage: ubuntu-latest
stages:
- stage: Build
displayName: Docker compose build & up
jobs:
- job: Build
displayName: Build
steps:
- script: |
docker compose build --no-cache
docker compose up --abort-on-container-exit
displayName: 'Docker Compose Build & Up'
The docker compose that pipeline calls:
version: "3.8"
services:
test_service:
container_name: test_service
image: test_service_image
build:
context: .
dockerfile: Dockerfile
environment:
ASPNETCORE_ENVIRONMENT: Staging
WAIT_HOSTS: integration_test_db_server:5432
volumes:
- ./TestResults:/var/temp
depends_on:
- integration_test_db_server
deploy:
resources:
limits:
memory: 4gb
integration_test_db_server:
image: postgres
container_name: db_server
restart: always
ports:
- "2345:5432"
environment:
POSTGRES_USER: test
POSTGRES_PASSWORD: test
POSTGRES_DB: db_server
Dockerfile refernced by test_service:
FROM mcr.microsoft.com/dotnet/sdk:6.0
WORKDIR /
COPY . ./
ADD https://github.com/ufoscout/docker-compose-wait/releases/download/2.9.0/wait /wait
#RUN chmod +x /wait
RUN /bin/bash -c 'ls -la /wait; chmod +x /wait; ls -la /wait'
CMD /wait && dotnet test ./src/MyPorject/MyProject.Tests.csproj --logger trx --results-directory /var/temp
UPDATE - Jan 3rd 2023:
I was able to reproduce this on my local machine. Because the MSFT agent is limited to 2 cores, I made that same restriction in the docker-compose file.
This caused a test to run for a very long time (over 8 minutes for one test). At that time, the CPU usage was < 3%.
Running docker stats while the test is running
So restricting the number of CPU cores, causes less CPU usage? I am confused as to what's happening here.
So there was an issue with "thread pool starvation". This didn't happen on my machine because I allocated all 4 CPU cores. However, once I limited the docker container to 2 cores, the problem appeared locally and I was able to figure out the underlying cause.
So lesson learned. Try to reproduce the issue locally. Set container resources close to MSFT agent specs. In this case 2 core CPU, and 7 GB of RAM.
Also, if your tests run for a long time and never finish, you can get more information by using blame-hang-timeout flag. This will set a time limit on a test.
dotnet test <your project> --blame-hang-timeout 2min
After that time limit a "hangdump" file will be generated with information on the errors. That's how I found out about the underlying issue.
Update on 1/4
Microsoft hosted agents have limited performance running the pipelines due to the fixed hardware configurations and network service.
Microsoft-hosted agents that run Windows and Linux images are
provisioned on Azure general purpose virtual machines with a 2 core
CPU, 7 GB of RAM, and 14 GB of SSD disk space.
Agents that run macOS images are provisioned on Mac pros with a 3 core
CPU, 14 GB of RAM, and 14 GB of SSD disk space.
If you pipeline have high performance required job, it's suggested to run your pipeline via self-hosted agent or VMSS.
================================================================
Origin
I suppose that your issue could be related to the Build job timeout setting. You could check it with the screenshot below.
By the way, you could look into this doc about Parallel Job Time Durance Limit for more reference.
===============================================================
first updated
I suppose that the duration of the pipeline could be effected by multiple factors, like network health, data and files transmission speed or agent machine performance. If your task contains large quantity of single-file transmission, you could try to use archive task when uploading to agent workspace and extract task when building or testing the project.
I am running Docker Enterprise Preview Edition on Windows Server 2019 and have managed to pull and run the docker-compose.yml file below. However shortly afterwards the container shuts down and when I run the command docker-compose logs it shows me the insufficient memory issue below:
Docker-compose file
version: '3.7'
services:
elasticsearch:
container_name: elasticsearch
# image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
image: docker.elastic.co/elasticsearch/elasticsearch:7.17.1
deploy:
resources:
limits:
cpus: 0.25
memory: 4096m
ports:
- 9200:9200
volumes:
- C:\DockerContainers\Elasticsearch\data:/usr/share/elasticsearch/data
- C:\DockerContainers\Elasticsearch\config\certs:/usr/share/elasticsearch/config/certs
environment:
- xpack.monitoring.enabled=true
- xpack.watcher.enabled=true
- ES_JAVA_OPTS=-Xms512m -Xmx512m
- discovery.type=single-node
# networks:
# - elastic
kibana:
container_name: kibana
# image: docker.elastic.co/kibana/kibana:7.9.2
image: docker.elastic.co/kibana/kibana:7.17.1
deploy:
resources:
limits:
cpus: 0.25
memory: 4096m
ports:
- 5601:5601
volumes:
- C:\DockerContainers\Elasticsearch\Kibana\config\certs:/usr/share/kibana/config/certs
depends_on:
- elasticsearch
# networks:
# - elastic
# networks:
# elastic:
# driver: nat
Docker logs
elasticsearch | # There is insufficient memory for the Java Runtime Environment to continue.
elasticsearch | # Native memory allocation (mmap) failed to map 65536 bytes for committing reserved memory.
elasticsearch | # An error report file with more information is saved as:
elasticsearch | # logs/hs_err_pid7.log
I read on the elasticsearch Docker guideline that it needs at least 4GB RAM. I have included the RAM limit in the docker compose yml file but it doesn't seem to take effect. Does anyone know how to set the memory usage for Docker which is running on Windows Server 2019?
I ran into this same issue trying to start Docker using almost exactly the same configuration as you, but doing so from Windows 10 instead of Windows Server 2019. I suspect the issue isn't the memory configuration for the Elastic containers, but rather the Docker host itself needs to be tweaked.
I'm not sure how to go about increasing memory for the Docker host when running Docker Enterprise Preview Edition, but for something like Docker Desktop, this can be done by changing the memory limit afforded to it by adjusting the Settings -> Advanced -> Memory slider. Docker Desktop may need to be restarted afterwards for this change to take effect.
Similarly, if you are running a Docker machine, like I am, it may be necessary to recreate it, but with more than the default 1GB that is allotted to new Docker machine instances. Something like this:
docker-machine create -d vmwareworkstation --vmwareworkstation-memory-size 8192 default
Swap in whatever vm type makes sense for you (e.g., VirtualBox) and remember that you need to run docker-machine env | Invoke-Expression in each console window in which you intend to run Docker commands.
I saw definite improvement after giving the Docker host more breathing room, as the memory-related errors disappeared. However, the containers still failed to start due to the following error:
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
This is a known issue (see https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#_set_vm_max_map_count_to_at_least_262144).
Again, if you're using Docker Desktop with WSL2 installed, you can run this:
wsl -d docker-desktop sysctl -w vm.max_map_count=262144
sysctl -w vm.max_map_count=262144
For a Docker machine, you'll need to ssh into it and set the vm.max_map_count directly:
docker-machine env | Invoke-Expression
docker-machine ssh default
sudo sysctl -w vm.max_map_count=262144
The above change to the Docker machine will last during your current session but will be lost after reboot. You can make it stick by:
adding sysctl -w vm.max_map_count=262144 to the /var/lib/boot2docker/bootlocal.sh file. Note this file may not exist before your changes.
running chmod +x /var/lib/boot2docker/bootlocal.sh
exit ssh and restart the Docker machine via docker-machine restart default.
to confirm the change, run docker-machine ssh default sudo sysctl vm.max_map_count. You should see it set to 262144.
After these changes, I was able to bring the elastic containers up. You can smoke test this by issuing a GET request for http://localhost:9200 in either Postman or curl. If you're using Docker machine, the ports you've set up are accessible to the machine, but not to your Windows box. To be able to connect to port 9200, you'll need to set up port forwarding via the docker-machine ssh -f -N -L 9200:localhost:9200 command.
Also, consider adding the following to your Docker compose file:
environment:
- bootstrap.memory_lock=true
ulimits:
memlock:
soft: -1
hard: -1
Hope this helps!
This is a late response but still if it's useful for anyone out there.
In fact in windows server, Docker desktop is the best option for running Docker. If you use Docker enterprise edition in windows server 2019, it has some restrictions in memory (RAM) allocation to container if you have not purchased the license keys to operate. Max to max it will allocate 9.7Mb of memory to container. You can confirm this by using below command
Docker stats <container id>
Here you cannot see the MEM_USAGE/LIMIT column at all in the output like Docker desktop.
So when your container request memory more than 9.7Mb, it will go down. Also you cannot use the deploy option in compose file to reserve memory for specific container in case of enterprise edition in windows server 2019. It will throw error 'memory reservation not supported'. However mem_limit will accept by the compose file but again it will not allocate the mentioned memory.
Note - The advantage of Docker enterprise edition is that it will always run in the background even if the user log off the server. But for Docker desktop, it will stop running upon user log off.
I am running docker on my
windows 10 home
machine. So it is the older version of docker not the hyper v version.
I have setup a sql server docker container however when I run it it exits with the error
Exited (1)
When I look at the logs it says
sqlservr: This program requires a machine with at least 2000 megabytes
of memory. /opt/mssql/bin/sqlservr: This program requires a machine
with at least 2000 megabytes of memory.
However I have 8Gb of memory on my machine and I have at least 3.5 Gb free when running docker. I have tried using the --memory flag to allocate over 2Gb for the container (as the docs state that it needs 2Gb for the sql server image) but it still exits...
Does anyone know what is potentially the issue?
I had the same issue and I got it solved by using this docker image
https://hub.docker.com/r/justin2004/mssql_server_tiny
this is my docker-compose file:
services:
db:
image: justin2004/mssql_server_tiny
environment:
- ACCEPT_EULA=Y
- SA_PASSWORD=#P1ssword#
ports:
- '1433:1433'
expose:
- 1433
create .wslconfig file in your user folder
[wsl2]
memory=4GB # Limits VM memory in WSL 2 up to GB
processors=2 # Makes the WSL 2 VM use two virtual processors
We are experiencing some very strange behaviour when mounting volumes with large amount of data (e.g a million files )
The current setup:
Docker host: Container Linux by CoreOS 1465.7.0 (Ladybug)
Docker version client: 18.06.1-ce
Docker version host: 17.09.0-ce
I have tried different versions of docker and CoreOs, both newer and older releases without any differences.
The data-directory on the docker-host is probably mapped to some storage tech, im not sure about the setup here, but I can fill out with details if necessary, but from my point of view it looks like a normal folder.
The initial error happened when switching from an anonymous volume mounted through a dummy-container (docker-compose v1) to a named volume (docker-compose v3). After creating a named volume, i shut down the docker-service and does a manual copy of the files from the old volume to the new volume. This has been tested with small data amounts, and that works so it doesnt seem to be an issue with the actual moving data in the internal /var/lib/docker-domain. I am also able to recreate this issue with a new installation where I copy a decently large amount of data.
Building container with compose works fine:
myservice:
build: myservice
restart: always
ports:
- "8080:8080"
volumes:
- type: volume
source: repo
target: /home/repo
volumes:
repo:
The repo-volume beeing the volume with a lot of data. Now, when trying to up the services, I get a timeout on the up, and the service gets stuck in "Created":
ERROR: for my-servce-new_exp_1 HTTPSConnectionPool(host='xxxx.xx.xxx.xxx', port=2376): Read timed out. (read timeout=60)
ERROR: for exp HTTPSConnectionPool(host='xxx.xx.xxx.xxx', port=2376): Read timed out. (read timeout=60)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).
I have tried to increase the timeout, but something I get another timeout after a while.
Now, if I RESTART the docker-service or host now, the service is getting up and running (but doing it this way causes issues with internal dns-mappings between services)
If i up the service with an empty / small volume, it works as expected.
As a curiosity, I found something possibly releated when trying to mount the same volume to a docker-container:
docker run -it --rm --name rmytest --volume my-service-new_repo:/data ubuntu:latest
This will time out after e.g 30 minutes or so.
If I, on the other hand, adds any option to the consistency-parameter of the volume-mapping, it runs within a couple of seconds:
docker run -it --rm --name rmytest --volume my-service-new_repo:/data:consistent ubuntu:latest
I have had no success adding the same options to the compose files either, e.g
volumes:
- type: volume
source: repo
target: /home/repo
consistency: delegated
Yields the same results; timeout and not working. Any help and pointers in the right direction would be much appreciated.
As mentioned in my own comment, this was due to SeLinux and labeling of date when mounting. To avoid this issue, we had to turn of the labeling:
mycontainer:
build: mycontainer
restart: always
# To avoid issue with named volume and mounting time
security_opt:
- "label=disable"
Disclaimer: Im not 100% sure about the full consequences of using this option, but in our situation this was the feasible way of solving this for now.
I had Docker for Windows, switched to Docker toolbox and now back to Docker for Windows and I ran into the issues with Volumes.
Before volumes were working perfectly fine and my containers which running with nodemon/tsnode/CLI watching files were restarting properly on source code change, but now they don't at all, so it looks like file changes from host are not populated in the container.
This is docker-compose for one service:
api:
build:
context: ./api
dockerfile: Dockerfile-dev
volumes:
- ./api:/srv
working_dir: /srv
links:
- mongo
depends_on:
- mongo
ports:
- 3030:3030
environment:
MONGODB: mongodb://mongo:27017/api_test
labels:
- traefik.enable=true
- traefik.frontend.rule=Host:api.mydomain.localhost
This id Dockerfile-dev
FROM node:10-alpine
ENV NODE_ENV development
WORKDIR /srv
EXPOSE 3030
CMD yarn dev // simply nodemon, working when ran from host
Can anyone help with that?
C drive is shared and verified with docker run --rm -v c:/Users:/data alpine ls /data showing list of files properly.
I will really appreciate any help.
We experienced the exact same problems in our team while developing nodejs/typescript applications with Docker on top of Windows and it has always been a big pain. To be honest, though, Windows does the right thing by not propagating the change event to the containers (Linux hosts also do not propagate the fsnotify events to containers unless the change is made from within the container). So bottom line: I do not think this issue will be avoidable unless you actually change the files within the container instead of changing them on the docker host. You can achieve this with a code sync tool like docker-sync, see this page for a list of available options: https://github.com/EugenMayer/docker-sync/wiki/Alternatives-to-docker-sync
Because we struggled with such issues for a long time, a colleague and I started an open source project called DevSpace CLI: https://github.com/covexo/devspace
The DevSpace CLI can establish a reliable and super fast 2-way code sync between your local folders and folders within your dev containers (works with any Kubernetes cluster, any volume and even with ephemeral / non-persistent folders) and it is designed to work perfectly with hot reloading tools such as nodemon. Setup minikube or a cluster with a one-click installer on some public cloud, run devspace up inside your project and you will be ready to program within your DevSpace without ever having to worry about local Docker issues and hot reloading problems. Let me know if it works for you or if there is anything you are missing.
I've been stuck into this recently (Feb 2020, Docker Desktop 2.2) and nothing from the base solutions really helped.
However when I tried WSL 2 and ran my docker-compose from inside Ubuntu shell, it became to pick up the changes in the files instantly. So if someone is observing this - try to up Docker from WSL 2.