Docker: runtime/cgo: pthread_create failed: Resource temporarily unavailable - docker

I wrote a docker-compose.yml which is working on my local ubuntu 18.04 system. I ran the same yml on my ubuntu server 18.04 lts machine and got the errormessage in the title. I tried to increase TasksMax for the process but it didnt work.
I also set DefaultTasksMax=infinity in /etc/systemd/system.conf.
Some outputs:
systemctl show -p TasksMax docker:
TasksMax=infinity
systemctl status docker | grep Tasks:
└─50-TasksMax.conf (This line is not showing up on my local
machine)
Tasks: 53
This is leading me to the point that i still havent made the configs correct.
Here is the Docker output when excecuting docker-compose up -d:
ERROR: for wordpress-cert Cannot start service wordpress-cert: b'OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \\"process_linux.go:413: running prestart hook 0 caused \\\\\\"error running hook: exit status 2, stdout: , stderr: runtime/cgo: pthread_create failed: Resource temporarily unavailable\\\\\\\\nSIGABRT: abort\\\\\\\\nPC=0x7f5cd5db7e97 m=0 sigcode=18446744073709551610\\\\\\\\n\\\\\\\\ngoroutine 0 [idle]:\\\\\\\\nruntime: unknown pc 0x7f5cd5db7e97\\\\\\\\nstack: frame={sp:0x7ffe59e1f6c0, fp:0x0} stack=[0x7ffe59620d50,0x7ffe59e1fd80)\\\\\\\\n00007ffe59e1f5c0: [...] ffffffffffffffff ffffffffffffffff \\\\\\\\nruntime: unknown pc 0x7f5cd5db7e97\\\\\\\\nstack: frame={sp:0x7ffe59e1f6c0, fp:0x0} stack=[0x7ffe59620d50,0x7ffe59e1fd80)\\\\\\\\n00007ffe59e1f5c0: [...] ffffffffffffffff ffffffffffffffff \\\\\\\\n\\\\\\\\ngoroutine 1 [runnable, locked to thread]:\\\\\\\\ngithub.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Register(0xc0005e3b00, 0x55922583c640, 0xc000760b90, 0x0, 0x0)\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus/registry.go:298 +0x1e6\\\\\\\\ngithub.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xc0005e3b00, 0xc00075d000, 0x1, 0x1)\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus/registry.go:403 +0x78\\\\\\\\ngithub.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus.MustRegister(...)\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus/registry.go:152\\\\\\\\ngithub.com/docker/docker/vendor/github.com/docker/go-metrics.Register(...)\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/vendor/github.com/docker/go-metrics/register.go:8\\\\\\\\ngithub.com/docker/docker/builder/dockerfile.init.0()\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/builder/dockerfile/metrics.go:43 +0x2d3\\\\\\\\n\\\\\\\\ngoroutine 20 [syscall]:\\\\\\\\nos/signal.signal_recv(0x559222e8e556)\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/sigqueue.go:139 +0x9e\\\\\\\\nos/signal.loop()\\\\\\\\n\\\\\\\\t/usr/local/go/src/os/signal/signal_unix.go:23 +0x24\\\\\\\\ncreated by os/signal.init.0\\\\\\\\n\\\\\\\\t/usr/local/go/src/os/signal/signal_unix.go:29 +0x43\\\\\\\\n\\\\\\\\nrax 0x0\\\\\\\\nrbx 0x7f5cd6165840\\\\\\\\nrcx 0xffffffffffffffff\\\\\\\\nrdx 0x0\\\\\\\\nrdi 0x2\\\\\\\\nrsi 0x7ffe59e1f6c0\\\\\\\\nrbp 0x559224b7603d\\\\\\\\nrsp 0x7ffe59e1f6c0\\\\\\\\nr8 0x0\\\\\\\\nr9 0x7ffe59e1f6c0\\\\\\\\nr10 0x8\\\\\\\\nr11 0x246\\\\\\\\nr12 0x5592271bd1f0\\\\\\\\nr13 0x11\\\\\\\\nr14 0x559224b1c640\\\\\\\\nr15 0x0\\\\\\\\nrip 0x7f5cd5db7Creating nextcloud-cert ... error
unknown'
ERROR: for nextcloud-cert Cannot start service nextcloud-cert: b'OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \\"process_linux.go:413: running prestart hook 0 caused \\\\\\"error running hook: exit status 2, stdout: , stderr: runtime/cgo: pthread_create failed: Resource temporarily unavailable\\\\\\\\nSIGABRT: abort\\\\\\\\nPC=0x7f2376cbce97 m=0 sigcode=18446744073709551610\\\\\\\\n\\\\\\\\ngoroutine 0 [idle]:\\\\\\\\nruntime: unknown pc 0x7f2376cbce97\\\\\\\\nstack: frame={sp:0x7fff898f4ad0, fp:0x0} stack=[0x7fff890f6070,0x7fff898f50a0)\\\\\\\\n00007fff898f49d0: [...] 00007fff898f4b40 0000564eca7254a3 <runtime.persistentalloc1+515> [...] \\\\\\\\nruntime: unknown pc 0x7f2376cbce97\\\\\\\\nstack: frame={sp:0x7fff898f4ad0, fp:0x0} stack=[0x7fff890f6070,0x7fff898f50a0)\\\\\\\\n00007fff898f49d0: [...] \\\\\\\\n\\\\\\\\ngoroutine 1 [running]:\\\\\\\\nruntime.systemstack_switch()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc00006e798 sp=0xc00006e790 pc=0x564eca774450\\\\\\\\nruntime.main()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/proc.go:130 +0x67 fp=0xc00006e7e0 sp=0xc00006e798 pc=0x564eca747cd7\\\\\\\\nruntime.goexit()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00006e7e8 sp=0xc00006e7e0 pc=0x564eca776551\\\\\\\\n\\\\\\\\nrax 0x0\\\\\\\\nrbx 0x7f237706a840\\\\\\\\nrcx 0xffffffffffffffff\\\Creating wordpress-it-mueller ... error
rsp 0x7fff898f4ad0\\\\\\\\nr8 0x0\\\\\\\\nr9 0x7fff898f4ad0\\\\\\\\nr10 0x8\\\\\\\\nr11 0x246\\\\\\\\nr12 0x564ed04ff990\\\\\\\\nr13 0x11\\\\\\\\nr14 0x564ecc404640\\\\\\\\nr15 0x0\\\\\\\\nrip 0x7f2376cbce97\\\\\\\\nrflags 0x246\\\\\\\\ncs 0x33\\\\\\\\nfs 0x0\\\\\\\\ngs Creating wordpress-cert-it-mueller ... done
ERROR: for wordpress-it-mueller Cannot start service wordpress-it-mueller: b'OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \\"process_linux.go:413: running prestart hook 0 caused \\\\\\"error running hook: exit status 2, stdout: , stderr: runtime/cgo: pthread_create failed: Resource temporarily unavailable\\\\\\\\nSIGABRT: abort\\\\\\\\nPC=0x7f3d844b2e97 m=0 sigcode=18446744073709551610\\\\\\\\n\\\\\\\\ngoroutine 0 [idle]:\\\\\\\\nruntime: unknown pc 0x7f3d844b2e97\\\\\\\\nstack: frame={sp:0x7ffeff965720, fp:0x0} stack=[0x7ffeff166cc0,0x7ffeff965cf0)\\\\\\\\n00007ffeff965620: [...] ffffffffffffffff ffffffffffffffff \\\\\\\\nruntime: unknown pc 0x7f3d844b2e97\\\\\\\\nstack: frame={sp:0x7ffeff965720, fp:0x0} stack=[0x7ffeff166cc0,0x7ffeff965cf0)\\\\\\\\n00007ffeff965620: [...] ffffffffffffffff ffffffffffffffff \\\\\\\\n\\\\\\\\ngoroutine 1 [running]:\\\\\\\\nruntime.systemstack_switch()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc00006e798 sp=0xc00006e790 pc=0x55a7e9792450\\\\\\\\nruntime.main()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/proc.go:130 +0x67 fp=0xc00006e7e0 sp=0xc00006e798 pc=0x55a7e9765cd7\\\\\\\\nruntime.goexit()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00006e7e8 sp=0xc00006e7e0 pc=0x55a7e9794551\\\\\\\\n\\\\\\\\nrax 0x0\\\\\\\\nrbx 0x7f3d84860840\\\\\\\\nrcx 0xffffffffffffffff\\\\\\\\nrdx 0x0\\\\\\\\nrdi 0x2\\\\\\\\nrsi 0x7ffeff965720\\\\\\\\nrbp 0x55a7eb47c03d\\\\\\\\nrsp 0x7ffeff965720\\\\\\\\nr8 0x0\\\\\\\\Creating nextcloud ... done
0x11\\\\\\\\nr14 0x55a7eb422640\\\\\\\\nr15 0x0\\\\\\\\nrip 0x7f3d844b2e97\\\\\\\\nrflags 0x246\\\\\\\\ncs 0x33\\\\\\\\nfs 0x0\\\\\\\\ngs 0x0\\\\\\\\n\\\\\\"\\"": unknown'
ERROR: for wordpress-cert Cannot start service wordpress-cert: b'OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \\"process_linux.go:413: running prestart hook 0 caused \\\\\\"error running hook: exit status 2, stdout: , stderr: runtime/cgo: pthread_create failed: Resource temporarily unavailable\\\\\\\\nSIGABRT: abort\\\\\\\\nPC=0x7f5cd5db7e97 m=0 sigcode=18446744073709551610\\\\\\\\n\\\\\\\\ngoroutine 0 [idle]:\\\\\\\\nruntime: unknown pc 0x7f5cd5db7e97\\\\\\\\nstack: frame={sp:0x7ffe59e1f6c0, fp:0x0} stack=[0x7ffe59620d50,0x7ffe59e1fd80)\\\\\\\\n00007ffe59e1f5c0: [...] ffffffffffffffff ffffffffffffffff \\\\\\\\nruntime: unknown pc 0x7f5cd5db7e97\\\\\\\\nstack: frame={sp:0x7ffe59e1f6c0, fp:0x0} stack=[0x7ffe59620d50,0x7ffe59e1fd80)\\\\\\\\n00007ffe59e1f5c0: [...]ffffffffffffffff ffffffffffffffff \\\\\\\\n\\\\\\\\ngoroutine 1 [runnable, locked to thread]:\\\\\\\\ngithub.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Register(0xc0005e3b00, 0x55922583c640, 0xc000760b90, 0x0, 0x0)\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus/registry.go:298 +0x1e6\\\\\\\\ngithub.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xc0005e3b00, 0xc00075d000, 0x1, 0x1)\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus/registry.go:403 +0x78\\\\\\\\ngithub.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus.MustRegister(...)\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus/registry.go:152\\\\\\\\ngithub.com/docker/docker/vendor/github.com/docker/go-metrics.Register(...)\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/vendor/github.com/docker/go-metrics/register.go:8\\\\\\\\ngithub.com/docker/docker/builder/dockerfile.init.0()\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/builder/dockerfile/metrics.go:43 +0x2d3\\\\\\\\n\\\\\\\\ngoroutine 20 [syscall]:\\\\\\\\nos/signal.signal_recv(0x559222e8e556)\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/sigqueue.go:139 +0x9e\\\\\\\\nos/signal.loop()\\\\\\\\n\\\\\\\\t/usr/local/go/src/os/signal/signal_unix.go:23 +0x24\\\\\\\\ncreated by os/signal.init.0\\\\\\\\n\\\\\\\\t/usr/local/go/src/os/signal/signal_unix.go:29 +0x43\\\\\\\\n\\\\\\\\nrax 0x0\\\\\\\\nrbx 0x7f5cd6165840\\\\\\\\nrcx 0xffffffffffffffff\\\\\\\\nrdx 0x0\\\\\\\\nrdi 0x2\\\\\\\\nrsi 0x7ffe59e1f6c0\\\\\\\\nrbp 0x559224b7603d\\\\\\\\nrsp 0x7ffe59e1f6c0\\\\\\\\nr8 0x0\\\\\\\\nr9 0x7ffe59e1f6c0\\\\\\\\nr10 0x8\\\\\\\\nr11 0x246\\\\\\\\nr12 0x5592271bd1f0\\\\\\\\nr13 0x11\\\\\\\\nr14 0x559224b1c640\\\\\\\\nr15 0x0\\\\\\\\nrip 0x7f5cd5db7e97\\\\\\\\nrflags 0x246\\\\\\\\ncs 0x33\\\\\\\\nfs 0x0\\\\\\\\ngs 0x0\\\\\\\\n\\\\\\"\\"": unknown'
ERROR: for nextcloud-cert Cannot start service nextcloud-cert: b'OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \\"process_linux.go:413: running prestart hook 0 caused \\\\\\"error running hook: exit status 2, stdout: , stderr: runtime/cgo: pthread_create failed: Resource temporarily unavailable\\\\\\\\nSIGABRT: abort\\\\\\\\nPC=0x7f2376cbce97 m=0 sigcode=18446744073709551610\\\\\\\\n\\\\\\\\ngoroutine 0 [idle]:\\\\\\\\nruntime: unknown pc 0x7f2376cbce97\\\\\\\\nstack: frame={sp:0x7fff898f4ad0, fp:0x0} stack=[0x7fff890f6070,0x7fff898f50a0)\\\\\\\\n00007fff898f49d0: [...]00007fff898f4ae8 00000000ffffffff \\\\\\\\n00007fff898f4af0: 00007fff898f4b40 0000564eca7254a3 <runtime.persistentalloc1+515> \\\\\\\\n00007fff898f4b00: 0000564ece51d0f0 0000000000000000 [...] \\\\\\\\nruntime: unknown pc 0x7f2376cbce97\\\\\\\\nstack: frame={sp:0x7fff898f4ad0, fp:0x0} stack=[0x7fff890f6070,0x7fff898f50a0)\\\\\\\\n00007fff898f49d0: [...] 00007fff898f4ae8 00000000ffffffff \\\\\\\\n00007fff898f4af0: 00007fff898f4b40 0000564eca7254a3 <runtime.persistentalloc1+515> \\\\\\\\n00007fff898f4b00: 0000564ece51d0f0 0000000000000000 [...] \\\\\\\\n\\\\\\\\ngoroutine 1 [running]:\\\\\\\\nruntime.systemstack_switch()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc00006e798 sp=0xc00006e790 pc=0x564eca774450\\\\\\\\nruntime.main()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/proc.go:130 +0x67 fp=0xc00006e7e0 sp=0xc00006e798 pc=0x564eca747cd7\\\\\\\\nruntime.goexit()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00006e7e8 sp=0xc00006e7e0 pc=0x564eca776551\\\\\\\\n\\\\\\\\nrax 0x0\\\\\\\\nrbx 0x7f237706a840\\\\\\\\nrcx 0xffffffffffffffff\\\\\\\\nrdx 0x0\\\\\\\\nrdi 0x2\\\\\\\\nrsi 0x7fff898f4ad0\\\\\\\\nrbp 0x564ecc45e03d\\\\\\\\nrsp 0x7fff898f4ad0\\\\\\\\nr8 [...] 0x0\\\\\\\\ngs 0x0\\\\\\\\n\\\\\\"\\"": unknown'
ERROR: for wordpress-it-mueller Cannot start service wordpress-it-mueller: b'OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \\"process_linux.go:413: running prestart hook 0 caused \\\\\\"error running hook: exit status 2, stdout: , stderr: runtime/cgo: pthread_create failed: Resource temporarily unavailable\\\\\\\\nSIGABRT: abort\\\\\\\\nPC=0x7f3d844b2e97 m=0 sigcode=18446744073709551610\\\\\\\\n\\\\\\\\ngoroutine 0 [idle]:\\\\\\\\nruntime: unknown pc 0x7f3d844b2e97\\\\\\\\nstack: frame={sp:0x7ffeff965720, fp:0x0} stack=[0x7ffeff166cc0,0x7ffeff965cf0)\\\\\\\\n00007ffeff965620:[...] ffffffffffffffff ffffffffffffffff \\\\\\\\nruntime: unknown pc 0x7f3d844b2e97\\\\\\\\nstack: frame={sp:0x7ffeff965720, fp:0x0} stack=[0x7ffeff166cc0,0x7ffeff965cf0)\\\\\\\\n00007ffeff965620: [...] ffffffffffffffff ffffffffffffffff \\\\\\\\n\\\\\\\\ngoroutine 1 [running]:\\\\\\\\nruntime.systemstack_switch()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc00006e798 sp=0xc00006e790 pc=0x55a7e9792450\\\\\\\\nruntime.main()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/proc.go:130 +0x67 fp=0xc00006e7e0 sp=0xc00006e798 pc=0x55a7e9765cd7\\\\\\\\nruntime.goexit()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00006e7e8 sp=0xc00006e7e0 pc=0x55a7e9794551\\\\\\\\n\\\\\\\\nrax 0x0\\\\\\\\nrbx 0x7f3d84860840\\\\\\\\nrcx 0xffffffffffffffff\\\\\\\\nrdx 0x0\\\\\\\\nrdi 0x2\\\\\\\\nrsi 0x7ffeff965720\\\\\\\\nrbp 0x55a7eb47c03d\\\\\\\\nrsp 0x7ffeff965720\\\\\\\\nr8 0x0\\\\\\\\nr9 0x7ffeff965720\\\\\\\\nr10 0x8\\\\\\\\nr11 0x246\\\\\\\\nr12 0x55a7ee582990\\\\\\\\nr13 0x11\\\\\\\\nr14 0x55a7eb422640\\\\\\\\nr15 0x0\\\\\\\\nrip 0x7f3d844b2e97\\\\\\\\nrflags 0x246\\\\\\\\ncs 0x33\\\\\\\\nfs 0x0\\\\\\\\ngs 0x0\\\\\\\\n\\\\\\"\\"": unknown'
ERROR: Encountered errors while bringing up the project.
Any ideas what i might doing wrong are very appriciated!

I also had this problem on a v-server from strato.
It seems like they removed the hard limit and allow changing the task limit now:
https://www.strato.de/faq/server/prozesse-vs-threads-bei-linux-v-servern/
To solve the error I just added the following line to /etc/systemd/system.conf and restarted
DefaultTasksMax=1000

The problem was that the server provider Strato has a hard process limit for their virtual servers. I configured it correctly, but I did not expect such a restriction. It cost me so much time... You should not go cheap with your provider, I have learned my lesson.

One solution:
Comment in and modify:
/etc/systemd/logind.conf UserTasksMax
then:
sudo systemctl daemon-reload
sudo systemctl daemon-reexec
sudo service systemd-logind restart

I have the same issue with grafana-agent and I've added security_opt option to fix it:
version: "3"
services:
agent:
image: grafana/agent:latest
security_opt:
- seccomp:unconfined
volumes:
- ./agent/config:/etc/agent-config
entrypoint:
- /bin/agent
- -config.file=/etc/agent-config/agent.yaml
- -metrics.wal-directory=/tmp/agent/wal
- -enable-features=integrations-next
- -config.expand-env
- -config.enable-read-api
ports:
- "12345:12345"

Related

Webdriver (selenium-standalone) tests with chromium - not starting ECONNREFUSED (on Apple M1)

our end to end tests run in a docker container and use Webdriver.io - with selenium-standalone.
Since updating to an M1/arm64 Mac I've been trying to update the tests to run.
Our unit tests (jest) were fixed easily by changing from using the ./google-chrome-stable_current_amd64.deb to using chromium.
Test container dockerfile:
FROM node:14-bullseye
ENV CHROME_BIN=/usr/bin/chromium
RUN apt-get update \
&& apt-get install -y chromium \
openjdk-11-jre \
xvfb \
x11vnc \
php \
&& mkdir ~/.vnc \
&& x11vnc -storepasswd "fakepwd" ~/.vnc/passwd \
&& rm -rf /var/lib/apt/lists/*
COPY ./entrypoint.sh /usr/local/bin/
ENTRYPOINT ["entrypoint.sh"]
I hoped it would be as simple for webdriver tests but no luck so far.
I've upgraded #wdio packages as below:
"#wdio/allure-reporter": "^7.10.1",
"#wdio/cli": "^7.19.3",
"#wdio/devtools-service": "^7.19.3",
"#wdio/junit-reporter": "^7.19.1",
"#wdio/local-runner": "^7.19.3",
"#wdio/mocha-framework": "^7.19.3",
"#wdio/selenium-standalone-service": "^7.19.1",
"#wdio/sync": "^7.19.3",
In the lock file I've confirmed the selenium-standalone version is 7.1.
(As above we use 7.19.1 of #wdio's service - this uses ^7.0 but only 7.1 works for M1's: https://github.com/webdriverio/selenium-standalone/commit/10e31d0e20444b0f6eb2d897cb3b471782dd243d)
I've confirmed that the correct chromedriver binary is used too (under ./node_modules/selenium-standalone/.selenium/chromedriver is 99.0.4844.51-arm64)
From wdio.conf.js:
services:[['selenium-standalone',{
installArgs: {
drivers: {
chrome: {
binary: '/usr/bin/chromium',
version: chromeDriverVersion,
arch : process.arch,
baseURL : 'https://chromedriver.storage.googleapis.com'
}
}
},
args: {
drivers: {
chrome: {
binary: '/usr/bin/chromium',
version: chromeDriverVersion,
arch : process.arch
}
}
}
}]];
capabilities: [{
browserName: process.env._BROWSER,
acceptInsecureCerts: true,
maxInstances: 4,
'goog:browserOptions': {
binary: '/usr/bin/chromium', // may not be needed, tried anyway
args: process.env._ENABLE_UI === 'false' ?
['--disable-dev-shm-usage','--no-sandbox','--headless','--disable-infobars','--deterministic-fetch', `--window-size=${default_viewport_width},${default_viewport_height}`] :
['--disable-dev-shm-usage','--no-sandbox','--disable-infobars', `--window-size=${default_viewport_width},${default_viewport_height}`]
}
}],
Watching wdio log outputs:
2022-04-01T08:09:37.336Z WARN #wdio/utils:shim: You are running tests with #wdio/sync which will be discontinued starting Node.js v16.Read more on https://github.com/webdriverio/webdriverio/discussions/6702
2022-04-01T08:09:40.101Z INFO #wdio/local-runner: Run worker command: run
2022-04-01T08:09:40.124Z DEBUG #wdio/config:ConfigParser: No compiler found, continue without compiling files
2022-04-01T08:09:42.042Z DEBUG #wdio/local-runner:utils: init remote session
2022-04-01T08:09:42.048Z DEBUG #wdio/utils:initialiseServices: initialise service "selenium-standalone" as NPM package
2022-04-01T08:09:42.648Z DEBUG #wdio/sync: Finished to run "beforeSession" hook in 0ms
2022-04-01T08:09:44.535Z DEBUG #wdio/local-runner:utils: init remote session
2022-04-01T08:09:44.536Z INFO webdriver: Initiate new session using the WebDriver protocol
2022-04-01T08:09:44.585Z INFO webdriver: [POST] http://127.0.0.1:4433/wd/hub/session
2022-04-01T08:09:44.586Z INFO webdriver: DATA {
capabilities: {
alwaysMatch: {
browserName: 'chromium',
acceptInsecureCerts: true,
'goog:browserOptions': [Object]
},
firstMatch: [ {} ]
},
desiredCapabilities: {
browserName: 'chromium',
acceptInsecureCerts: true,
'goog:browserOptions': { binary: '/usr/bin/chromium', args: [Array] }
}
}
2022-04-01T08:09:44.600Z ERROR webdriver: RequestError: connect ECONNREFUSED 127.0.0.1:4433
at ClientRequest.<anonymous> (/srv/api/author/node_modules/got/dist/source/core/index.js:956:111)
at Object.onceWrapper (events.js:520:26)
at ClientRequest.emit (events.js:412:35)
at ClientRequest.emit (domain.js:475:12)
at ClientRequest.origin.emit (/srv/api/author/node_modules/#szmarczak/http-timer/dist/source/index.js:43:20)
at Socket.socketErrorListener (_http_client.js:475:9)
at Socket.emit (events.js:400:28)
at Socket.emit (domain.js:475:12)
at emitErrorNT (internal/streams/destroy.js:106:8)
at emitErrorCloseNT (internal/streams/destroy.js:74:3)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1159:16)
2022-04-01T08:09:44.601Z ERROR #wdio/runner: Error: Failed to create session.
Unable to connect to "http://127.0.0.1:4433/wd/hub", make sure browser driver is running on that address.
If you use services like chromedriver see initialiseServices logs above or in wdio.log file as the service might had problems to start the driver.
at startWebDriverSession (/srv/api/author/node_modules/webdriver/build/utils.js:72:15)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at async Function.newSession (/srv/api/author/node_modules/webdriver/build/index.js:46:45)
at async remote (/srv/api/author/node_modules/#wdio/runner/node_modules/webdriverio/build/index.js:77:22)
at async Runner._startSession (/srv/api/author/node_modules/#wdio/runner/build/index.js:223:56)
at async Runner._initSession (/srv/api/author/node_modules/#wdio/runner/build/index.js:176:25)
at async Runner.run (/srv/api/author/node_modules/#wdio/runner/build/index.js:88:19)
2022-04-01T08:09:44.601Z DEBUG #wdio/sync: Finished to run "after" hook in 0ms
I presume it's chromium having an issue starting - but I can't find any relevant logs.
I've tried to start it inside the container manually, but I have no idea if there's anything useful here:
root#15adddf1e180:/srv/mydir# export DISPLAY=:99
root#15adddf1e180:/srv/mydir# /usr/bin/Xvfb :99 -ac -screen 0 1920x1080x24 > /dev/null 2>&1 &
[1] 9
root#15adddf1e180:/srv/mydir# x11vnc -listen 0.0.0.0 -display :99.0 -usepw -forever -nevershared > /dev/null 2>&1 &
[2] 24
root#15adddf1e180:/srv/mydir# /usr/bin/chromium --no-sandbox
[25:67:0401/081257.532385:ERROR:bus.cc(397)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
[25:67:0401/081257.532481:ERROR:bus.cc(397)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
[25:66:0401/081257.540896:ERROR:bus.cc(397)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
[25:66:0401/081257.540933:ERROR:bus.cc(397)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
[25:66:0401/081257.571813:ERROR:bus.cc(397)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
[25:66:0401/081257.571995:ERROR:bus.cc(397)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
[25:25:0401/081257.619504:ERROR:gpu_process_host.cc(972)] GPU process exited unexpectedly: exit_code=6
*** stack smashing detected ***: terminated
[25:89:0401/081257.630541:ERROR:bus.cc(397)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
[25:89:0401/081257.631138:ERROR:bus.cc(397)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
[25:89:0401/081257.631196:ERROR:bus.cc(397)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
[25:89:0401/081257.631336:ERROR:bus.cc(397)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
[25:89:0401/081257.631366:ERROR:bus.cc(397)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
[25:25:0401/081257.665149:ERROR:gpu_process_host.cc(972)] GPU process exited unexpectedly: exit_code=6
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
[25:25:0401/081257.679383:ERROR:gpu_process_host.cc(972)] GPU process exited unexpectedly: exit_code=6
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
[25:25:0401/081257.691523:ERROR:gpu_process_host.cc(972)] GPU process exited unexpectedly: exit_code=6
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
[25:25:0401/081257.701605:ERROR:gpu_process_host.cc(972)] GPU process exited unexpectedly: exit_code=6
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
[25:25:0401/081257.705098:ERROR:gpu_process_host.cc(972)] GPU process exited unexpectedly: exit_code=6
[25:25:0401/081257.705122:FATAL:gpu_data_manager_impl_private.cc(447)] GPU process isn't usable. Goodbye.
*** stack smashing detected ***: terminated
[0401/081257.720823:ERROR:file_io_posix.cc(144)] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory (2)
[0401/081257.720915:ERROR:file_io_posix.cc(144)] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq: No such file or directory (2)
Trace/breakpoint trap
root#15adddf1e180:/srv/mydir#
Any help would be appreciated, thanks!
It turns out that the "drivers" section must use "chromium" and not "chrome"
And for "capabilities" it should be "chrome"
I can't explain why unfortunately as selenium/webdriver docs seem to only mention chrome as the driver, not chromium - but this is the only way to get it to work
I.e.
const drivers = {};
drivers[process.env._BROWSER] = {
binary: '/usr/bin/chromium',
version: chromeDriverVersion,
arch : process.arch,
baseURL : 'https://chromedriver.storage.googleapis.com'
};
services:[['selenium-standalone', {
installArgs: {
drivers: drivers
},
args: {
drivers: drivers
}
}]];
capabilities: [{
browserName: process.env._BROWSER === "chromium" ? "chrome" : process.env._BROWSER,
...

GPU Acceleration with WSL 2

I'm trying to setup tensorflow to use GPU acceleration with WSL 2 running Ubuntu 20.04. I'm following this tutorial and am running into the error seen here. However, when I follow the solution there and try to start docker with sudo service docker start I get told docker is an unrecognized service. However, considering I can access the help menu and whatnot, I know docker is installed. While I can get docker to work with the desktop tool, since it doesn't support Cuda as mentioned in the SO post from earlier, it's not very helpful. It's not really giving me error logs or anything, so please ask if you need more details.
Edit:
Considering the lack of details, here are a list of solutions I've tried to no avail. 1 2 3
Update: I used sudo dockerd to get the container started and tried running the nvidia benchmark container only to be met with
INFO[2020-07-18T21:04:05.875283800-04:00] shim containerd-shim started address=/containerd-shim/021834ef5e5600bdf62a6a9e26dff7ffc1c76dd4ec9dadb9c1fcafb6c88b6e1b.sock debug=false pid=1960
INFO[2020-07-18T21:04:05.899420200-04:00] shim reaped id=70316df254d6b2633c743acb51a26ac2d0520f6f8e2f69b69c4e0624eaac1736
ERRO[2020-07-18T21:04:05.909710600-04:00] stream copy error: reading from a closed fifo
ERRO[2020-07-18T21:04:05.909753500-04:00] stream copy error: reading from a closed fifo
ERRO[2020-07-18T21:04:06.001006700-04:00] 70316df254d6b2633c743acb51a26ac2d0520f6f8e2f69b69c4e0624eaac1736 cleanup: failed to delete container from containerd: no such container
ERRO[2020-07-18T21:04:06.001045100-04:00] Handler for POST /v1.40/containers/70316df254d6b2633c743acb51a26ac2d0520f6f8e2f69b69c4e0624eaac1736/start returned error: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: driver error: failed to process request\\\\n\\\"\"": unknown
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: driver error: failed to process request\\\\n\\\"\"": unknown.
ERRO[0000] error waiting for container: context canceled
Update 2: After installing windows insider and making everything as up to date as possible, I encountered a different error.
Run "nbody -benchmark [-numbodies=<numBodies>]" to measure performance.
-fullscreen (run n-body simulation in fullscreen mode)
-fp64 (use double precision floating point values for simulation)
-hostmem (stores simulation data in host memory)
-benchmark (run benchmark to measure performance)
-numbodies=<N> (number of bodies (>= 1) to run in simulation)
-device=<d> (where d=0,1,2.... for the CUDA device to use)
-numdevices=<i> (where i=(number of CUDA devices > 0) to use for simulation)
-compare (compares simulation results running once on the default GPU and once on the CPU)
-cpu (run n-body simulation on the CPU)
-tipsy=<file.bin> (load a tipsy model file for simulation)
NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.
Error: only 0 Devices available, 1 requested. Exiting.
I have a GTX 970, so I'm not sure why it's not being detected. After running sudo lshw -C display, it was confirmed that my graphics card isn't being detected. I got:
*-display UNCLAIMED
description: 3D controller
product: Microsoft Corporation
vendor: Microsoft Corporation
physical id: 4
bus info: pci#941e:00:00.0
version: 00
width: 32 bits
clock: 33MHz
capabilities: bus_master cap_list
configuration: latency=0

How to fix docker OCI runtime create failed

I'm trying to start a server in Pterodactyl panel, But when I click the Start button, I got an error in daemon:
Error: (HTTP code 500) server error - OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"process_linux.go:396: setting cgroup config for procHooks process caused \\\"failed to write 1 to memory.oom_control: write /sys/fs/cgroup/memory/docker/5bf33f851f0628cdb2a087b3c3e9b7f0442281c12a95a532abbb1f797efd9c38/memory.oom_control: invalid argument\\\"\"": unknown
at /data/daemon/node_modules/docker-modem/lib/modem.js:254:17
at getCause (/data/daemon/node_modules/docker-modem/lib/modem.js:284:7)
at Modem.buildPayload (/data/daemon/node_modules/docker-modem/lib/modem.js:253:5)
at IncomingMessage.<anonymous> (/data/daemon/node_modules/docker-modem/lib/modem.js:229:14)
at emitNone (events.js:111:20)
at IncomingMessage.emit (events.js:208:7)
at endReadableNT (_stream_readable.js:1064:12)
at _combinedTickCallback (internal/process/next_tick.js:139:11)
at process._tickDomainCallback (internal/process/next_tick.js:219:9)
I've tried to reinstall Docker and disabled the OOM Killer in panel, but it doesn't work.
Maybe you can enable OOM Killer in your server build configuration : https://discordapp.com/channels/122900397965705216/272796619592695808/612444175367012352

Dataflow (Beam 2.12) does not start due to ext4 not found

I am seeing all types of strange errors when running a dataflow job (Beam 2.12).
The job basically takes input from pubsub, read/writes from/to Datastore writes the result to pubsub.
Several Warnings W and Errors E appear in the Stackdriver logs. It is unclear how to resolve these. Up to now we were using Beam 2.9 and were not experiencing any of these issues.
A partial (redacted) log dump is available below.
W acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
W ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
W ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10
W i8042: Warning: Keylock active
W GPT:Primary header thinks Alt. header is not at the end of the disk.
W GPT:52428799 != 62914559
W GPT:Alternate GPT header not at the end of the disk.
W GPT:52428799 != 62914559
W GPT: Use GNU Parted to correct GPT errors.
W device-mapper: verity: Argument 0: 'payload=PARTUUID=XXX'
W device-mapper: verity: Argument 1: 'hashtree=PARTUUID=XXX'
W device-mapper: verity: Argument 2: 'hashstart=2539520'
W device-mapper: verity: Argument 3: 'alg=sha1'
W device-mapper: verity: Argument 4: 'root_hexdigest=900...'
W device-mapper: verity: Argument 5: 'salt=b113702...'
W [/usr/lib/tmpfiles.d/var.conf:12] Duplicate line for path "/var/run", ignoring.
W Cannot set file attribute for '/var/log/journal', value=0x00800000, mask=0x00800000: Operation not supported
W Cannot set file attribute for '/var/log/journal/2a0c3f0af65e8318a0b8f3eb...', value=0x00800000, mask=0x00800000: Operation not supported
W Could not load the device policy file.
W [WARNING:persistent_integer.cc(96)] cannot open /var/lib/metrics/version.cycle for reading: No such file or directory
W WARNING Could not update the authorized keys file for user root. [Errno 30] Read-only file system: '/root/.ssh'.
W [CLOUDINIT] cc_write_files.py[WARNING]: Undecodable permissions None, assuming 420
...
E Error initializing dynamic plugin prober: Error (re-)creating driver directory: mkdir /usr/libexec/kubernetes: read-only file system
W No api server defined - no node status update will be sent.
W Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
W Unknown healthcheck type 'NONE' (expected 'CMD') in container 7df5acdbd1ad6756e3e409c6e8760d274bdc03f83bf...
E while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
E while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
W Unknown healthcheck type 'NONE' (expected 'CMD') in container 3aa8c92a0b7d746d7004768d5182f0558a0c0c90dfcd5...
W Unknown healthcheck type 'NONE' (expected 'CMD') in container 64b9fb0459f88833dee78943c32598761154e4a49d708...
W Unknown healthcheck type 'NONE' (expected 'CMD') in container d2edf1c5e89b746e8c9c96b2a39a9d7ac7da2ecf52f96d...
W Unknown healthcheck type 'NONE' (expected 'CMD') in container b2448a8792ad63059bb70f1f6f12385caae7a833018d05...
E EXT4-fs (sdb): VFS: Can't find ext4 filesystem
E Error syncing pod c386113... ("dataflow-...-harness-z656_default(c386113...)"), skipping: failed to "StartContainer" for "java-streaming" with CrashLoopBackOff: "Back-off 10s restarting failed container=java-streaming pod=dataflow-...-harness-z656_default(c386113...)"
W [WARNING:metrics_daemon.cc(619)] cannot read /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq
E EXT4-fs (sdd): VFS: Can't find ext4 filesystem
...
W Unknown healthcheck type 'NONE' (expected 'CMD') in container 675eb66a9e794b3dea03b62c3bdaf539034c998bf11c...
E Error syncing pod c386113... ("dataflow-...-harness-z656_default(c386113...)"), skipping: failed to "StartContainer" for "java-streaming" with CrashLoopBackOff: "Back-off 40s restarting failed container=java-streaming pod=dataflow-...-harness-z656_default(c386113...)"
E Error syncing pod c386113... ("dataflow-...-harness-z656_default(c386113...)"), skipping: failed to "StartContainer" for "java-streaming" with CrashLoopBackOff: "Back-off 40s restarting failed container=java-streaming pod=dataflow-...-harness-z656_default(c386113...)"
E Error syncing pod c386113... ("dataflow-...-harness-z656_default(c386113...)"), skipping: failed to "StartContainer" for "java-streaming" with CrashLoopBackOff: "Back-off 40s restarting failed container=java-streaming pod=dataflow-...-harness-z656_default(c386113...)"
W Unknown healthcheck type 'NONE' (expected 'CMD') in container 7d7536b93cb92562bdd12da3fd25a53caea8c9a9e1cee603b3999dfdd5681a27
E Error syncing pod c386113... ("dataflow-...-harness-z656_default(c386113...)"), skipping: failed to "StartContainer" for "java-streaming" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=java-streaming pod=dataflow-...-harness-z656_default(c386113...)"
I resolved this by upgrading several dependencies.
The maven versions plugin helped me do this, I installed the plugin by adding the following to my .pom file:
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>versions-maven-plugin</artifactId>
<version>2.5</version>
</plugin>
Then I checked the libraries to update and updated them. I suspect it to be an older BigTable client, as mentioned here.
mvn versions:display-dependency-updates
mvn versions:use-latest-versions

Elasticbeanstalk Deployment Fails

I am trying to deploy a docker based environment that is using 1-1 application load balancer in AWS elasticbeanstalk. I am getting the below error message. It seemed to deploy very well before but for unknown reasons the new deploys fail.
2019-01-31 18:52:23 INFO Environment update is starting.
2019-01-31 18:53:03 INFO Deploying new version to instance(s).
2019-01-31 18:53:12 INFO Successfully pulled node:10.15.0
2019-01-31 18:57:34 INFO Successfully built aws_beanstalk/staging-app
2019-01-31 18:57:42 ERROR Failed to start nginx, abort deployment
2019-01-31 18:57:44 ERROR [Instance: i-0d4b3811324b90c27] Command failed on instance. Return code: 1 Output: nginx: [emerg] no host in upstream ":8080" in /etc/nginx/conf.d/elasticbeanstalk-nginx-docker-upstream.conf:2
nginx: configuration file /etc/nginx/nginx.conf test failed
Failed to start nginx, abort deployment.
Hook /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
2019-01-31 18:57:45 INFO Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
2019-01-31 18:57:45 ERROR Unsuccessful command execution on instance id(s) 'i-0d4b3811324b90c27'. Aborting the operation.
2019-01-31 18:57:45 ERROR Failed to deploy application.
This issue was caused of an Uncaught error in my Node js application. Issue is not with Beanstalk.

Resources