Infinite Loop when deploying to Elastic Beanstalk - docker
I would like to deploy a Docker image containing Keycloak to Elastic Beanstalk.
To do so I created a Dockerfile like this.
FROM bitnami/keycloak:19.0.2
EXPOSE 5000
This deploys successfully when running eb deploy.
I would now like to compile some Java code and copy it into the Keycloak image.
FROM gradle:7.5.0-jdk11-alpine AS spi
COPY --chown=gradle:gradle ./spi /spi
WORKDIR /spi
RUN gradle build --no-daemon
FROM bitnami/keycloak:19
COPY --from=spi /spi/build/libs/* /opt/keycloak/providers/keycloak-spi-0.0.1-SNAPSHOT.jar
EXPOSE 5000
This builds successfully using docker build --no-cache -t keycloak . and can be run using docker run locally.
If I try deploying this to Elastic Beanstalk it deploys indefinitely. I can't cancel the deployment and have to force it to stop by setting the Auto Scaling Group capacity to 0.
----------------------------------------
/var/log/eb-hooks.log
----------------------------------------
----------------------------------------
/var/log/nginx/access.log
----------------------------------------
172.31.15.133 - - [29/Sep/2022:17:05:19 +0000] "GET / HTTP/1.1" 502 157 "-" "ELB-HealthChecker/2.0" "-"
172.31.35.107 - - [29/Sep/2022:17:05:19 +0000] "GET / HTTP/1.1" 502 157 "-" "ELB-HealthChecker/2.0" "-"
172.31.15.133 - - [29/Sep/2022:17:05:32 +0000] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36" "86.11.104.152"
172.31.15.133 - - [29/Sep/2022:17:05:34 +0000] "GET / HTTP/1.1" 502 157 "-" "ELB-HealthChecker/2.0" "-"
172.31.35.107 - - [29/Sep/2022:17:05:34 +0000] "GET / HTTP/1.1" 502 157 "-" "ELB-HealthChecker/2.0" "-"
----------------------------------------
/var/log/nginx/error.log
----------------------------------------
2022/09/29 17:05:19 [error] 4202#4202: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.31.15.133, server: , request: "GET / HTTP/1.1", upstream: "http://172.17.0.2:5000/", host: "172.31.7.44"
2022/09/29 17:05:19 [error] 4202#4202: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 172.31.35.107, server: , request: "GET / HTTP/1.1", upstream: "http://172.17.0.2:5000/", host: "172.31.7.44"
2022/09/29 17:05:32 [error] 4202#4202: *5 connect() failed (111: Connection refused) while connecting to upstream, client: 172.31.15.133, server: , request: "GET / HTTP/1.1", upstream: "http://172.17.0.2:5000/", host: "keycloak-dev.eu-west-2.elasticbeanstalk.com"
2022/09/29 17:05:34 [error] 4202#4202: *7 connect() failed (111: Connection refused) while connecting to upstream, client: 172.31.15.133, server: , request: "GET / HTTP/1.1", upstream: "http://172.17.0.2:5000/", host: "172.31.7.44"
2022/09/29 17:05:34 [error] 4202#4202: *9 connect() failed (111: Connection refused) while connecting to upstream, client: 172.31.35.107, server: , request: "GET / HTTP/1.1", upstream: "http://172.17.0.2:5000/", host: "172.31.7.44"
----------------------------------------
/var/log/eb-docker/containers/eb-current-app/eb-48320d59a48b-stdouterr.log
----------------------------------------
[38;5;6mkeycloak [38;5;5m17:05:06.31 [0m
[38;5;6mkeycloak [38;5;5m17:05:06.32 [0m[1mWelcome to the Bitnami keycloak container[0m
[38;5;6mkeycloak [38;5;5m17:05:06.32 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/containers[0m
[38;5;6mkeycloak [38;5;5m17:05:06.32 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/containers/issues[0m
[38;5;6mkeycloak [38;5;5m17:05:06.32 [0m
[38;5;6mkeycloak [38;5;5m17:05:06.32 [0m[38;5;2mINFO [0m ==> ** Starting keycloak setup **
[38;5;6mkeycloak [38;5;5m17:05:06.33 [0m[38;5;2mINFO [0m ==> Validating settings in KEYCLOAK_* env vars...
[38;5;6mkeycloak [38;5;5m17:05:06.35 [0m[38;5;2mINFO [0m ==> Trying to connect to PostgreSQL server awseb-e-2mwnvjecwe-stack-awsebrdsdatabase-qb51rlaqbaxv.czdgrvr4inhi.eu-west-2.rds.amazonaws.com...
[38;5;6mkeycloak [38;5;5m17:05:06.38 [0m[38;5;2mINFO [0m ==> Found PostgreSQL server listening at awseb-e-2mwnvjecwe-stack-awsebrdsdatabase-qb51rlaqbaxv.czdgrvr4inhi.eu-west-2.rds.amazonaws.com:5432
[38;5;6mkeycloak [38;5;5m17:05:06.38 [0m[38;5;2mINFO [0m ==> Configuring database settings
[38;5;6mkeycloak [38;5;5m17:05:06.41 [0m[38;5;2mINFO [0m ==> Enabling statistics
[38;5;6mkeycloak [38;5;5m17:05:06.41 [0m[38;5;2mINFO [0m ==> Configuring http settings
[38;5;6mkeycloak [38;5;5m17:05:06.42 [0m[38;5;2mINFO [0m ==> Configuring hostname settings
[38;5;6mkeycloak [38;5;5m17:05:06.43 [0m[38;5;2mINFO [0m ==> Configuring cache count
[38;5;6mkeycloak [38;5;5m17:05:06.43 [0m[38;5;2mINFO [0m ==> Configuring log level
[38;5;6mkeycloak [38;5;5m17:05:06.44 [0m[38;5;2mINFO [0m ==> Configuring proxy
[38;5;6mkeycloak [38;5;5m17:05:06.45 [0m[38;5;2mINFO [0m ==> ** keycloak setup finished! **
[38;5;6mkeycloak [38;5;5m17:05:06.46 [0m[38;5;2mINFO [0m ==> ** Starting keycloak **
Updating the configuration and installing your custom providers, if any. Please wait.
2022-09-29 17:05:19,703 WARN [org.keycloak.services] (build-43) KC-SERVICES0047: metrics (org.jboss.aerogear.keycloak.metrics.MetricsEndpointFactory) is implementing the internal SPI realm-restapi-extension. This SPI is internal and may change without notice
2022-09-29 17:05:21,233 WARN [org.keycloak.services] (build-43) KC-SERVICES0047: metrics-listener (org.jboss.aerogear.keycloak.metrics.MetricsEventListenerFactory) is implementing the internal SPI eventsListener. This SPI is internal and may change without notice
2022-09-29 17:05:32,212 INFO [io.quarkus.deployment.QuarkusAugmentor] (main) Quarkus augmentation completed in 21445ms
2022-09-29 17:05:35,762 INFO [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: <unset>, Hostname: <request>, Strict HTTPS: false, Path: <request>, Strict BackChannel: false, Admin URL: <unset>, Admin: <request>, Port: -1, Proxied: true
2022-09-29 17:05:38,486 INFO [org.keycloak.common.crypto.CryptoIntegration] (main) Detected crypto provider: org.keycloak.crypto.def.DefaultCryptoProvider
----------------------------------------
/var/log/docker-events.log
----------------------------------------
2022-09-29T17:05:03.177179628Z image pull bitnami/keycloak:19 (name=bitnami/keycloak, org.opencontainers.image.authors=https://bitnami.com/contact, org.opencontainers.image.description=Application packaged by Bitnami, org.opencontainers.image.ref.name=19.0.2-debian-11-r4, org.opencontainers.image.source=https://github.com/bitnami/containers/tree/main/bitnami/keycloak, org.opencontainers.image.title=keycloak, org.opencontainers.image.vendor=VMware, Inc., org.opencontainers.image.version=19.0.2)
2022-09-29T17:05:03.398516443Z container create edf1b48a9117e98a17c3c65139a2c3131ba40c31cc036faba9d7e541eb1308a4 (image=sha256:2a52f7074b400c46837253be0cd6719054373aa15c556a2505cfc4875c732191, name=crazy_keldysh, org.opencontainers.image.authors=https://bitnami.com/contact, org.opencontainers.image.description=Application packaged by Bitnami, org.opencontainers.image.ref.name=19.0.2-debian-11-r4, org.opencontainers.image.source=https://github.com/bitnami/containers/tree/main/bitnami/keycloak, org.opencontainers.image.title=keycloak, org.opencontainers.image.vendor=VMware, Inc., org.opencontainers.image.version=19.0.2)
2022-09-29T17:05:03.467499766Z container destroy edf1b48a9117e98a17c3c65139a2c3131ba40c31cc036faba9d7e541eb1308a4 (image=sha256:2a52f7074b400c46837253be0cd6719054373aa15c556a2505cfc4875c732191, name=crazy_keldysh, org.opencontainers.image.authors=https://bitnami.com/contact, org.opencontainers.image.description=Application packaged by Bitnami, org.opencontainers.image.ref.name=19.0.2-debian-11-r4, org.opencontainers.image.source=https://github.com/bitnami/containers/tree/main/bitnami/keycloak, org.opencontainers.image.title=keycloak, org.opencontainers.image.vendor=VMware, Inc., org.opencontainers.image.version=19.0.2)
2022-09-29T17:05:03.471362994Z image tag sha256:5d99cf30b3aec76495a2db5b6e5292125f31f29ddfe7aca21789abd2f13fc30c (name=aws_beanstalk/staging-app:latest, org.opencontainers.image.authors=https://bitnami.com/contact, org.opencontainers.image.description=Application packaged by Bitnami, org.opencontainers.image.ref.name=19.0.2-debian-11-r4, org.opencontainers.image.source=https://github.com/bitnami/containers/tree/main/bitnami/keycloak, org.opencontainers.image.title=keycloak, org.opencontainers.image.vendor=VMware, Inc., org.opencontainers.image.version=19.0.2)
2022-09-29T17:05:05.646632020Z container create 48320d59a48b027ab5d08531271a2cd0c4cafac5e80f322fa7fc98c0849ba5b4 (image=5d99cf30b3ae, name=cool_pasteur, org.opencontainers.image.authors=https://bitnami.com/contact, org.opencontainers.image.description=Application packaged by Bitnami, org.opencontainers.image.ref.name=19.0.2-debian-11-r4, org.opencontainers.image.source=https://github.com/bitnami/containers/tree/main/bitnami/keycloak, org.opencontainers.image.title=keycloak, org.opencontainers.image.vendor=VMware, Inc., org.opencontainers.image.version=19.0.2)
2022-09-29T17:05:05.718178496Z network connect b4145308580fd568a5789e6f5345557a657c66187a05059e0e1d0e7d0813bab6 (container=48320d59a48b027ab5d08531271a2cd0c4cafac5e80f322fa7fc98c0849ba5b4, name=bridge, type=bridge)
2022-09-29T17:05:06.289309003Z container start 48320d59a48b027ab5d08531271a2cd0c4cafac5e80f322fa7fc98c0849ba5b4 (image=5d99cf30b3ae, name=cool_pasteur, org.opencontainers.image.authors=https://bitnami.com/contact, org.opencontainers.image.description=Application packaged by Bitnami, org.opencontainers.image.ref.name=19.0.2-debian-11-r4, org.opencontainers.image.source=https://github.com/bitnami/containers/tree/main/bitnami/keycloak, org.opencontainers.image.title=keycloak, org.opencontainers.image.vendor=VMware, Inc., org.opencontainers.image.version=19.0.2)
2022-09-29T17:05:12.038251773Z image tag sha256:5d99cf30b3aec76495a2db5b6e5292125f31f29ddfe7aca21789abd2f13fc30c (name=aws_beanstalk/current-app:latest, org.opencontainers.image.authors=https://bitnami.com/contact, org.opencontainers.image.description=Application packaged by Bitnami, org.opencontainers.image.ref.name=19.0.2-debian-11-r4, org.opencontainers.image.source=https://github.com/bitnami/containers/tree/main/bitnami/keycloak, org.opencontainers.image.title=keycloak, org.opencontainers.image.vendor=VMware, Inc., org.opencontainers.image.version=19.0.2)
2022-09-29T17:05:12.101612937Z image untag sha256:5d99cf30b3aec76495a2db5b6e5292125f31f29ddfe7aca21789abd2f13fc30c (name=sha256:5d99cf30b3aec76495a2db5b6e5292125f31f29ddfe7aca21789abd2f13fc30c, org.opencontainers.image.authors=https://bitnami.com/contact, org.opencontainers.image.description=Application packaged by Bitnami, org.opencontainers.image.ref.name=19.0.2-debian-11-r4, org.opencontainers.image.source=https://github.com/bitnami/containers/tree/main/bitnami/keycloak, org.opencontainers.image.title=keycloak, org.opencontainers.image.vendor=VMware, Inc., org.opencontainers.image.version=19.0.2)
2022-09-29T17:05:13.916310158Z image tag sha256:5d99cf30b3aec76495a2db5b6e5292125f31f29ddfe7aca21789abd2f13fc30c (name=aws_beanstalk/current-app:latest, org.opencontainers.image.authors=https://bitnami.com/contact, org.opencontainers.image.description=Application packaged by Bitnami, org.opencontainers.image.ref.name=19.0.2-debian-11-r4, org.opencontainers.image.source=https://github.com/bitnami/containers/tree/main/bitnami/keycloak, org.opencontainers.image.title=keycloak, org.opencontainers.image.vendor=VMware, Inc., org.opencontainers.image.version=19.0.2)
2022-09-29T17:05:14.010857853Z image tag sha256:2a52f7074b400c46837253be0cd6719054373aa15c556a2505cfc4875c732191 (name=bitnami/keycloak:19, org.opencontainers.image.authors=https://bitnami.com/contact, org.opencontainers.image.description=Application packaged by Bitnami, org.opencontainers.image.ref.name=19.0.2-debian-11-r4, org.opencontainers.image.source=https://github.com/bitnami/containers/tree/main/bitnami/keycloak, org.opencontainers.image.title=keycloak, org.opencontainers.image.vendor=VMware, Inc., org.opencontainers.image.version=19.0.2)
----------------------------------------
/var/log/eb-docker-process.log
----------------------------------------
2022/09/29 17:05:12.516005 [INFO] Loading Manifest...
2022/09/29 17:05:12.553556 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:eu-west-2:882618350644:stack/awseb-e-2mwnvjecwe-stack/e6e0d050-3f75-11ed-a0e9-0a1f03d7e8a8 -r AWSEBAutoScalingGroup --region eu-west-2
2022/09/29 17:05:14.188072 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:eu-west-2:882618350644:stack/awseb-e-2mwnvjecwe-stack/e6e0d050-3f75-11ed-a0e9-0a1f03d7e8a8 -r AWSEBBeanstalkMetadata --region eu-west-2
2022/09/29 17:05:15.706457 [INFO] Checking if docker is running...
2022/09/29 17:05:15.706480 [INFO] Fetch current app container id...
2022/09/29 17:05:15.706802 [INFO] Running command /bin/sh -c docker ps | grep 48320d59a48b
2022/09/29 17:05:15.851077 [INFO] 48320d59a48b 5d99cf30b3ae "/opt/bitnami/script…" 10 seconds ago Up 9 seconds 5000/tcp cool_pasteur
2022/09/29 17:05:15.851106 [INFO] Running command /bin/sh -c docker wait 48320d59a48b
----------------------------------------
/var/log/docker
----------------------------------------
Sep 29 17:04:41 ip-172-31-7-44 docker: time="2022-09-29T17:04:41.522348910Z" level=info msg="Starting up"
Sep 29 17:04:41 ip-172-31-7-44 docker: time="2022-09-29T17:04:41.546445360Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Sep 29 17:04:41 ip-172-31-7-44 docker: time="2022-09-29T17:04:41.546469970Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Sep 29 17:04:41 ip-172-31-7-44 docker: time="2022-09-29T17:04:41.546492925Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Sep 29 17:04:41 ip-172-31-7-44 docker: time="2022-09-29T17:04:41.546507007Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Sep 29 17:04:41 ip-172-31-7-44 docker: time="2022-09-29T17:04:41.585273534Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Sep 29 17:04:41 ip-172-31-7-44 docker: time="2022-09-29T17:04:41.585318052Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Sep 29 17:04:41 ip-172-31-7-44 docker: time="2022-09-29T17:04:41.585339847Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Sep 29 17:04:41 ip-172-31-7-44 docker: time="2022-09-29T17:04:41.585350796Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Sep 29 17:04:41 ip-172-31-7-44 docker: time="2022-09-29T17:04:41.870138260Z" level=info msg="Loading containers: start."
Sep 29 17:04:42 ip-172-31-7-44 docker: time="2022-09-29T17:04:42.576967756Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Sep 29 17:04:42 ip-172-31-7-44 docker: time="2022-09-29T17:04:42.737972210Z" level=info msg="Loading containers: done."
Sep 29 17:04:43 ip-172-31-7-44 docker: time="2022-09-29T17:04:43.044803469Z" level=info msg="Docker daemon" commit=a89b842 graphdriver(s)=overlay2 version=20.10.17
Sep 29 17:04:43 ip-172-31-7-44 docker: time="2022-09-29T17:04:43.045249096Z" level=info msg="Daemon has completed initialization"
Sep 29 17:04:43 ip-172-31-7-44 docker: time="2022-09-29T17:04:43.082693306Z" level=info msg="API listen on /run/docker.sock"
Sep 29 17:05:03 ip-172-31-7-44 docker: time="2022-09-29T17:05:03.459921076Z" level=info msg="Layer sha256:35c7205075f4f3bd7f9b0e488668782fa244a75cccb6a6ef6a3b14251b55236d cleaned up"
----------------------------------------
/var/log/eb-engine.log
----------------------------------------
2022/09/29 17:05:11.511727 [INFO] Running command /bin/sh -c systemctl show -p PartOf nginx.service
2022/09/29 17:05:11.523154 [INFO] register process nginx
2022/09/29 17:05:11.523176 [INFO] Running command /bin/sh -c systemctl show -p PartOf nginx.service
2022/09/29 17:05:11.537946 [INFO] Running command /bin/sh -c systemctl daemon-reload
2022/09/29 17:05:11.659032 [INFO] Running command /bin/sh -c systemctl reset-failed
2022/09/29 17:05:11.666538 [INFO] Running command /bin/sh -c systemctl enable nginx.service
2022/09/29 17:05:11.786233 [INFO] Created symlink from /etc/systemd/system/multi-user.target.wants/nginx.service to /usr/lib/systemd/system/nginx.service.
2022/09/29 17:05:11.786293 [INFO] Running command /bin/sh -c systemctl show -p PartOf nginx.service
2022/09/29 17:05:11.802086 [INFO] Running command /bin/sh -c systemctl restart nginx.service
2022/09/29 17:05:11.904366 [INFO] Running command /bin/sh -c service iptables save
2022/09/29 17:05:11.966054 [INFO] iptables: Saving firewall rules to /etc/sysconfig/iptables: [ OK ]
2022/09/29 17:05:11.966069 [INFO] Traffic forward set-up completed.
2022/09/29 17:05:11.966074 [INFO] making STAGING app container current...
2022/09/29 17:05:11.966088 [INFO] Running command /bin/sh -c docker tag 5d99cf30b3ae aws_beanstalk/current-app
2022/09/29 17:05:12.041952 [INFO] Running command /bin/sh -c docker rmi aws_beanstalk/staging-app
2022/09/29 17:05:12.106692 [INFO] Untagged: aws_beanstalk/staging-app:latest
2022/09/29 17:05:12.106769 [INFO] Running command /bin/sh -c systemctl show -p PartOf eb-docker.service
2022/09/29 17:05:12.118591 [INFO] Running command /bin/sh -c systemctl daemon-reload
2022/09/29 17:05:12.242722 [INFO] Running command /bin/sh -c systemctl reset-failed
2022/09/29 17:05:12.250703 [INFO] Running command /bin/sh -c systemctl enable eb-docker.service
2022/09/29 17:05:12.365666 [INFO] Running command /bin/sh -c systemctl show -p PartOf eb-docker.service
2022/09/29 17:05:12.382129 [INFO] Running command /bin/sh -c systemctl is-active eb-docker.service
2022/09/29 17:05:12.390717 [INFO] Running command /bin/sh -c systemctl start eb-docker.service
2022/09/29 17:05:12.465476 [INFO] Running command /bin/sh -c systemctl show -p PartOf eb-docker-log.service
2022/09/29 17:05:12.479384 [INFO] Running command /bin/sh -c systemctl daemon-reload
2022/09/29 17:05:12.620319 [INFO] Running command /bin/sh -c systemctl reset-failed
2022/09/29 17:05:12.638521 [INFO] Running command /bin/sh -c systemctl enable eb-docker-log.service
2022/09/29 17:05:12.787312 [INFO] Created symlink from /etc/systemd/system/eb-docker.service.wants/eb-docker-log.service to /etc/systemd/system/eb-docker-log.service.
2022/09/29 17:05:12.787345 [INFO] Running command /bin/sh -c systemctl show -p PartOf eb-docker-log.service
2022/09/29 17:05:12.803790 [INFO] Running command /bin/sh -c systemctl is-active eb-docker-log.service
2022/09/29 17:05:12.814626 [INFO] Running command /bin/sh -c systemctl start eb-docker-log.service
2022/09/29 17:05:12.939662 [INFO] docker container 48320d59a48b is running aws_beanstalk/current-app
2022/09/29 17:05:12.939681 [INFO] Executing instruction: Clean up Docker
2022/09/29 17:05:12.939710 [INFO] Running command /bin/sh -c docker ps -aq
2022/09/29 17:05:13.129931 [INFO] 48320d59a48b
2022/09/29 17:05:13.129970 [INFO] Running command /bin/sh -c docker images | sed 1d
2022/09/29 17:05:13.282274 [INFO] aws_beanstalk/current-app latest 5d99cf30b3ae 10 seconds ago 622MB
bitnami/keycloak 19 2a52f7074b40 3 days ago 622MB
2022/09/29 17:05:13.282303 [INFO] save docker tag command: docker tag 5d99cf30b3ae aws_beanstalk/current-app:latest
2022/09/29 17:05:13.282309 [INFO] save docker tag command: docker tag 2a52f7074b40 bitnami/keycloak:19
2022/09/29 17:05:13.282321 [INFO] Running command /bin/sh -c docker rm `docker ps -aq`
2022/09/29 17:05:13.529450 [INFO] Error response from daemon: You cannot remove a running container 48320d59a48b027ab5d08531271a2cd0c4cafac5e80f322fa7fc98c0849ba5b4. Stop the container before attempting removal or force remove
2022/09/29 17:05:13.529511 [INFO] Running command /bin/sh -c docker rmi `docker images -aq`
2022/09/29 17:05:13.789049 [INFO] Error response from daemon: conflict: unable to delete 5d99cf30b3ae (cannot be forced) - image is being used by running container 48320d59a48b
Error response from daemon: conflict: unable to delete 2a52f7074b40 (cannot be forced) - image has dependent child images
2022/09/29 17:05:13.789067 [INFO] restore docker image name with command: docker tag 5d99cf30b3ae aws_beanstalk/current-app:latest
2022/09/29 17:05:13.789078 [INFO] Running command /bin/sh -c docker tag 5d99cf30b3ae aws_beanstalk/current-app:latest
2022/09/29 17:05:13.918414 [INFO] restore docker image name with command: docker tag 2a52f7074b40 bitnami/keycloak:19
2022/09/29 17:05:13.918438 [INFO] Running command /bin/sh -c docker tag 2a52f7074b40 bitnami/keycloak:19
2022/09/29 17:05:14.013863 [INFO] Executing instruction: start X-Ray
2022/09/29 17:05:14.013880 [INFO] X-Ray is not enabled.
2022/09/29 17:05:14.013885 [INFO] Executing instruction: configureSqsd
2022/09/29 17:05:14.013891 [INFO] This is a web server environment instance, skip configure sqsd daemon ...
2022/09/29 17:05:14.013896 [INFO] Executing instruction: startSqsd
2022/09/29 17:05:14.013900 [INFO] This is a web server environment instance, skip start sqsd daemon ...
2022/09/29 17:05:14.013905 [INFO] Executing instruction: Track pids in healthd
2022/09/29 17:05:14.013910 [INFO] This is an enhanced health env...
2022/09/29 17:05:14.013939 [INFO] Running command /bin/sh -c systemctl show -p ConsistsOf aws-eb.target | cut -d= -f2
2022/09/29 17:05:14.032650 [INFO] eb-docker.service healthd.service nginx.service eb-docker-events.service cfn-hup.service eb-docker-log.service docker.service
2022/09/29 17:05:14.032683 [INFO] Running command /bin/sh -c systemctl show -p ConsistsOf eb-app.target | cut -d= -f2
2022/09/29 17:05:14.055602 [INFO]
2022/09/29 17:05:14.057747 [INFO] Executing instruction: Configure Docker Container Logging
2022/09/29 17:05:14.066518 [INFO] Executing instruction: RunAppDeployPostDeployHooks
2022/09/29 17:05:14.066536 [INFO] Executing platform hooks in .platform/hooks/postdeploy/
2022/09/29 17:05:14.066560 [INFO] The dir .platform/hooks/postdeploy/ does not exist
2022/09/29 17:05:14.066564 [INFO] Finished running scripts in /var/app/current/.platform/hooks/postdeploy
2022/09/29 17:05:14.066572 [INFO] Executing cleanup logic
2022/09/29 17:05:14.068115 [INFO] Platform Engine finished execution on command: self-startup
2022/09/29 17:05:39.149172 [INFO] Starting...
2022/09/29 17:05:39.149959 [INFO] Starting EBPlatform-PlatformEngine
2022/09/29 17:05:39.149999 [INFO] reading event message file
2022/09/29 17:05:39.151835 [INFO] Engine received EB command cfn-hup-exec
2022/09/29 17:05:39.291599 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:eu-west-2:882618350644:stack/awseb-e-2mwnvjecwe-stack/e6e0d050-3f75-11ed-a0e9-0a1f03d7e8a8 -r AWSEBAutoScalingGroup --region eu-west-2
2022/09/29 17:05:40.014546 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:eu-west-2:882618350644:stack/awseb-e-2mwnvjecwe-stack/e6e0d050-3f75-11ed-a0e9-0a1f03d7e8a8 -r AWSEBBeanstalkMetadata --region eu-west-2
2022/09/29 17:05:40.681829 [INFO] checking whether command tail-log is applicable to this instance...
2022/09/29 17:05:40.681845 [INFO] this command is applicable to the instance, thus instance should execute command
2022/09/29 17:05:40.681850 [INFO] Engine command: (tail-log)
2022/09/29 17:05:40.682976 [INFO] Executing instruction: GetTailLogs
2022/09/29 17:05:40.682984 [INFO] Tail Logs...
2022/09/29 17:05:40.685700 [INFO] Running command /bin/sh -c tail -n 100 /var/log/eb-hooks.log
2022/09/29 17:05:40.687855 [INFO] Running command /bin/sh -c tail -n 100 /var/log/nginx/access.log
2022/09/29 17:05:40.689449 [INFO] Running command /bin/sh -c tail -n 100 /var/log/nginx/error.log
2022/09/29 17:05:40.693994 [INFO] Running command /bin/sh -c tail -n 100 /var/log/eb-docker/containers/eb-current-app/eb-48320d59a48b-stdouterr.log
2022/09/29 17:05:40.695588 [INFO] Running command /bin/sh -c tail -n 100 /var/log/docker-events.log
2022/09/29 17:05:40.702009 [INFO] Running command /bin/sh -c tail -n 100 /var/log/eb-docker-process.log
2022/09/29 17:05:40.703972 [INFO] Running command /bin/sh -c tail -n 100 /var/log/docker
2022/09/29 17:05:40.707914 [INFO] Running command /bin/sh -c tail -n 100 /var/log/eb-engine.log
The logs show that it is unable to delete the current docker images as the container is still running. I don't how I've caused that.
What is likely causing this?
Related
Directory Nonexistent when Attaching VSCode to running Docker Container
I have a running docker container. I was previously able to connect to this container. During use, I was disconnected. Attempts to reconnect to the docker container (or other variations thereof) are unsuccessful. I've included the logs for this below.
[225 ms] Dev Containers 0.275.1 in VS Code 1.75.0 (e2816fe719a4026ffa1ee0189dc89bdfdbafb164).
[225 ms] Start: Resolving Remote
[227 ms] Setting up container: container_name_redacted
[227 ms] Host: host_redacted
[230 ms] Start: Run: host_redacted /bin/sh
[251 ms] Start: Run in host: id -un
[3684 ms] name_redacted
[3685 ms]
[3685 ms] Start: Run in host: cat /etc/passwd
[3702 ms] Start: Run in host: echo ~
[3719 ms] /home/name_redacted
[3720 ms]
[3721 ms] Start: Run in host: test -x '/home/name_redacted/.vscode-remote-containers/bin/e2816fe719a4026ffa1ee0189dc89bdfdbafb164/node'
[3738 ms]
[3738 ms]
[3739 ms] Start: Run in host: test -f '/home/name_redacted/.vscode-remote-containers/dist/vscode-remote-containers-server-0.275.1.js'
[3756 ms]
[3756 ms]
[3769 ms] userEnvProbe: loginInteractiveShell (default)
[3770 ms] userEnvProbe: not found in cache
[3770 ms] userEnvProbe shell: /bin/bash
[3852 ms] userEnvProbe PATHs:
Probe: '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games'
Container: None
[3871 ms] Start: Run in Host: docker inspect --type container /redacted_container_name
[3933 ms] Start: Check Docker is running
[3933 ms] Start: Run in Host: docker version --format {{.Server.APIVersion}}
[3999 ms] Server API version: 1.41
[4047 ms] Command failed: /bin/sh -c cat >'/tmp/devcontainer-config-91de0ba6-e023-476c-b3a0-df813264bd3f.json'
[4047 ms] /bin/sh: 1: cannot create /tmp/devcontainer-config-91de0ba6-e023-476c-b3a0-df813264bd3f.json: Directory nonexistent
[4047 ms] Exit code 2
It looks like it cannot access the /tmp/ directory. I've tried mapping this directory to the container using -v. I've also tried opening up permissions. I've tried creating the container specific to my user. I've tried using privileged parameter. I'm at a loss here.
Using docker in Jenkins pipeline fails
In Jenkinsfile I am trying to download docker image and use in this docker image for all the git pull and frontend build stuff.
My jenkinsfile is so far:
pipeline {
agent any
stages {
stage('Install Docker-CE') {
steps {
sh '''curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install -y docker-ce'''
}
}
stage('Start Docker') {
steps {
sh 'sudo service docker start'
sh 'sudo service docker status'
}
}
stage('Verify Docker') {
steps {
sh 'sudo docker run hello-world'
}
}
stage('Build Back End') {
steps {
git ([url : 'https://github....git', branch : 'develop', credentialsId : 'xxx' ])
}
}
}
The docker is installed and running, because in the 'Start Docker' step when I run sudo service docker status, it says: Docker is running
but when trying to do 'hello world' in the next step, it says docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
I also tried to run docker daemon with: sudo dockerd but it didn't help, this is the part of the output:
time="2021-07-28T08:45:04.782471391Z" level=error msg="failed to mount overlay: permission denied" storage-driver=overlay2
time="2021-07-28T08:45:04.782647045Z" level=error msg="exec: \"fuse-overlayfs\": executable file not found in $PATH" storage-driver=fuse-overlayfs
time="2021-07-28T08:45:04.783389746Z" level=warning msg="[graphdriver] WARNING: the aufs storage-driver is deprecated, and will be removed in a future release"
time="2021-07-28T08:45:04.810999832Z" level=warning msg="Your kernel does not support CPU realtime scheduler"
time="2021-07-28T08:45:04.811283522Z" level=info msg="Loading containers: start."
time="2021-07-28T08:45:04.816297382Z" level=warning msg="Running iptables --wait -t nat -L -n failed with message: `iptables v1.6.0: can't initialize iptables table `nat': Permission denied (you must be root)\nPerhaps iptables or your kernel needs to be upgraded.`, error: exit status 3"
time="2021-07-28T08:45:04.887136525Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
time="2021-07-28T08:45:04.887859247Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
time="2021-07-28T08:45:04.887889167Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.6.0: can't initialize iptables table `nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.
(exit status 3)
Ok, so the problem was I used sudo in sudo docker run hello-world. Without sudo it works...
amazonlinux:2 systemctl Failed to get D-Bus connection
I would like to use this image https://hub.docker.com/_/amazonlinux to script my EC2 machine Here is my Dockefile : FROM amazonlinux:2 MAINTAINER "xxxxx" <xxxxx#xxxx.xx> RUN yum -y update RUN yum -y install httpd WORKDIR /var/www/html CMD [ "/usr/sbin/httpd","-D","FOREGROUND" ] systemctl do not works : bash-4.2# systemctl list-units Failed to get D-Bus connection: Operation not permitted bash-4.2# systemctl status httpd.service Failed to get D-Bus connection: Operation not permitted I cannot restart bash-4.2# systemctl restart httpd.service Failed to get D-Bus connection: Operation not permitted /etc/os-release bash-4.2# cat /etc/os-release NAME="Amazon Linux" VERSION="2" ID="amzn" ID_LIKE="centos rhel fedora" VERSION_ID="2" PRETTY_NAME="Amazon Linux 2" ANSI_COLOR="0;33" CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2" HOME_URL="https://amazonlinux.com/" No problem on real ec2 : [ec2-user#ip-xxxxxxxxx]$ cat /etc/os-release NAME="Amazon Linux" VERSION="2" ID="amzn" ID_LIKE="centos rhel fedora" VERSION_ID="2" PRETTY_NAME="Amazon Linux 2" ANSI_COLOR="0;33" CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2" HOME_URL="https://amazonlinux.com/" [ec2-user#ip-xxxxxxxxx]$ systemctl status httpd.service ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/httpd.service.d └─php-fpm.conf Active: active (running) since mer. 2020-05-20 16:36:19 UTC; 1h 50min ago Docs: man:httpd.service(8) Main PID: 3047 (httpd) Status: "Total requests: 7; Idle/Busy workers 100/0;Requests/sec: 0.00106; Bytes served/sec: 20 B/sec" CGroup: /system.slice/httpd.service ├─3047 /usr/sbin/httpd -DFOREGROUND ├─3075 /usr/sbin/httpd -DFOREGROUND ├─3076 /usr/sbin/httpd -DFOREGROUND ├─3077 /usr/sbin/httpd -DFOREGROUND ├─3078 /usr/sbin/httpd -DFOREGROUND ├─3079 /usr/sbin/httpd -DFOREGROUND └─3495 /usr/sbin/httpd -DFOREGROUND mai 20 16:36:19 xxxxxxxxx.internal systemd[1]: Starting The Apache HTTP Server... mai 20 16:36:19 xxxxxxxxx.internal systemd[1]: Started The Apache HTTP Server. It's a famous image, has anyone found a solution ?
The "httpd" is run as PID-1 of the container. If you do "docker stop (container)" then httpd will get a SIGTERM. If you do "docker restart (container)" then it will cycle through docker-stop / docker-start also recreating the httpd daemon. https://docs.docker.com/engine/reference/commandline/restart/ Using systemd in a container is not needed here. If you really want to do that then there are multiple options. It could be to run a full systemd in a container or to use another service manager instead. https://github.com/gdraheim/docker-systemctl-images/blob/master/centos-httpd.dockerfile
elasticbeanstalk deployment failing with Error response from daemon:
AWS Elastic Beanstalk failing with error response from daemon: You cannot remove a running container c911e29b2ce2283bae300645bf2f034f7bf51c07b7ec72c6ebfa133d531a6129. Stop the container before attempting removal or force remove Error log bellow: 2020/04/26 07:15:31.279931 [INFO] Running command /bin/sh -c docker rmi `docker images -aq` 2020/04/26 07:15:31.369738 [ERROR] Error response from daemon: conflict: unable to delete 274bac0a2621 (cannot be forced) - image has dependent child images Error response from daemon: conflict: unable to delete 6040329cd021 (cannot be forced) - image has dependent child images Error response from daemon: conflict: unable to delete 1020a81a3f38 (cannot be forced) - image is being used by running container c911e29b2ce2 Error response from daemon: conflict: unable to delete 68e7be49c28c (cannot be forced) - image has dependent child images 2020/04/26 07:15:31.369787 [INFO] restore docker image name with command: docker tag 1020a81a3f38 aws_beanstalk/current-app:latest 2020/04/26 07:15:31.369804 [INFO] Running command /bin/sh -c docker tag 1020a81a3f38 aws_beanstalk/current-app:latest 2020/04/26 07:15:31.414707 [INFO] restore docker image name with command: docker tag 68e7be49c28c python:2.7 2020/04/26 07:15:31.414786 [INFO] Running command /bin/sh -c docker tag 68e7be49c28c python:2.7 2020/04/26 07:15:31.462282 [INFO] Running command /bin/sh -c /bin/file -m /opt/elasticbeanstalk/config/private/beanstalk-magic -b --mime-type /opt/elasticbeanstalk/deployment/app_source_bundle 2020/04/26 07:15:31.464666 [INFO] application/zip 2020/04/26 07:15:31.464679 [INFO] app source bundle is zip file ... 2020/04/26 07:15:31.464689 [INFO] extracting /opt/elasticbeanstalk/deployment/app_source_bundle to /var/app/current/ 2020/04/26 07:15:31.464703 [INFO] Running command /bin/sh -c /usr/bin/unzip -q -o /opt/elasticbeanstalk/deployment/app_source_bundle -d /var/app/current/ 2020/04/26 07:15:31.475355 [INFO] finished extracting /opt/elasticbeanstalk/deployment/app_source_bundle to /var/app/current/ successfully 2020/04/26 07:15:31.475367 [INFO] Executing instruction: RunPreBuildHooks 2020/04/26 07:15:31.475381 [INFO] The dir .platform/hooks/prebuild/ does not exist in the application. Skipping this step... 2020/04/26 07:15:31.475385 [INFO] Executing instruction: Docker Specific Build Application 2020/04/26 07:15:31.475404 [INFO] start build docker app 2020/04/26 07:15:31.475414 [INFO] fetch image name 2020/04/26 07:15:31.475437 [INFO] authenticate with ECR if the image is in an ECR repo 2020/04/26 07:15:31.475442 [INFO] pull docker image if update is not false in dockerrun.aws.json 2020/04/26 07:15:31.475455 [INFO] Running command /bin/sh -c docker pull node:alpine as builder 2020/04/26 07:15:31.516536 [ERROR] "docker pull" requires exactly 1 argument. See 'docker pull --help'. Usage: docker pull [OPTIONS] NAME[:TAG|#DIGEST] Pull an image or a repository from a registry 2020/04/26 07:15:31.516557 [WARN] failed to execute command: docker pull node:alpine as builder, retrying... 2020/04/26 07:15:31.516577 [INFO] Running command /bin/sh -c docker pull node:alpine as builder 2020/04/26 07:15:31.557381 [ERROR] "docker pull" requires exactly 1 argument. See 'docker pull --help'. Usage: docker pull [OPTIONS] NAME[:TAG|#DIGEST] Pull an image or a repository from a registry 2020/04/26 07:15:31.557405 [ERROR] An error occurred during execution of command [app-deploy] - [Docker Specific Build Application]. Stop running the command. Error: failed to pull docker image: Command /bin/sh -c docker pull node:alpine as builder failed with error exit status 1. Stderr:"docker pull" requires exactly 1 argument. See 'docker pull --help'.
We had a problem like this on the following Platform: Docker running on 64bit Amazon Linux 2/3.0.0 Switching to Docker running on 64bit Amazon Linux/2.14.3 solved the problem - Docker image builds as expected.
Can't start the local fabric in VSC environment when trying to deploy smart contract
When using the IBM Blockchain Extension for VS Code; I wrote a sample smart contract and packaged it. But when I try to start the local fabric environment, I get an error saying: Failed to start Local Fabric: Failed to execute command "cmd" with arguments "/c, start.cmd" return code 1 I was following the IBM tutorials for blockchain and installed all required software tools for the extension as listed by IBM in the following link: N/A This is part of the error: [9/2/2019 8:09:46 PM] [INFO] C:\Users\sufij\.fabric-vscode\environments\local_fabric>echo Hyperledger Fabric started in seconds. [9/2/2019 8:09:46 PM] [INFO] Hyperledger Fabric started in seconds [9/2/2019 8:09:46 PM] [INFO] C:\Users\sufij\.fabric-vscode\environments\local_fabric>rem Check to see if the channel already exists [9/2/2019 8:09:46 PM] [INFO] C:\Users\sufij\.fabric-vscode\environments\local_fabric>docker exec -e "CORE_PEER_LOCALMSPID=Org1MSP" -e "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/Admin#org1.example.com/msp" fabricvscodelocalfabric_peer0.org1.example.com peer channel getinfo -c mychannel [9/2/2019 8:09:46 PM] [INFO] Error: No such container: fabricvscodelocalfabric_peer0.org1.example.com [9/2/2019 8:09:46 PM] [INFO] C:\Users\sufij\.fabric-vscode\environments\local_fabric>if errorlevel 1 [9/2/2019 8:09:46 PM] [INFO] ( [9/2/2019 8:09:46 PM] [INFO] rem Create the channel [9/2/2019 8:09:46 PM] [INFO] docker exec -e "CORE_PEER_LOCALMSPID=Org1MSP" -e "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/Admin#org1.example.com/msp" fabricvscodelocalfabric_peer0.org1.example.com peer channel create -o orderer.example.com:17050 -c mychannel -f /etc/hyperledger/configtx/channel.tx [9/2/2019 8:09:46 PM] [INFO] rem Update the channel with the anchor peers [9/2/2019 8:09:46 PM] [INFO] docker exec -e "CORE_PEER_LOCALMSPID=Org1MSP" -e "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/Admin#org1.example.com/msp" fabricvscodelocalfabric_peer0.org1.example.com peer channel update -o orderer.example.com:17050 -c mychannel -f /etc/hyperledger/configtx/Org1MSPanchors.tx [9/2/2019 8:09:46 PM] [INFO] rem Join peer0.org1.example.com to the channel. [9/2/2019 8:09:46 PM] [INFO] docker exec -e "CORE_PEER_LOCALMSPID=Org1MSP" -e "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/Admin#org1.example.com/msp" fabricvscodelocalfabric_peer0.org1.example.com peer channel join -b mychannel.block [9/2/2019 8:09:46 PM] [INFO] ) [9/2/2019 8:09:46 PM] [INFO] Error: No such container: fabricvscodelocalfabric_peer0.org1.example.com [9/2/2019 8:09:46 PM] [INFO] Error: No such container: fabricvscodelocalfabric_peer0.org1.example.com [9/2/2019 8:09:46 PM] [INFO] Error: No such container: fabricvscodelocalfabric_peer0.org1.example.com [9/2/2019 8:09:46 PM] [ERROR] Failed to start Local Fabric: Error: Failed to execute command "cmd" with arguments "/c, start.cmd" return code 1