I am facing the issue
(gcloud.run.deploy) Cloud Run error: Container failed to start. Failed
to start and then listen on the port defined by the PORT environment
variable. Logs for this revision might contain more information.
There are a few post with this error but I couldn't find my particular case.
I am running a background task, nothing to expose, it connects to firebase process some data and store it back. I wanted this process to run on a container on Cloud Run so I made it a container, which runs perfectly locally, but when uploading it to CR it fails with the above error.
I tried to expose 8080 on dockerfile and a few more things but if you try to connect to they container it has no server running to connect to. It is a batch task.
Can anyone tell me if it is possible at all to upload this type of tasks to Cloud Run, I do not know how to solve the issue. I wouldnt believe google requires a server running on the container to allow it, I saw some posts with dev pulling an nginx on the image so they can expose the port but this would be totally unnecessary in my case.
Thanks for your advice
UPDATE
Cloud Logging: The error simply say there was a fail to start the container, which is funny because the container starts and also shows some logs like if it were working but then it stops.
Build on MAC yes.
DockerFile is pretty simple.
FROM openjdk:11
ENV NOTIFIER_HOME /opt/app/
ENV NOTIFIER_LOGS /opt/notifications/logs/
RUN mkdir -p $NOTIFIER_HOME RUN mkdir -p $NOTIFIER_LOGS
RUN apt update
#RUN apt install curl
COPY docker/* $NOTIFIER_HOME
EXPOSE 8080
ENV TMP_OPTS -Djava.io.tmpdir=/tmp ENV LOG4j_OPTS
-Dlog4j.configurationFile=$NOTIFIER_HOME/logback.xml ENV NOTIFIER_OPTS $TMP_OPTS $LOG4j_OPTS
ENV JAVA_GC_OPTS -Xms1g -Xmx1g
WORKDIR $NOTIFIER_HOME ENTRYPOINT ["sh", "-c", "/opt/app/entrypoint.sh"]
You can't run background jobs on Cloud Run. Wrap it in a webserver as proposed by MBHA if the process take less than 1h.
Else you can you GKE Autopilot to run your container for a while. you pay only when your container run. And the first cluster is free. You can have a try on it!
As hack you can run your container in Cloud Build also, or in Vertex AI custom container training.
I've run in to a similar issue with building custom image on MAC + deploying in to Cloud Run. In my case, it turned out to be the docker platform causing the problem. The way I isolated this was by building the same image in Cloud Shell and that would work perfectly fine in Cloud Run.
Now, if you need to build it locally on MAC go ahead and test it by changing the Docker platform:
export DOCKER_DEFAULT_PLATFORM=linux/amd64
docker build -t mytag:myver .
Once the image has been built, you can inspect the architecture:
docker image inspect mytag:myver | grep -i Architecture
Then deploy it to Cloud Run.
The explanation is in your question:
I am running a background task, nothing to expose
A cloud run application, so your container, must be listening for incoming HTTP requests as stated in the Container runtime contract. That's why in all cloud run examples, java in your case, spring boot is used with #RestController. Other explanation can be found in this answer.
Update:
So the solution is either to
add a webserver to your code and wrap it with spring boot and controller logic
use Cloud Function rather than Cloud Run and get rid of the Dockerfile and in the same time have simpler code and less configuration
Related
I used to run a long training process on a remote server with GPU capabilities. Now my work schedule changes, so I can't have my computer connected to a network all the time till I finish the process. I found that nohup is the solution for me. But I don't know how to keep invoke the process correctly related my situation.
I use ssh to connect to the remote server.
I have to use docker to access to GPU.
Then I start the process in the docker.
If I start the process with nohup in docker, I can't really leave docker, right. So, do I use nohup at each step?
Edit:
I need the terminal output of the process at step 3, because I need that information to carry out the rest of the work. Consider, step 3 is training a neural network. So, the training log tells me the accuracy of different models at different iterations. I use that information to do the testing.
Following #David Maze's suggestion, I did this (a slightly different approach as I was not familiar with docker a whole lot)
Logged in to the remote server.
Configured the docker script to have remote workdir.
...
WORKDIR /workspace
...
After building the docker container, run docker with mount option to mount the local project to docker workdir. When running docker, I used nohup. Since I don't need interactive mode I ignored the -it flag.
nohup docker run --gpus all -v $(pwd)/path-to-project-root:/workspace/ docker-image:tag bash -c "command1; command2" > project.out 2>&1 &
To test this, I logged out from the server and see the content of project.out later. It contained the expected output.
I have a NodeJS application that is using ioredis to connect to redis and publish data and other redisy things.
I am trying to write a component test against redis and was able to create a setup/teardown script via jest that runs redis via docker on a random port and tears it down when the tests are done via docker run -d -p 6379 --rm redis and docker stop {containerId}.
This works great locally, but we have the tests running in a multi-stage build in our Dockerfile:
RUN yarn test
which I try to build via docker build . it goes great until it gets to the tests and then complains with the following error - /bin/sh: docker: not found
Hence, Docker is unavailable to the docker-build process to run the tests?
Is there a way to run docker-build to give it the ability to spin up sibling processes during the process?
This smells to me like a "docker-in-docker" situation.
You can't spin up siblings, but you can spawn a container within a container, by doing some tricks: (you might need to do some googling to get it right)
install the docker binaries in the "host container"
mount the docker socket from the actual host inside the "host" container, like so docker run -v /var/run/docker.sock:/var/run/docker.sock ...
But you won't be able to do it in the build step, so it won't be easy for your case.
I suggest you prepare a dedicated build container capable of running nested containers, which would basically emulate your local env and use that in your CI. Still, you might need to refactor your process a bit make it work.
Good luck :)
In my practice, tests shouldn't be concerned with initializing the database, they should only be concerned about how to connect to the database, so you just pass your db connection data via environment variables.
The way you are doing it it won't scale, imagine that you need a lot more services for your application, it will be difficult and not practical to start them via tests.
When you are developing locally, it's your responsibility to have the services running before doing the tests.
You can have docker compose scripts in your repository that create and start all the services you need when you start developing.
And when you are using CI in the cloud, you would still use docker containers and run tests in them( node container with your tests, redis container, mysql container, etc...) and again just pass the appropriate connection data via environment variables.
I am trying to play with Google Cloud Run, I have the same service that works fine in App Engine Flex. Any thoughts what could be the issue?
Somehow it shows that service is healthy.
This means infrastructure (container manager) scales down the number instances when traffic drops.
It's safe to ignore.
For others who find this question when their container didn't start the first time you deployed it: It's important to note that you need to have it listening on the environment variable PORT.
It appears that Cloud Run will dynamically map your container to a port at invocation, and the service that you're running needs to (dynamically) use this to serve it's content.
For reference, here's how I got the base Apache Docker image to work with Cloud Run to host a static site built via Node:
FROM node:lts AS build
COPY . .
RUN npm install
RUN npm run build
FROM httpd:latest
ENV PORT=80
RUN sed -i 's/80/${PORT}/g' /usr/local/apache2/conf/httpd.conf
COPY --from=build ./dist/ /usr/local/apache2/htdocs/
For me, it was because billing was disabled
Make sure billing is enabled on your GCP project
https://console.cloud.google.com/billing
In my Dockerfile I am trying to install multiple services and want to have them all start up automatically when I launch the container.
One among the services is mysql and when I launch the container I don't see the mysql service starting up. When I try to start manually, I get the error:
Failed to get D-Bus connection: Operation not permitted
Dockerfile:
FROM centos:7
RUN yum -y install mariadb mariadb-server
COPY start.sh start.sh
CMD ["/bin/bash", "start.sh"]
My start.sh file:
service mariadb start
Docker build:
docker build --tag="pbellamk/mariadb" .
Docker run:
docker run -it -d --privileged=true pbellamk/mariadb bash
I have checked the centos:systemd image and that doesn't help too. How do I launch the container with the services started using systemctl/service commands.
When you do docker run with bash as the command, the init system (e.g. SystemD) doesn’t get started (nor does your start script, since the command you pass overrides the CMD in the Dockerfile). Try to change the command you use to /sbin/init, start the container in daemon mode with -d, and then look around in a shell using docker exec -it <container id> sh.
Docker is designed around the idea of a single service/process per container. Although it definitely supports running multiple processes in a container and in no way stops you from doing that, you will run into areas eventually where multiple services in a container doesn't quite map to what Docker or external tools expect. Things like moving to scaling of services, or using Docker swarm across hosts only support the concept of one service per container.
Docker Compose allows you to compose multiple containers into a single definition, which means you can use more of the standard, prebuilt containers (httpd, mariadb) rather than building your own. Compose definitions map to Docker Swarm services fairly easily. Also look at Kubernetes and Marathon/Mesos for managing groups of containers as a service.
Process management in Docker
It's possible to run systemd in a container but it requires --privileged access to the host and the /sys/fs/cgroup volume mounted so may not be the best fit for most use cases.
The s6-overlay project provides a more docker friendly process management system using s6.
It's fairly rare you actually need ssh access into a container, but if that's a hard requirement then you are going to be stuck building your own containers and using a process manager.
You can avoid running a systemd daemon inside a docker container altogether. You can even avoid to write a special start.sh script - that is another benefit when using the docker-systemctl-replacement script.
The docker systemctl.py can parse the normal *.service files to know how to start and stop services. You can register it as the CMD of an image in which case it will look for all the systemctl-enabled services - those will be started and stopped in the correct order.
The current testsuite includes testcases for the LAMP stack including centos, so it should run fine specifically in your setup.
I found this project:
https://github.com/defn/docker-systemd
which can be used to create an image based on the stock ubuntu image but with systemd and multiuser mode.
My use case is the first one mentioned in its Readme. I use it to test the installer script of my application that is installed as a systemd service. The installer creates a systemd service then enables and starts it. I need CI tests for the installer. The test should create the installer, install the application on an ubuntu, and connect to the service from outside.
Without systemd the installer would fail, and it would be much more difficult to write the test with vagrant. So, there are valid use cases for systemd in docker.
My goal is to use Docker to create a mail setup running postfix + dovecot, fully configured and ready to go (on Ubuntu 14.04), so I could easily deploy on several servers. As far as I understand Docker, the process to do this is:
Spin up a new container (docker run -it ubuntu bash).
Install and configure postfix and dovecot.
If I need to shut down and take a break, I can exit the shell and return to the container via docker start <id> followed by docker attach <id>.
(here's where things get fuzzy for me)
At this point, is it better to export the image to a file, import on another server, and run it? How do I make sure the container will automatically start postfix, dovecot, and other services upon running it? I also don't quite understand the difference between using a Dockerfile to automate installations vs just installing it manually and exporting the image.
Configure multiple docker images using Dockerfiles
Each docker container should run only one service. So one container for postfix, one for another service etc. You can have your running containers communicate with each other
Build those images
Push those images to a registry so that you can easily pull them on different servers and have the same setup.
Pull those images on your different servers.
You can pass ENV variables when you start a container to configure it.
You should not install something directly inside a running container.
This defeat the pupose of having a reproducible setup with Docker.
Your step #2 should be a RUN entry inside a Dockerfile, that is then used to run docker build to create an image.
This image could then be used to start and stop running containers as needed.
See the Dockerfile RUN entry documentation. This is usually used with apt-get install to install needed components.
The ENTRYPOINT in the Dockerfile should be set to start your services.
In general it is recommended to have just one process in each image.