I'm a beginner in docker, as well as in team city, I set up a pipeline for a build of a docker container and wanted to configure it to run after a successful build, I tried to use a step with a docker, but they advise using the command line with executable parameter and some way with docker socket, I crossed the Internet / YouTube did not see normal examples for starting a container after a build. I saw some examples of launching with agents, but again I did not understand anything in what was written, I looked for examples on YouTube, I also did not find it. Please give an example of running docker as a step in the pipeline on Linux.
I solved my similar requirement on Jenkins by applying following..
Add a shell file (e.g. run.sh) in your project. In there have the docker run command that you will use from command line adding > /dev/null 2>&1 & at the end so that the process can be run in background and O/P streams to null.
docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:tag > /dev/null 2>&1 &
Then in your Jenkins (Teamcity) script add a sh step to run this file
steps {
dir (whatever-dir-run.sh-is-in) {
sh "JENKINS_NODE_COOKIE=dontKillMe sh run.sh"
}
}
Note: If JENKINS_NODE_COOKIE has an equivalent in Teamcity, use that.
Related
I'm trying to add a script to a docker run command , command i'm using is :
docker run -dit --name 1.4 ubuntu sh -c 'echo "Input website:"; read website; echo "Searching.."; sleep 1; curl http://$website;'
and then install curl , then enter a website as input and it should reply to me as per the course i'm studying , but running this exact command makes the container exit immediately
any guidance on why would that be ?
also how should i send the input to the container so it can use it afterwards , do i just attach to it after installing curl in the terminal ?
I'm going to recommend an extremely different workflow from what you suggest. Rather than manually installing software and trying to type arguments into the stdin of a shell script, you can build this into a reusable Docker image and provide its options as environment variables.
In comments you describe a workflow where you first start a container, then get a debugging shell inside of it, and then install curl. Unless you're really truly debugging, this is a pretty unusual workflow: anything you install this way will get lost as soon as the container exits, and you'll have to repeat this step every time you re-run the container. Instead, create a new empty directory, and inside that create a file named Dockerfile (exactly that name, no extension, capital D) containing
# Start our new image from this base
FROM ubuntu
# Install any OS-level packages we need
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive \ # avoid post-installation questions
apt-get install \
--no-install-recommends \ # don't install unneeded extra packages
--assume-yes \ # (-y) skip an "are you sure" prompt
curl
Rather than try to read from the container's input, you can take the URL as an environment variable. In most cases the best way to give the main command to a container is by specifying it in the Dockerfile. You can imagine running a larger script or program here as well, and it would take the some environment-variable setting (using Python's os.environ, Node's process.env, Ruby's ENV, etc.).
In our case, let's make the main container command be the single curl command that you're trying to run. We haven't specified the value of the environment variable yet, and that's okay: this shell command isn't evaluated until the container actually runs.
# at the end of the Dockerfile
CMD curl "$website"
Now let's build and run it. When we do launch the container, we need to provide that $website environment variable value, which we can do with a docker run -e option.
# Build the image:
docker build \
-t my/curl # giving it a name
. # using the content in the current directory
docker run \
--rm # deleting the container when done
-e website=https://stackoverflow.com \
my/curl # with the same name as above
So note that we're starting the container in the foreground (no -d option) since we want to see its output and we expect it to exit promptly; we're cleaning up the container when it's done; we're not trying to pass a full shell script as a command-line argument; and we are providing our options on the command line, so we don't need to make the container's stdin work (no -i or -t option).
A Docker container is a wrapper around a single process. When that process exits, the container exits too. In this example, the thing you want the container to do is run a curl command; that's not a long-running process, hence docker run --rm but not -d. There's not an "afterwards" here, if you need to query a different Web site then launch a new container. It's very normal to destroy and recreate containers, especially since there are many options that can only be specified when you first start a container.
With the image and container we've built here, in fact, it's useful to think about them as analogous to the /usr/bin/curl binary on your host. You build it once into a reusable artifact (here the Docker image), and you run multiple instances of it (curl commands or new Docker containers) giving options on the command line at startup time. You do not typically "get a shell" inside a curl command-line invocation, and I'd similarly avoid docker exec outside of debugging tasks.
You can also use alpine/curl image to use curl command without needing to install anything.
First start the container in detached mode with -d flag.
Then run your script with exec sub command.
docker run -d --name 1.4 alpine/curl sleep 600
docker exec -it 1.4 sh -c 'echo "Input website:"; read website; echo "Searching.."; sleep 1; curl http://$website;'
I've created a docker image with all the modules required for our build environment. If I start a container in interactive mode, I can build fine.
docker run -v <host:container> -w my_working_dir -it my_image
$make -j16
But if I try to do this from a command line I get compile failures (well into the process)
docker run -v <host:container> -w my_working_dir my_image bash -c "make -j16"
Also if I run the container detached and use docker exec I also get compile failures (same point)
docker run -v <host:container> -t --detach --name star_trek my_image
docker exec star_trek bash -c "cd my_working_dir; make -j16"
Entering an interactive session with the detached container also seems seems to pass though I though I have seen this fail as well.
docker exec -it star_trek_d bash
$make -j16
This will be part of an automated build system so I need to be able run this without user intervention.
I'm not sure why these are behaving differently but I ran multiple combination and the only way I've been able to get a success build is through the interactive method above. Other then the interactive system having more of a logged in user configuration, what is the difference between running interactive or passing on command line?
My preferred method would to be run the container detached so I can send several sequential commands as we have a complex build and test process but if I have to spin the container up each time I'm OK with that as this point because I really need to get this running like last week.
*Commands are pseudo-code and simplified to aid visibility and using bash -c because I'm needing to run a script for our test and therefore doing something like bash -c "my_script.sh; run_test"
UPDATE - We need custom paths for our build tools. I believe this is not working except in the interactive session. Our /etc/bashrc file is used to build the correct path and export it. When I do a docker run I've tried running a script that does a "source /etc/bashrc", among other initialization things we need, before doing the make but this doesn't seem to work. Note have to pipe in password as this needs to be run using sudo. The other commands seem to work fine.
bash -c 'echo su_password | sudo -S /tmp/startup.sh; make -j16'
I've also tried to just set on command without success
bash -c 'export <path>; make -j16'
What is the best way to set the path in the container so installed applications can be found? I don't want to hard code them in the dockerfile but will at this point if I must.
I have this working. As our path is very long I set it to a variable and was passing it in on the command line. Seems this was causing issues.
export PATH=$PATH/...
vs
export PATH=$PATH:/...
Now I am just specifying the whole path each time and everything is working.
bash -c 'export PATH=$PATH/<dir>/<program>/bin:/<dir>/<program>/bin:...; make -j16'
Here's the situation:
I have a docker container (jenkins). I've mounted the sockets to my container so that I can perform docker commands inside my jenkins container.
Manually, everything works in the container. However, when Jenkins executes the job, it doesn't "wait" for the docker exec command to run to completion.
Below, is an extract from the Jenkinsfile. The short-lived printenv command runs correctly, and prints the environment variables. The next command (python) just gets run and then Jenkins moves on immediately, not waiting for completion. The Jenkins agent (slave) is running on an Ubuntu image. Running all these commands outside Jenkins work as expected.
echo "Running the app docker container in detached tty mode to keep it up"
docker run --detach --tty --name "${CONTAINER_NAME}" "${IMAGE_NAME}"
echo "Listing environment variables"
docker exec --interactive "${CONTAINER_NAME}" bash -c "printenv"
echo "Running test coverage"
docker exec --interactive "${CONTAINER_NAME}" bash -c "python -m coverage run --source . --branch -m pytest -vs"
It seems maybe related to this question.
Please can anyone explain how to get Jenkins to wait for the docker exec command to complete before proceeding to the next step.
Have considered alternatives, like the Docker Pipeline Plugin, but would much prefer to use something close to what I have above where possible.
Ok, another approach, I've tried using Docker Pipeline plugin here.
You can use docker.sock as volume mount to orchestrate containers on your host machine like this in your docker-compose.yml
volumes:
- /var/run/docker.sock:/var/run/docker.sock
Depending on your setup you might need to run
chmod 666 /var/run/docker.sock
to get going in the first place.
This works on macOS as well as Linux.
Ugh. This was down to the way that I'd set up docker support on the slave container.
I'd used socat to provide a TCP server proxy. Instead, switched that out for a plain old docker.sock volume between host & container.
volumes:
- /var/run/docker.sock:/var/run/docker.sock
The very first time, I had to also sort out a permissions issue by doing (inside the container):
rm -Rf ~/.docker
chmod 666 /var/run/docker.sock
After that, everything "just worked". Very painful experience.
I need to set environment variable in a running docker container. I am already aware of the way of setting environment variable while creating a container. As far I found there is no available straight forward way to do this with docker and docker is planning to add something with new version 1.13.
But I found that some people able to manage it which is not working for me now. I tried following ways but did not work for me-
docker exec -it -u=root test /bin/bash -c "export port=8090"
echo "export port=8090" to /etc/bash.bashrc using a script and then source it
docker exec -it test /bin/bash -c "source /etc/bash.bashrc"
configuring the whole thing in a script and run it from host also did not work. While running script from host all the other command successfully executes except "export port=8090" or "source /etc/bash.bashrc" or "source /root/.bashrc".
Can anyone explain why sourcing file from host does not work in docker container even when I set user("-u=root")? Can anyone help me to solve this? When I source the file from inside the container it works perfectly. But in my case I have to do it from host machine
NOTE:, I am using docker 1.12 and tried the above in ubuntu:16.04 and ubuntu:14.04
If you have a running process in the docker and you are attempting to change the environment variable in the docker so the running process will dynamically change - this will not work. The environment variables of a process are set when it starts. You can see here ways to overcome that, but I don't think that is the right way to go.
I would instead, have a configuration file that the file reads (or listens to) periodically. And when you want to change the configuration change the file.
If this isn't your scenario, please describe your scenario so we can better assist you.
I find a way to provide environment variable to a running container. Fist upgrade your docker-engine. I am using V1.12.5.
create a script with environment variables-
#!/bin/bash
echo "export VAR1=VAL1
export VAR2=VAL2" >> /etc/bash.bashrc
source /etc/bash.bashrc
Now start a container. Here, 'test' is the container name:
docker run -idt --name=test ubuntu
Copy your script to container:
docker cp script.sh test:/
Run the script :
docker exec -it test /bin/bash -c "/script.sh"
Restart your container:
docker restart test
Go to container shell
docker exec -it test /bin/bash
Check the variable
echo $VAR1
I'm trying to create a Docker setup (using docker-compose) to test one of my Python applications during development. It the docker-compose.yml starts up a Postgres Server, a Redis server and a PhantomJS server and then runs the tests using pytest.
This is what my test.sh looks like:
#!/bin/bash
UP=$(docker-compose up -d redis postgres phantomjs 2>&1)
echo $UP
if [[ $UP == *"Starting radar_postgres"* ]]; then
echo "Sleeping 10 seconds to wait for PostgreSQL server..."
sleep 10
fi
docker-compose build tests && \
docker-compose run \
--rm \
-e GOOGLE_OAUTH2_CLIENT_ID='$GOOGLE_OAUTH2_CLIENT_ID' \
-e GOOGLE_OAUTH2_CLIENT_SECRET='$GOOGLE_OAUTH2_CLIENT_SECRET' \
-e GOOGLE_DEVELOPER_TOKEN='$GOOGLE_DEVELOPER_TOKEN' \
tests $#
First the dependencies are started. Due to the way docker-compose up works, they're automatically rebuilt when necessary.
Then I run a one-off job in my tests container. I use a one-off job instead of using docker-compose up because this way I can pass in arguments to the test framework.
The problem is that the container is always being rebuilt, even if the Dockerfile didn't change and no rebuild would be necessary. (Of course each step uses the cache, but it still takes 4-5 seconds.) On contrast, if I leave away the docker-compose build line, then the container is not being rebuilt when I change the Dockerfile.
Is there a way to rebuilt an image only if necessary?
Note that there is a discussion (issue 1455) to remove build from docker-compose.
Compose's primary job is orchestration and not building, and that the Docker image is the natural place to draw that line
So it would be best to use docker build commands (which should only build images when necessary), instead of using docker-compose build (which might build a bit too aggressively)