I'm getting some unexpected behaviour when trying to run a GitLab CI job inside a custom Docker container. It runs as expected outside GitLab. I don't even know what it is I'm not understanding... perhaps there's something about how GitLab runs docker containers that I'm missing, or how Docker ENTRYPOINT and CMD work together, or is it even just a mistake in my shell script?
My custom Docker image is intended to run Percy CLI commands. It's based on node:12-slim and basically just installs Puppeteer and Percy. It sets the ENTRYPOINT to a custom shell script:
ENTRYPOINT ["/usr/bin/local/percy-entrypoint.sh"]
but doesn't override the base image CMD.
My entrypoint script essentially just checks for a couple of env vars (and chucks out some debug stuff), then runs a "sitemap" function:
echo "All args:"
echo "$#"
if [ "$1" = "sitemap" ] ; then
percy exec -- node /sitemap.js
else
exec "$#"
fi
I'm running it from a GitLab-CI job as:
snapshot:
stage: test
image: my/percy-buildbox:latest
script:
- sitemap
variables:
PERCY_TOKEN: ${PERCY_TOKEN}
SITEMAP_URL: percy-sitemap.xml
The output generated by this job is:
All args:
sh -c if [ -x /usr/local/bin/bash ]; then
exec /usr/local/bin/bash
elif [ -x /usr/bin/bash ]; then
exec /usr/bin/bash
elif [ -x /bin/bash ]; then
exec /bin/bash
elif [ -x /usr/local/bin/sh ]; then
exec /usr/local/bin/sh
elif [ -x /usr/bin/sh ]; then
exec /usr/bin/sh
elif [ -x /bin/sh ]; then
exec /bin/sh
elif [ -x /busybox/sh ]; then
exec /busybox/sh
else
echo shell not found
exit 1
fi
$ sitemap
/bin/bash: line 134: sitemap: command not found
I don't even... where's all that if stuff coming from? What's it doing? Where am I going wrong?
Related
I am running into a problem where my shell script doesn't work if the variable is not set/set to empty string.
For this I am using the alpine image
docker run -dt alpine
docker exec -it <container> sh
Here is the problematic code:
x=""
sh -c "if [ "$x" != "required" ]; then sed; fi"
When x is not set, I get the error:
sh: required: unknown operand
This seems to only be the problem on an empty string. If I set x="lkajsdfasl", it will work just fine.
This just breaks down on an empty string/not set.
Due to the way my docker-compose is setup, I can only use sh and have to use sh -c
It's a quoting issue. Use single quotes around the command like this
sh -c 'if [ "$x" != "required" ]; then sed; fi'
I've got pipelines for dev, staging and production.
The staging pipeline is where I've got the issue. The pipeline builds just fine on dev (on and off the CI runner) but staging code builds only locally and on live server but will fail
in the CI runner. I indicated suspecting code with <--.
I've checked whether the database container is running at the time of testing and it is up and running. Logs show nothing unusual.
Cypress tests fail on tests where interaction with the database is being tested:
test-ci.sh:
#!/bin/bash
env=$1
fails=""
inspect() {
if [ $1 -ne 0 ]; then
fails="${fails} $2"
fi
}
# run server-side tests
dev() {
docker-compose up -d --build
docker-compose exec -T users python manage.py recreate_db
docker-compose exec -T users python manage.py test
inspect $? users
docker-compose exec -T client npm test -- --coverage --watchAll --watchAll=false
inspect $? client
docker-compose down
}
# run e2e tests
e2e() {
if [ "${env}" = "staging" ]; then
docker-compose -f docker-compose-stage.yml up -d --build
docker-compose -f docker-compose-stage.yml exec -T users python manage.py recreate_db # <--
docker run -e REACT_APP_USERS_SERVICE_URL=$REACT_APP_USERS_SERVICE_URL -v $PWD:/e2e -w /e2e -e CYPRESS_VIDEO=$CYPRESS_VIDEO --network flaskondocker_default cypress/included:6.0.0 --config baseUrl=http://nginx
inspect $? e2e
docker-compose -f docker-compose-stage.yml down
else
docker-compose -f docker-compose-prod.yml up -d --build
docker-compose -f docker-compose-prod.yml exec -T users python manage.py recreate_db
docker run -e REACT_APP_USERS_SERVICE_URL=$REACT_APP_USERS_SERVICE_URL -v $PWD:/e2e -w /e2e -e CYPRESS_VIDEO=$CYPRESS_VIDEO --network flaskondocker_default cypress/included:6.0.0 --config baseUrl=http://nginx
inspect $? e2e
docker-compose -f docker-compose-prod.yml down
fi
}
# run specific tests
if [ "${env}" = "staging" ]; then
echo "****************************************"
echo "Running e2e tests ..."
echo "****************************************"
e2e
elif [ "${env}" = "production" ]; then
echo "****************************************"
echo "Running e2e tests ..."
echo "****************************************"
e2e
else
echo "****************************************"
echo "Running client and server-side tests ..."
echo "****************************************"
dev
fi
if [ -n "${fails}" ]; then
echo "Test failed: ${fails}"
exit 1
else
echo "Tests passed!"
exit 0
fi
The tests are behaving like docker-compose -f docker-compose-stage.yml exec -T users python manage.py recreate_db failed or hasn't been executed but logs show no errors.
gitlab-ci.yml file:
image: docker:stable
services:
- docker:19.03.12-dind
variables:
COMMIT: ${CI_COMMIT_SHORT_SHA}
MAIN_REPO: https://gitlab.com/coding_hedgehog/flaskondocker.git
USERS: training-users
USERS_REPO: ${MAIN_REPO}#${CI_COMMIT_BRANCH}:services/users
USERS_DB: training-users-db
USERS_DB_REPO: ${MAIN_REPO}#${CI_COMMIT_BRANCH}:services/users-db
CLIENT: training-client
CLIENT_REPO: ${MAIN_REPO}#${CI_COMMIT_BRANCH}:services/client
SWAGGER: training-swagger
SWAGGER_REPO: ${MAIN_REPO}#${CI_COMMIT_BRANCH}:services/swagger
stages:
- build
- push
before_script:
- export REACT_APP_USERS_SERVICE_URL=http://127.0.0.1
- export CYPRESS_VIDEO=false
- export SECRET_KEY=pythonrocks
- export AWS_ACCOUNT_ID=nada
- export AWS_ACCESS_KEY_ID=nada
- export AWS_SECRET_ACCESS_KEY=nada
- apk add --no-cache py-pip python2-dev python3-dev libffi-dev openssl-dev gcc libc-dev make npm
- pip install docker-compose
- npm install
compile:
stage: build
script:
- docker pull cypress/included:6.0.0
- sh test-ci.sh $CI_COMMIT_BRANCH
deployment:
stage: push
script:
- sh ./docker-push.sh
when: on_success
Let me just emphasize that the tests are passing locally on my computer as well as on live server. The database-related e2e tests fail when ran headlessly within CI.
What debugging steps I can take knowing that no containers are crashing, logs show no errors, same code builds locally and runs OK live but fails in the CI ?
We have had some issues where database checks worked locally, but not in headless CI. We found out that it was because of datetime fields. The markup response in CI was different than locally. Thus, all assertions that checked dates failed. We fixed this by writing MySQL queries that format the datetime result. Then adjust the assertions in Cypress accordingly. Maybe your problem has to do with this issue.
SELECT DATE_FORMAT(columnname, "%d-%c-%Y") as columnname FROM table
So for further debugging, do you have any simple tests that run correctly in CI? Or does nothing work?
I am trying to import a pipeline into streamsets, during container start up, by using the Docker CMD command in Dockerfile. The image builds, but while creating the container there is no error but it exits with code 0. So it never comes up. Here is what I did:
Dockerfile:
FROM streamsets/datacollector:3.18.1
COPY myPipeline.json /pipelinejsonlocation/
EXPOSE 18630
ENTRYPOINT ["/bin/sh"]
CMD ["/opt/streamsets-datacollector-3.18.1/bin/streamsets","cli","-U", "http://localhost:18630", \
"-u", \
"admin", \
"-p", \
"admin", \
"store", \
"import", \
"-n", \
"myPipeline", \
"--stack", \
"-f", \
"/pipelinejsonlocation/myPipeline.json"]
Build image:
docker build -t cmp/sdc .
Run image:
docker run -p 18630:18630 -d --name sdc cmp/sdc
This outputs the container id. But the container is in the Exited status as shown below.
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
537adb1b05ab cmp/sdc "/bin/sh /opt/stream…" 5 seconds ago Exited (0) 3 seconds ago sdc
When I do not specify the CMD command in the Dockerfile, the streamsets container spins up and then when I run the streamsets import command in the running container in shell, it works. But how do I get it done during provisioning itself? Is there something I am missing in the Dockerfile?
In your Dockerfile you overwrite the default CMD and ENTRYPOINT from the StreamSets Data Collector Dockerfile. So the container only executes your command during startup and exits without errors afterwards. This is the reason why your container is in Exited (0) status.
In general this is good and expected behavior. If you want to keep your container alive you need to execute another command in the foreground, which never ends. But unfortunately, you cannot run multiple CMDs in your docker file.
I dug a little deeper. The default entry point of the image is ENTRYPOINT ["/docker-entrypoint.sh"]. This script sets up a few things and starts the Data Collector.
It is required that the Data Collector is running before the pipeline is imported. So a solution could be to copy the default docker-entrypoint.sh and modify it to start the Data Collector and import the pipeline afterwards. You could to it like this:
Dockerfile:
FROM streamsets/datacollector:3.18.1
COPY myPipeline.json /pipelinejsonlocation/
# Replace docker-entrypoint.sh
COPY docker-entrypoint.sh /docker-entrypoint.sh
EXPOSE 18630
docker-entrypoint.sh (https://github.com/streamsets/datacollector-docker/blob/master/docker-entrypoint.sh):
#!/bin/bash
#
# Copyright 2017 StreamSets Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
set -e
# We translate environment variables to sdc.properties and rewrite them.
set_conf() {
if [ $# -ne 2 ]; then
echo "set_conf requires two arguments: <key> <value>"
exit 1
fi
if [ -z "$SDC_CONF" ]; then
echo "SDC_CONF is not set."
exit 1
fi
grep -q "^$1" ${SDC_CONF}/sdc.properties && sed 's|^#\?\('"$1"'=\).*|\1'"$2"'|' -i ${SDC_CONF}/sdc.properties || echo -e "\n$1=$2" >> ${SDC_CONF}/sdc.properties
}
# support arbitrary user IDs
# ref: https://docs.openshift.com/container-platform/3.3/creating_images/guidelines.html#openshift-container-platform-specific-guidelines
if ! whoami &> /dev/null; then
if [ -w /etc/passwd ]; then
echo "${SDC_USER:-sdc}:x:$(id -u):0:${SDC_USER:-sdc} user:${HOME}:/sbin/nologin" >> /etc/passwd
fi
fi
# In some environments such as Marathon $HOST and $PORT0 can be used to
# determine the correct external URL to reach SDC.
if [ ! -z "$HOST" ] && [ ! -z "$PORT0" ] && [ -z "$SDC_CONF_SDC_BASE_HTTP_URL" ]; then
export SDC_CONF_SDC_BASE_HTTP_URL="http://${HOST}:${PORT0}"
fi
for e in $(env); do
key=${e%=*}
value=${e#*=}
if [[ $key == SDC_CONF_* ]]; then
lowercase=$(echo $key | tr '[:upper:]' '[:lower:]')
key=$(echo ${lowercase#*sdc_conf_} | sed 's|_|.|g')
set_conf $key $value
fi
done
# MODIFICATIONS:
#exec "${SDC_DIST}/bin/streamsets" "$#"
check_data_collector_status () {
watch -n 1 ${SDC_DIST}/bin/streamsets cli -U http://localhost:18630 ping | grep -q 'version' && echo "Data Collector has started!" && import_pipeline
}
function import_pipeline () {
sleep 1
echo "Start to import pipeline"
${SDC_DIST}/bin/streamsets cli -U http://localhost:18630 -u admin -p admin store import -n myPipeline --stack -f /pipelinejsonlocation/myPipeline.json
echo "Finished importing pipeline"
}
# Start checking if Data Collector is up (in background) and start Data Collector
check_data_collector_status & ${SDC_DIST}/bin/streamsets $#
I commented out the last line exec "${SDC_DIST}/bin/streamsets" "$#" of the default docker-entrypoint.sh and added two functions. check_data_collector_status () pings the Data Collector service until it is available. import_pipeline () imports your pipeline.
check_data_collector_status () runs in background and ${SDC_DIST}/bin/streamsets $# is started in foreground as before. So the pipeline is imported after the Data Collector service is started.
Run this image with sleep command:
docker run -p 18630:18630 -d --name sdc cmp/sdc sleep 300
300 is the time to sleep in seconds.
Then exec your script manually within the docker container and find out what's wrong.
I am trying to find a "global" solution for injecting an SSH key into a container. I know that there are several solutions including docker build kit and so on...but I don't want to build an image and inject the SSH key. I want to inject the SSH key by using an existing image with docker compose.
I use the following docker compose file:
version: '3.1'
services:
server1:
image: XXXXXXX
container_name: server1
command: bash -c "/root/init.sh && python3 /root/my_python.py"
environment:
- MANAGED_HOST=mserver
volumes:
- ./init.sh:/root/init.sh
secrets:
- id_rsa
secrets:
id_rsa:
file: /home/user/.ssh/id_rsa
The init.sh is as follows:
#!/bin/bash
eval "$(ssh-agent -s)" > /dev/null
if [ ! -d "/root/.ssh/" ]; then
mkdir /root/.ssh
ssh-keyscan $MANAGED_HOST > /root/.ssh/known_hosts
fi
ssh-add -k /run/secrets/id_rsa
If I run docker compose with the parameter command
bash -c "/root/init.sh && python3 /root/my_python.py", then the SSH authentication to the appropriate remote host ($MANAGED_HOST) is not working.
An agent process is running:
root 8 1 0 12:50 ? 00:00:00 ssh-agent -s
known_hosts is OK:
root#c67655d87ced:~# cat /root/.ssh/known_hosts
BLABLABLA ssh-rsa AAAAB3BLABLABLA....
and the agent is running, but the private key is not added:
root#c67655d87ced:~# ssh-add -l
Could not open a connection to your authentication agent.
Now, if I log in the container (docker exec -it server1 /bin/bash) and run the commands from init.sh one by one from the command line, then the SSH authentication to the appropriate remote host ($MANAGED_HOST) is working?!?
Any idea, how I can get it working by using the docker compose?
It should be enough to cause the file $HOME/.ssh/id_rsa to exist with appropriate permissions; you don't need an ssh agent running.
#!/bin/sh
if ! [ -d "$HOME/.ssh" ]; then
mkdir "$HOME/.ssh"
fi
chmod 0700 "$HOME/.ssh"
if [ -n "$MANAGED_HOST" ]; then
ssh-keyscan "$MANAGED_HOST" >> "$HOME/.ssh/known_hosts"
fi
if [ -f /run/secrets/id_rsa ]; then
cp /run/secrets/id_rsa "$HOME/.ssh/id_rsa"
chmod 0400 "$HOME/.ssh/id_rsa"
fi
# exec "$#"
A typical pattern is to use the Dockerfile ENTRYPOINT to do first-time setup tasks like this. That will get passed the CMD as arguments, and the commented exec "$#" line at the end of the file runs that as a command. You'd set this up in your image's Dockerfile like:
FROM XXXXXX
...
# Script must be executable on the host, and must start with a
# #!/bin/sh "shebang" line
COPY init.sh /root
# MUST use JSON-array form
ENTRYPOINT ["/root/init.sh"]
# Can use any Dockerfile syntax
CMD ["python3", "/root/my_python.py"]
In your specific example, you're launching init.sh as a subprocess. The ssh-agent setup sets some environment variables, like $SSH_AUTH_SOCK, but when these run as a subprocess they don't get propagated back out to the host process. You can use the standard POSIX shell . builtin (the bash source builtin is equivalent, but non-standard) to cause those environment variables to be set in the context of the parent shell:
command: sh -c ". /root/init.sh && exec python3 /root/my_python.py"
The exec replaces the shell wrapper with the Python script, which you generally want. This will also wind up being the parent process of ssh-agent, which could potentially surprise your process if it happens to exit.
I need to set some environment variable for all users and processes inside docker container. It should be set at container start, not in Dockerfile, because it depends on running environment.
So the simple Dockerfile
FROM ubuntu
RUN echo 'export TEST=test' >> '/root/.bashrc'
works well for interactive sessions
docker run -ti test bash
then
env
and there is TEST=test
but when docker run -ti test env there is no TEST
I was trying
RUN echo 'export TEST=test' >> '/etc/environment'
RUN echo 'TEST="test"' >> '/etc/environment'
RUN echo 'export TEST=test' >> /etc/profile.d/1.sh
ENTRYPOINT export TEST=test
Nothing helps.
Why I need this. I have http_proxy variable inside container automatically set by docker, I need to set another variables, based on it, i.e. JAVA_OPT, do it system wide, for all users and processes, and in running environment, not at build time.
I would create a script which would be an entrypoint:
#!/bin/bash
# if env variable is not set, set it
if [ -z $VAR ];
then
# env variable is not set
export VAR=$(a command that gives the var value);
fi
# pass the arguments received by the entrypoint.sh
# to /bin/bash with command (-c) option
/bin/bash -c $#
And in Dockerfile I would set the entrypoint:
ENTRYPOINT entrypoint.sh
Now every time I run docker run -it <image> <any command> it uses my script as entrypoint so will always run it before the command then pass the arguments to the right place which is /bin/bash.
Improvements
The above script is enough to work if you are always using the entrypoint with arguments, otherwise your $# variable will be empty and will give you an error /bin/bash: -c: option requires an argument. A easy fix is an if statement:
if [ ! -z $# ];
then
/bin/bash -c $#;
fi
Setting the parameter in ENTRYPOINT would solve this issue.
In docker file pass parameter in ENTRYPOINT