Rewrite BashOperator to DockerOperator to run Haskell docker container at Airflow.
Hello, I need some help from the airflow community.
I've tried to use DockerOperator to start the Haskell container and always get an error
ERROR - Failed to execute job 345 for task docker_command_sleep (Docker container failed: {'Error': None, 'StatusCode': 1} lines elp-exe: results/adjacency_output/2022-09-30_2022-08-02/output.csv: openFile: does not exist (No such file or directory); 3755141)
DockerOperator(
task_id='docker_elp',
image='elp',
auto_remove=True,
docker_url='unix://var/run/docker.sock',
api_version='auto',
command=['/bin/bash -c "cd /opt/ELP_unweighted && stack --resolver lts-18.28 -- exec elp-exe -- results/adjacency_output/2022-09-30_2022-09-01/output.csv -- results/elp_output/2022-09-30_2022-09-01/output.txt"'],
network_mode="bridge",
mounts=[Mount(target='/opt/ELP_unweighted/results', source='/home/german/airflow/dags/predict/results', type='bind')],
working_dir="/opt/ELP_unweighted"
)
But it is working with BashOperator perfectly fine
BashOperator(
task_id='bash_elp',
bash_command='docker run -it --rm -v /home/german/airflow/dags/predict/results:/opt/ELP_unweighted/results elp /bin/bash -c "cd /opt/ELP_unweighted && stack --resolver lts-18.28 -- exec elp-exe -- results/adjacency_output/2022-09-30_2022-08-01/output.csv -- results/elp_output/2022-09-30_2022-08-01/output1.txt"'
)
How to rewrite bashoperator to dockeroperator?
Related
I use jenkins to build and run my containers.
After the run via jenkins the container change status to Exited (0)
I think there is somethings wrong in my shell that run the application inside the contaier but i cannot find where the problem is.
This is the command I use to run the container:
sh "podman run -d --name igfsbase -p 36360:36360 -p 46460:46460 --restart=always igfsbase:5.5.0.17"
This is the command that I execute in my Dockerfile at the end of the image creation.
ENTRYPOINT ["./startdockerigfsbase.sh"]
CMD [""]
And this is the very simple shell startdockerigfsbase.sh where I just run two JVM:
#!/bin/bash
echo " start IGFS_ONLINE.." &
java -DAPPL=IGFS_ONLINE -Djava.rmi.server.hostname=127.0.0.1 -Djava.rmi.server.useLocalHostname=true -Xms256M -Xmx256M -Xss256K -XX:+UseParallelGC -jar lib/jtmsStarter.jar ./cfg/online/jtms.properties &
sleep 5 &
echo " start IGFS_BATCH.." &
java -DAPPL=IGFS_BATCH -Djava.rmi.server.hostname=127.0.0.1 -Djava.rmi.server.useLocalHostname=true -Xms256M -Xmx256M -Xss256K -XX:+UseParallelGC -jar lib/jtmsStarter.jar ./cfg/batch/batch.properties &
sleep 5 &
echo "Task Completed" &
tail -f /dev/null &
echo "End process"
This is the container logs:
2022-08-09T13:43:43.302395979+02:00 stdout F start IGFS_ONLINE..
2022-08-09T13:43:43.304333937+02:00 stdout F start IGFS_BATCH..
2022-08-09T13:43:43.307389201+02:00 stdout F Task Completed
2022-08-09T13:43:43.308149080+02:00 stdout F End process
After that, the container status is in: Exited (0)
I really don't know where the problem is.
Could you please indicate me if there is somethings wrong or to change within the Shell script?
Thanks in advance
I'm using containerized Vespa.ai DB, and I want to execute the following commands from the host:
vespa-stop-services
vespa-remove-index
vespa-start-services
If I execute the following vespa-stop-services && vespa-remove-index && vespa-start-services from my shell after I attach the container, it works fine. But when I use docker exec it fails.
I tried the following commands:
docker exec bash -c 'vespa-stop-services && vespa-remove-index && vespa-start-services'
docker exec bash -l 'vespa-stop-services && vespa-remove-index && vespa-start-services'
The only way I successfully managed to execute those commands, is when I execute them sequentially, which I would like to avoid:
docker exec bash -l 'vespa-stop-services'
docker exec bash -l 'vespa-remove-index'
docker exec bash -l 'vespa-start-services'
What am I doing wrong?
Thanks in advance!
You need to specify the location of these commands when running from the parent host system
The following works/should work :
docker exec vespa bash -c "/opt/vespa/bin/vespa-stop-services && /opt/vespa/bin/vespa-remove-index -force && /opt/vespa/bin/vespa-start-services"
Notice the -force, which will not ask for confirmation before deleting the data, also note that indexes is not the only persistent data, configuration state is still retained.
Example run of a docker container called 'vespa':
docker exec vespa bash -c "/opt/vespa/bin/vespa-stop-services && /opt/vespa/bin/vespa-remove-index -force && /opt/vespa/bin/vespa-start-services"
Executing /opt/vespa/libexec/vespa/stop-vespa-base.sh
config-sentinel was running with pid 7788, sending SIGTERM
Waiting for exit (up to 15 minutes)
.. DONE
configproxy was running with pid 7666, sending SIGTERM
Waiting for exit (up to 15 minutes)
. DONE
[info] You have 23088 kilobytes of data for cluster msmarco
[info] For cluster msmarco distribution key 0 you have:
[info] 23084 kilobytes of data in var/db/vespa/search/cluster.msmarco/n0
[info] removing data: rm -rf var/db/vespa/search/cluster.msmarco/n0
[info] removed.
Running /opt/vespa/libexec/vespa/start-vespa-base.sh
Starting config proxy using tcp/localhost:19070 as config source(s)
runserver(configproxy) running with pid: 10553
Waiting for config proxy to start
config proxy started after 1s (runserver pid 10553)
runserver(config-sentinel) running with pid: 10679
I am doing MLperf, object_detection project test.
https://github.com/mlperf/training/tree/master/object_detection
Question 1:
When doing:
nvidia-docker run -v .:/workspace -t -i --rm --ipc=host mlperf/object_detection \
"cd mlperf/training/object_detection && ./install.sh"
It responses:
docker: Error response from daemon: create .: volume name is too short, names should be at least two alphanumeric characters.
I need to change -v .: to -v $(pwd):/workspace
Question 2:
When applied to the modification above, I got a new error:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"cd mlperf/training/object_detection && ./install.sh\": stat cd mlperf/training/object_detection && ./install.sh: no such file or directory": unknown.
It seems docker can't accept a string with space, ex: "cd xxxxxxx && ./install.sh"
If I modified the string to single command (./install.sh)
nvidia-docker run -v $(pwd):/workspace -t -i --rm --ipc=host mlperf/object_detection \
"./install.sh"
This will work, it doesn't look like an incorrect path problem, I tested to use an absoluted path it got the same error.
Question 3:
After followed the steps in the webpage, I always got an error:
ModuleNotFoundError: No module named 'maskrcnn_benchmark'
root#nvme:/markkang/mlperf/training/object_detection# nvidia-docker run -v $(pwd):/workspace -t -i --rm --ipc=host mlperf/object_detection "./run_and_time.sh"
/workspace/pytorch /workspace
Traceback (most recent call last):
File "tools/train_mlperf.py", line 8, in <module>
from maskrcnn_benchmark.utils.env import setup_environment # noqa F401 isort:skip
ModuleNotFoundError: No module named 'maskrcnn_benchmark'
Edit train_mlperf.py and insert the following path code before the invocation of maskrcnn_benchmark.utils.env, e.g.
import sys
sys.path.append('/workspace/pytorch/')
from maskrcnn_benchmark.utils.env import setup_environment # noqa F401 isort:skip
1) I am running a docker container with following cmd (passing few env variables with -e option)
$ docker run --name=xyz -d -e CONTAINER_NAME=xyz -e SSH_PORT=22 -e NWMODE=HOST -e XDG_RUNTIME_DIR=/run/user/0 --net=host -v /mnt:/mnt -v /dev:/dev -v /etc/sysconfig/network-scripts:/etc/sysconfig/network-scripts -v /:/hostroot/ -v /etc/hostname:/etc/host_hostname -v /etc/localtime:/etc/localtime -v /var/run/docker.sock:/var/run/docker.sock --privileged=true cf3681e04bfb
2) After running the container as above, i check the env variable NWMODE inside the container, and it shows correctly as shown below :
$ docker exec -it xyz bash
$ env | grep NWMODE
NWMODE=HOST
3) Now, i created a sample service 'b' shown below which executes a script b.sh (where i try to access NWMODE) :
root#ubuntu16:/etc/systemd/system# cat b.service
[Unit]
Description=testing service b
[Service]
ExecStart=/bin/bash /etc/systemd/system/b.sh
root#ubuntu16:/etc/systemd/system# cat b.sh
#!/bin/bash`
systemctl import-environment
echo "NWMODE:" $NWMODE`
4) Now if i start service 'b' and see its logs, it shows that it is not able to access NWMODE env variable
$ systemctl start b
$ journalctl -fu b
...
systemd[1]: Started testing service b.
bash[641]: NWMODE: //blank for $NWMODE here`
5) Now rather than having 'systemctl import-environment' in b.sh, if i do following then the b.service logs show the correct value of NWMODE env variable:
$ systemctl import-environment
$ systemctl start b
Though the step 5 above works i can't go for it, as all the services in my system will be started automatically by systemd. In that case, can anyone please let me know how can i access the environment variables (passed using 'docker run...' cmd above) in a service file (say for e.g. in b.sh above). Can this be achieved somehow with systemctl import-environment or there is some other way ?
systemd unsets all environment variables to provide a clean environment. Afaik that is intended to be a security feature.
Workaround: Create a file /etc/systemd/system.conf.d/myenvironment.conf:
[Manager]
DefaultEnvironment=CONTAINER_NAME=xyz NWMODE=HOST XDG_RUNTIME_DIR=/run/user/0
systemd will set the environment variables declared in this file.
You can set up an ENTRYPOINT script that automatically creates this file before running systemd. Example:
RUN echo '#! /bin/bash \n\
echo "[Manager] \n\
DefaultEnvironment=$(while read -r Line; do echo -n "$Line" ; done < <(env) \n\
" >/etc/systemd/system.conf.d/myenvironment.conf \n\
exec /lib/systemd/systemd \n\
' >/usr/local/bin/setmyenv && chmod +x /usr/bin/setmyenv
ENTRYPOINT /usr/bin/setmyenv
Instead of creating the script within Dockerfile you can store it outside and add it with COPY:
#! /bin/bash
echo "[Manager]
DefaultEnvironment=$(while read -r Line; do echo -n "$Line" ; done < <(env)
" >/etc/systemd/system.conf.d/myenvironment.conf
exec /lib/systemd/systemd
TL;DR
Run the the command using bash, first store the docker environment variables to a file (or just pipe them two awk), extract & export the variable and finally run your main script.
ExecStart=/bin/bash -c "cat /proc/1/environ | tr '\0' '\n' > /home/env_file; export MY_ENV_VARIABLE=$(awk -F= -v key="MY_ENV_VARIABLE" '$1==key {print $2}' /home/env_file); /usr/bin/python3 /usr/bin/my_python_script.py"
Whatever #mviereck is saying is true, still I have found another solution to this problem.
My use case is to pass an environment variable to my system-d container in the Docker run command (docker run -e MY_ENV_VARIABLE="some_val") and use that in the python script that is run through the system-d unit file.
According to this post (https://forums.docker.com/t/where-are-stored-the-environment-variables/65762) the container environment variables can be found in the running process /proc/1/environ inside the container. Performing a cat does show that the environment variable MY_ENV_VARIABLE=some_val does exist, though in some mangled form.
$ cat /proc/1/environ
HOSTNAME=271fbnd986bdMY_ENV_VARIABLE=some_valcontainer=dockerLC_ALL=CDEBIAN_FRONTEND=noninteractiveHOME=/rootroot#271fb0d986bd
The main task now would be to extract MY_ENV_VARIABLE="some_val" value and pass it to the ExecStart directive in the system-d unit file.
(extraction code referenced from How to grep for value in a key-value store from plain text)
# this outputs a nice key,value pair
$ cat /proc/1/environ | tr '\0' '\n'
HOSTNAME=861f23cd1b33
MY_ENV_VARIABLE=some_val
container=docker
LC_ALL=C
DEBIAN_FRONTEND=noninteractive
HOME=/root
# we can store this in a file for use, too
$ cat /proc/1/environ | tr '\0' '\n' > /home/env_var_file
# we can then reuse the file to extract the value of interest against a key
$ awk -F= -v key="MY_ENV_VARIABLE" '$1==key {print $2}' /home/env_file
some_val
Now in the ExecStart directive in the system-d unit file we can do this:
[Service]
Type=simple
ExecStart=/bin/bash -c "cat /proc/1/environ | tr '\0' '\n' > /home/env_file; export MY_ENV_VARIABLE=$(awk -F= -v key="MY_ENV_VARIABLE" '$1==key {print $2}' /home/env_file); /usr/bin/python3 /usr/bin/my_python_script.py"
When issuing grunt shell:test, I'm getting warning "the input device is not a TTY" & don't want to have to use -f:
$ grunt shell:test
Running "shell:test" (shell) task
the input device is not a TTY
Warning: Command failed: /bin/sh -c ./run.sh npm test
the input device is not a TTY
Use --force to continue.
Aborted due to warnings.
Here's the Gruntfile.js command:
shell: {
test: {
command: './run.sh npm test'
}
Here's run.sh:
#!/bin/sh
# should use the latest available image to validate, but not LATEST
if [ -f .env ]; then
RUN_ENV_FILE='--env-file .env'
fi
docker run $RUN_ENV_FILE -it --rm --user node -v "$PWD":/app -w /app yaktor/node:0.39.0 $#
Here's the relevant package.json scripts with command test:
"scripts": {
"test": "mocha --color=true -R spec test/*.test.js && npm run lint"
}
How can I get grunt to make docker happy with a TTY? Executing ./run.sh npm test outside of grunt works fine:
$ ./run.sh npm test
> yaktor#0.59.2-pre.0 test /app
> mocha --color=true -R spec test/*.test.js && npm run lint
[snip]
105 passing (3s)
> yaktor#0.59.2-pre.0 lint /app
> standard --verbose
Remove the -t from the docker run command:
docker run $RUN_ENV_FILE -i --rm --user node -v "$PWD":/app -w /app yaktor/node:0.39.0 $#
The -t tells docker to configure the tty, which won't work if you don't have a tty and try to attach to the container (default when you don't do a -d).
This solved an annoying issue for me. The script had these lines:
docker exec **-it** $( docker ps | grep mysql | cut -d' ' -f1) mysql --user= ..... > /var/tmp/temp.file
mutt -s "File is here" someone#somewhere.com < /var/tmp/temp.file
The script would run great if run directly and the mail would come with the correct output. However, when run from cron, (crontab -e) the mail would come with no content. Tried many things around permissions and shells and paths etc. However no joy!
Finally found this:
*/20 * * * * scriptblah.sh > $HOME/cron.log 2>&1
And on that cron.log file found this output:
the input device is not a TTY
Search led me here. And after I removed the -t, it's working great now!
docker exec **-i** $( docker ps | grep mysql | cut -d' ' -f1) mysql --user= ..... > /var/tmp/temp.file