I'm not sure that what's I'm looking for is possible or not... I'm a newbie in docker-compose world and I've read a lot of documentation and posts but I wasn't able to find out a solution.
I need to pass the stdout of a service defined in docker-compose to the stdin of another service. So the output of ServiceA will be the input of ServiceB.
Is it possible?
I see the function stdin_open but I cannot understand how to use the stdout of the other service as input.
Any suggestion?
Thanks
You can't do this in Docker easily.
Container processes' stdin and stdout aren't usually used for much. Most often the stdout receives log messages that can get reviewed later, and containers actually communicate through network sockets. (A container would typically run Apache but not grep.)
Docker doesn't have a native cross-container pipe, beyond the networking setup. If you're docker running containers from the shell, you can use an ordinary pipe there:
sudo sh -c 'docker run image-a | docker run image-b'
If it's practical to run both processes in the same container, you can use a shell pipe as the main container command:
docker run image sh -c 'process_a | process_b'
A differently hacky approach is to use a tool like Netcat to bridge between "stdin" and a network port. For example, consider a "server":
#!/bin/sh
# server.sh
# (Note, this uses busybox nc syntax)
nc -l -p 12345 \
| cat \ # <-- a process that reads from stdin
> out.txt
And a matching "client":
#!/bin/sh
# client.sh
cat in.txt \ # <-- a process that writes to stdout
| nc "$1" 12345
Build these into an image
FROM busybox
COPY client.sh server.sh /bin/
EXPOSE 12345
WORKDIR /data
CMD ["server.sh"]
Now run both containers:
docker network create testnet
docker build -t testimg .
echo hello world > in.txt
docker run -d -v $PWD:/data --net testnet --name server testimg \
server.sh
docker run -d -v $PWD:/data --net testnet --name client testimg \
client.sh server
docker wait client
docker wait server
cat out.txt
A more robust path would be to wrap the server process in a simple HTTP server that accepted an HTTP POST on some path and launched a subprocess to handle the request; then you'd have a single long-running server process instead of having to re-launch it for each request. The client would use a tool like curl or any other HTTP client.
Related
There are two containers A and B. Once container A starts, one process will be executed, then the container will stop. Container B is just an web application (say expressjs). Is it possible to kickstart A from container B ?
It is possible to grant a container access to docker so that it can spawn other containers on your host. You do this by exposing the docker socket inside the container, e.g:
docker run -v /var/run/docker.sock:/var/run/docker.sock --name containerB myimage ...
Now, if you have the docker client available inside the container, you will be able to control the docker daemon on your host and use that to spawn your "container A".
Before trying this approach, you should be aware of the security considerations: access to docker is the same as having root access on the host, which means if your web application has a remote compromise you have just handed the keys to your host to the attackers. This is described more fully in this article.
It is possible by mounting the docker socket.
Container A
It will print the time to the stdout (and its logs) and exit.
docker run --name contA ubuntu date
Container B
The trick is to mount the host's docker socket then install the docker client on the container. It will then interact with the daemon just as if you were using docker from the host. Once docker is installed, it simply restart container A every 5 seconds.
docker run --name contB -v /var/run/docker.sock:/var/run/docker.sock ubuntu bash -c "
apt-get update && apt-get install -y curl &&
curl -sSL https://get.docker.com/ | sh &&
watch --interval 5 docker restart contA"
You can see that contA is being called by looking at its logs
docker logs contA
That said, Docker is really meant for long running services. There's some talk over at the Docker github issues about specifying short lived "job" services for things like maintenance, cron jobs, etc, but nothing has been decided, much less coded. So it's best to build your system so that containers are up and stay up.
docker-compose.yml (credits to larsks)
# ...
volumes:
- /var/run/docker.sock:/var/run/docker.sock
# ...
Dockerfile (credits to Aaron V)
# ...
ENV DOCKERVERSION=19.03.12
RUN curl -fsSLO https://download.docker.com/linux/static/stable/x86_64/docker-${DOCKERVERSION}.tgz \
&& tar xzvf docker-${DOCKERVERSION}.tgz --strip 1 -C /usr/local/bin docker/docker \
&& rm docker-${DOCKERVERSION}.tgz
# ...
Node.js index.js (credits to Arpan Abhishek, Maulik Parmar and anishsane)
# ...
const { exec } = require("child_process");
# ...
exec('docker container ls -a --format "table {{.ID}}\t{{.Names}}" | grep <PART_OF_YOUR_CONTAINER_NAME> | cut -d" " -f1 | cut -f1 | xargs -I{} docker container restart -t 0 {}', (error, stdout, stderr) => {
if (error) {
console.log(`error: ${error.message}`);
return;
}
if (stderr) {
console.log(`stderr: ${stderr}`);
return;
}
console.log(`stdout: ${stdout}`);
});
# ...
Please make sure that your application is at least behind a password protection. Exposing docker.sock in any way is a security thing.
Here you can find other Docker client versions: https://download.docker.com/linux/static/stable/x86_64/
Please replace <PART_OF_YOUR_CONTAINER_NAME> with a part of your container name.
I have a small Spring Boot API running in docker. Shown below Is the command I used to up the container.
docker run -d --rm --name factorialorialContainer --memory=$2 --cpus=$3 -p 8080:8080 -e JAVA_OPTIONS="$(cat /Users/sulekahelmini/Documents/fyp/fyp_work/MLscripts/flags.txt)" suleka96/factorial:latest
Then I have a dockerized JMeter which I up using the below command
export volume_path=/Users/sulekahelmini/Documents/fyp/fyp_work/MLscripts/jmeter_resource && export jmeter_path=/jmeter && docker run --rm --name jmeterContainer --memory='512m' --cpus=2 -e JAVA_OPTS="-Xms512 -Xmx512" --volume ${volume_path}:${jmeter_path} egaillardon/jmeter --nongui -t factorial.jmx -l jmeter_results.jtl -q user.properties
but all the tests fail and requests are not getting sent to the API. This is how the CLI of JMeter looks
test config of request:
Protocol: htttp
Server: localhost
Port:8080
Method:GET
Path:/api/factorial
This is what the complete bash file looks like:
#!/bin/bash
cd /Users/sulekahelmini/Documents/fyp/fyp_work/demo/target && docker build . -t suleka96/factorial
docker run -d --rm --name factorialorialContainer --memory='512m' --cpus=2 -p 8080:8080 -e JAVA_OPTIONS="$(cat /Users/sulekahelmini/Documents/fyp/fyp_work/MLscripts/flags_base.txt)" suleka96/factorial:latest
sleep 15
#run test
export volume_path=/Users/sulekahelmini/Documents/fyp/fyp_work/MLscripts/jmeter_resource && export jmeter_path=/jmeter && docker run --rm --name jmeterContainer --memory='512m' --cpus=2 -e JAVA_OPTS="-Xms512 -Xmx512" --volume ${volume_path}:${jmeter_path} egaillardon/jmeter --nongui -t factorial.jmx -l jmeter_results.jtl -q user.properties
sleep 15
#jtl split
java -jar /Users/sulekahelmini/Documents/fyp/fyp_work/MLscripts/jtl-splitter-0.4.6-SNAPSHOT.jar -f /Users/sulekahelmini/Documents/fyp/fyp_work/MLscripts/jmeter_resource/jmeter_results.jtl -s -t 1;
docker stop factorialorialContainer
docker stop jmeterContainer
What am I doing wrong? How can I fix this?
You're doing wrong absolutely everything.
When it comes to Spring Boot even "small" API is not small at all, if you want something really small - consider i.e. Jersey
I fail to see why do you need containers at all, in your situation they don't add any value but only consume resources
You're running the application under test and the load generator at the same physical machine. Both can be very resource intensive when it comes to high load and you won't be able to tell for sure where is the bottleneck
If you still want to ignore previous 2 points and proceed: you're using localhost in JMeter container and there is nothing deployed on port 8080 in that container. You need to run the following command:
docker inspect factorialorialContainer
you will see a line which looks like:
"IPAddress": "xxx.xxx.xxx.xxx",
you will need to get this IP address from the docker inspect command output and replace the localhost with this IP address in the JMeter's HTTP Request sampler
References:
Docker Network
JMeter Distributed Testing with Docker
Say I run a docker container as a daemon:
docker run -d foo
is there a way to write to the stdin of that container? Something like:
docker exec -i foo echo 'hi'
last time I checked the -i and -d flags were mutually exclusive when used with the docker run command.
According to another answer on ServerFault, you can use socat to pipe input to a docker container like this:
echo 'hi' | socat EXEC:"docker attach container0",pty STDIN
Note that the echo command includes a newline at the end of the output, so the line above actually sends hi\n. Use echo -n if you don't want a newline.
Let's see how this looks like with the example script from David's answer:
# Create a new empty directory
mkdir x
# Run a container, in the background, that copies its stdin
# to a file in that directory
docker run -itd --rm -v $PWD/x:/x --name cattainer busybox sh -c 'cat >/x/y'
# Send some strings in
echo 'hi' | socat EXEC:"docker attach cattainer",pty STDIN
echo 'still there?' | socat EXEC:"docker attach cattainer",pty STDIN
# Stop container (cleans up itself because of --rm)
docker stop cattainer
# See what we got out
cat x/y
# should output:
# hi
# still there?
You could also wrap it in a shell function:
docker_send() {
container="$1"; shift
echo "$#" | socat EXEC:"docker attach $container",pty STDIN
}
docker_send cattainer "Hello cat!"
docker_send cattainer -n "No newline here:" # flag -n is passed to echo
Trivia: I'm actually using this approach to control a Terraria server running in a docker container, because TerrariaServer.exe only accepts server commands (like save or exit) on stdin.
In principle you can docker attach to it. CTRL+C will stop the container (by sending SIGINT to the process); CTRL+P, CTRL+Q will detach from it and leave it running (if you started the container with docker run -it).
The one trick here is that docker attach expects to be running in a terminal of some sort; you can do something like run it under script to meet this requirement. Here's an example:
# Create a new empty directory
mkdir x
# Run a container, in the background, that copies its stdin
# to a file in that directory
docker run -itd -v $PWD/x:/x --name cat busybox sh -c 'cat >/x/y'
# Send a string in
echo foo | script -q /dev/null docker attach cat
# Note, EOF here stops the container, probably because /bin/cat
# exits normally
# Clean up
docker rm cat
# See what we got out
cat x/y
In practice, if the main way a program communicates is via text on its standard input and standard output, Docker isn't a great packaging mechanism for it. In higher-level environments like Docker Compose or Kubernetes, it becomes progressively harder to send content this way, and there's frequently an assumption that a container can run completely autonomously. Just invoking the program gets complicated quickly (as this question hints at). If you have something like, say, the create-react-app setup tool that asks a bunch of interactive questions then writes things to the host filesystem, it will be vastly easier to run it directly on the host and not in Docker.
I have an application that runs dockerized commands via docker run --rm .... Is it possible to dockerize this application?
E.g. I want the app RUNNER to run app RUNNABLE and read the stdout result.
(I need multiple instances of RUNNABLE in async calling fashion but that's RUNNER application business)
I know it is possible to just export root access to docker socket to the RUNNER application but this doesn't feel right. Especially with no-root-running rule for *nix applications.
Is there any other method to communicate containers rather than exporting socket to the container? Am I doing the system design wrong?
Basically it's possible to let the container communicate over host based files. Take a look at the following example.
# create a test dir
mkdir -p docker_test/bin
cd docker_test
# an endless running script that writes output in a file
vim bin/printDate.sh
chmod 700 bin/*.sh
# start docker container from debian image
# the container got a local host mount of /tmp to container /opt/output
# printDate.sh writes its output to container /opt/output/printDate.txt
docker run --name cont-1 \
-v /tmp:/opt/output -it -v `pwd`/bin:/opt/test \
debian /opt/test/printDate.sh
# start a second container in another terminal and mount /tmp again
docker run --name cont-2 -v /tmp:/opt/output -it \
debian tail -f /opt/output/printDate.txt
# the second container prints the output of program in cont-1
The endless script for container 1 output
#!/bin/bash
while true; do
sleep 1
date >> /opt/output/printDate.txt
done
There are two containers A and B. Once container A starts, one process will be executed, then the container will stop. Container B is just an web application (say expressjs). Is it possible to kickstart A from container B ?
It is possible to grant a container access to docker so that it can spawn other containers on your host. You do this by exposing the docker socket inside the container, e.g:
docker run -v /var/run/docker.sock:/var/run/docker.sock --name containerB myimage ...
Now, if you have the docker client available inside the container, you will be able to control the docker daemon on your host and use that to spawn your "container A".
Before trying this approach, you should be aware of the security considerations: access to docker is the same as having root access on the host, which means if your web application has a remote compromise you have just handed the keys to your host to the attackers. This is described more fully in this article.
It is possible by mounting the docker socket.
Container A
It will print the time to the stdout (and its logs) and exit.
docker run --name contA ubuntu date
Container B
The trick is to mount the host's docker socket then install the docker client on the container. It will then interact with the daemon just as if you were using docker from the host. Once docker is installed, it simply restart container A every 5 seconds.
docker run --name contB -v /var/run/docker.sock:/var/run/docker.sock ubuntu bash -c "
apt-get update && apt-get install -y curl &&
curl -sSL https://get.docker.com/ | sh &&
watch --interval 5 docker restart contA"
You can see that contA is being called by looking at its logs
docker logs contA
That said, Docker is really meant for long running services. There's some talk over at the Docker github issues about specifying short lived "job" services for things like maintenance, cron jobs, etc, but nothing has been decided, much less coded. So it's best to build your system so that containers are up and stay up.
docker-compose.yml (credits to larsks)
# ...
volumes:
- /var/run/docker.sock:/var/run/docker.sock
# ...
Dockerfile (credits to Aaron V)
# ...
ENV DOCKERVERSION=19.03.12
RUN curl -fsSLO https://download.docker.com/linux/static/stable/x86_64/docker-${DOCKERVERSION}.tgz \
&& tar xzvf docker-${DOCKERVERSION}.tgz --strip 1 -C /usr/local/bin docker/docker \
&& rm docker-${DOCKERVERSION}.tgz
# ...
Node.js index.js (credits to Arpan Abhishek, Maulik Parmar and anishsane)
# ...
const { exec } = require("child_process");
# ...
exec('docker container ls -a --format "table {{.ID}}\t{{.Names}}" | grep <PART_OF_YOUR_CONTAINER_NAME> | cut -d" " -f1 | cut -f1 | xargs -I{} docker container restart -t 0 {}', (error, stdout, stderr) => {
if (error) {
console.log(`error: ${error.message}`);
return;
}
if (stderr) {
console.log(`stderr: ${stderr}`);
return;
}
console.log(`stdout: ${stdout}`);
});
# ...
Please make sure that your application is at least behind a password protection. Exposing docker.sock in any way is a security thing.
Here you can find other Docker client versions: https://download.docker.com/linux/static/stable/x86_64/
Please replace <PART_OF_YOUR_CONTAINER_NAME> with a part of your container name.