I have a BusyBox based system, and another one with vanilla Ubuntu LTS.
I made a C++ program which takes main()'s argv[1] as a command name, to fork() and execl() that command in the child process. Right before, I did dup2() to redirect the child's standard output, similar to this, so the parent process can read() its output.
Then, the text read from the child is written to the console, with markers, to see that it was the parent who output this.
Now if I run this program with, as its arg, "dd --help", then two different things happen:
on the Ubuntu system, the output clearly comes from the parent process (and only)
on the BusyBox system, the parent process reads back nothing, and the output of dd writes directly to the console, apparently bypassing my (attempt at) redirection.
Since all the little commands on the BusyBox system are symlinks to the one BusyBox executable and I thought there could be a problem, I also tried making a child process out of "busybox dd --help".
That changed nothing, though.
But: if I do "busybox --help", all of the output is caught by the child process and nothing "spilled besides" it. (note I left out the "sub command" dd here, only --help for BusyBox itself)
What's the reason for this happening, and (how) can I get this to work as intended, on the BusyBox system, too?
Busybox is outputting its own output, e.g. when calling it with options like --help, on stdout. But its implemented commands, like "dd", are output on stderr - even if it's not errors, which I found rather unintuitive and hence didn't look down that alley at first.
Related
I'm new to Docker, and I'm not sure how to quite deal with this situation.
So I'm trying to run a docker container in order to replicate some results from a research paper, specifically from here: https://github.com/danhper/bigcode-tools/blob/master/doc/tutorial.md
(image link: https://hub.docker.com/r/tuvistavie/bigcode-tools/).
I'm using a windows machine, and every time I try to run the docker image (via: docker run -p 80:80 tuvistavie/bigcode-tools), it instantly closes. I've tried running other images, such as the getting-started, but that image doesn't close instantly.
I've looked at some other potential workarounds, like using -dit, but since the instructions require setting an alias/doskey for a docker run command, using the alias and chaining it with other commands multiple times results in creating a queue for the docker container since the port is tied to the alias.
Like in the instructions from the GitHub link, I'm trying to set an alias/doskey to make api calls to pull data, but I am unable to get any data nor am I getting any errors when performing the calls on the command prompt.
Sorry for the long question, and thank you for your time!
Going in order of the instructions:
0. I can run this, it added the image to my Docker Desktop
1.
Since I'm using a windows machine, I had to use 'set' instead of 'export'
I'm not exactly sure what the $ is meant for in UNIX, and whether or not it has significant meaning, but from my understanding, the whole purpose is to create a directory named 'bigcode-workspace'
Instead of 'alias,' I needed to use doskey.
Since -dit prevented my image from instantly closing, I added that in as well, but I'm not 100% sure what it means. Running docker run (...) resulted in the docker image instantly closing.
When it came to using the doskey alias + another command, I've tried:
(doskey macro) (another command)
(doskey macro) ^& (another command)
(doskey macro) $T (another command)
This also seemed to be using github api call, so I also added a --token=(github_token), but that didn't change anything either
Because the later steps require expected data pulled from here, I am unable to progress any further.
Looks like this image is designed to be used as a command-line utility. So it should not be running continuously, but you run it via alias docker-bigcode for your tasks.
$BIGCODE_WORKSPACE is an environment variable expansion here. So on a Windows machine it's %BIGCODE_WORKSPACE%. You might want to set this variable in Settings->System->About->Advanced System Settings, because variables set with SET command will apply to the current command prompt session only. Or you can specify the path directly, without environment variable.
As for alias then I would just create a batch file with the following content:
docker run -p 6006:6006 -v %BIGCODE_WORKSPACE%:/bigcode-tools/workspace tuvistavie/bigcode-tools %*
This will run the specified command appending the batch file parameters at the end. You might need to add double quotes if BIGCODE_WORKSPACE path contains spaces.
I'm trying to run a function in my lisp program. It is a bot that is connected to an IRC channel and with a special command you can query the bot to evaluate a simple lisp command. Because it is extremely dangerous to execute arbitrary code from people on the internet I want the actual evaluation to happen in a VM that is running a docker for every evaluation query the bot gets.
My function looks like this:
(defun run-command (cmd)
(uiop:run-program (list "docker" "run" "--rm" "-it" "my/docker" "sbcl" "--noinform" "--no-sysinit" "--no-userinit" "--noprint" "--disable-debugger" "--eval" (string-trim '(#\^M) (format nil "~A" cmd))) "--eval" "'(quit)'") :output '(:string :stripped t))
The idea behind this function is to start a docker that contains SBCL, runs the command via SBCL --eval and prints it to the docker std-out. And this printed string should be the result of run-command.
If I call
docker run --rm -it my/docker sbcl --noinform --no-sysinit --no-userinit --noprint --disable-debugger --eval "(print (+ 2 3))" --eval "(quit)"
on my command line I just get 5 as an result, what is exactly what I want.
But if I run the same command within lisp, with the uiop:run-program function I get
Subprocess #<UIOP/LAUNCH-PROGRAM::PROCESS-INFO {1004FC3923}>
with command ("docker" "run" "--rm" "-it"
"my/docker" "sbcl" "--noinform"
"--no-sysinit" "--no-userinit" "--noprint"
"--disable-debugger" "--eval" "(+ 2 3)")
as an error message, which means the process failed somehow. But I don't know what exactly could be wrong here. If I just execute for example "ls" I get the output, so the function seems to work properly.
Is there some special knowledge I need about uiop:run-program or am I doing something completely wrong?
Thanks in advance.
Edit: So it turns out that the -it flag caused issues. After removing the flag a new error emerges. Now the bot has not the permissions to execute docker. Is there a way to give it the permissions without granting it sudo rights?
There's, probably, something wrong with the way docker is invoked here (or SBCL). To get the error message, invoke uiop:run-program with :error-output :string argument, and then choose the continue restart to, actually, terminate execution and get the error output printed (if you're running from SLIME or some other REPL). If you call this in a non-interactive environment, you can wrap the call in a handler-bind:
(handler-bind ((uiop/run-program:subprocess-error
(lambda (e) (invoke-restart (find-restart 'continue)))))
(run-command ...))
It turned out the -it did cause trouble. After removing it and elevating the correct permissions to the bot everything worked out fine.
I am unable to run repo non-interactively inside a container as part of a freestyle job.
It prompts for the user-name and email. I got round that by doing a git config --global inside the job.
But then it does the color test, and that hangs indefinitely.
Looking at the source code for repo I see this
if os.isatty(0) and os.isatty(1) and not self.manifest.IsMirror:
if opt.config_name or self._ShouldConfigureUser():
self._ConfigureUser()
self._ConfigureColor()
So, I ran the following inside the container:
python -C "import os; print os.isatty(0), os.isatty(1)"
and, sure enough, it printed out True True
Looking at the Jenkins log, it launches the container with --tty specified, and there seems no way to configure that option.
I can't find a bash option to force a script to be run in a non-interactive shell. If I put the above python line in a file and execute it with almost any combination of commands and options, it still prints out True True
The only way I see something different is if I use I/O redirection
bash <a.sh
which prints out False True - i.e. stdin is not a tty, and
bash <a.sh >a.log
which prints False False.
For a complex script, are there any problems using the bash <script approach?
Does anyone know any jenkins magic to prevent docker being launched using --tty?
I know that the --tty is the culprit. I built the container locally and ran the following
$ docker run repotest python -c "import os;print os.isatty(0), os.isatty(1)"
False False
$ docker run --tty repotest python -c "import os;print os.isatty(0), os.isatty(1)"
True True
Running Versions:
repo: 1.12.37 (per Ubuntu 16.04 apt-get)
Jenkins: 2.149
Cloudbees Docker Plugin: 1.7.3
Container base is ubuntu:xenial
I'm using the "Build inside a docker container" option.
To run bash script repo_script.sh "non-interactively", or more exactly speaking without having terminals associated with standard streams, you could run your script simply as
repo_script.sh < /dev/null 2>&1 | cat
assuming you want to see the output the way you would see it running simply as repo_script.sh. By piping the standard output and error to a different process the file descriptor appears as a pipe and not TTY to repo_script.sh. You could also direct output to a file, or even to /dev/null if you do not care about the output:
log_file=/dev/null
repo_script.sh < /dev/null > "${log_file}" 2>&1
Running the script as
bash < repo_script.sh | cat
might would work too, though it is very unorthodox and to my mind hackish way of running a script just to break the association of TTY to the standard input. From script engine point of view, it is different to read a script program from a file than from standard input (which typically, if it is a terminal, is not seekable), so there might be some subtle differences that could possibly bite you in unexpected ways. This way does not as clearly communicate your intention to the next person that need to understand your code, and may lead to partial hair loss in that person due to extraneous head scratching.
There is no need for any bash options, just using the output directions from within the interpreting shell as above described is an easy-to-comprehend, multi-platform compatible standard convention for changing the standard stream associations.
P.S. I think it should be enough for your repo script to just test if the standard input is a TTY. It looks to me like the author of that script did not think deeply enough there. There is simply no use waiting for input if you do not have terminal device associated with standard input, and you could determine that everything needs to run without user interaction from there or stop with an error if that is not possible.
**NOTE: I've added updates in order, just keep reading, thanks. :) **
I've been very curious about this -- please see this screenshot of me running:
ls -lah build, and
yarn run assets, which runs ls -lah build.
Let me start by saying that this is a WIP build in webpack, so no need to tell me that a 31M bundle is less than optimal. :)
But why do I get the colors and the more detailed font with the native command and not when yarn executes the command? It may be relevant: this screen shot is:
- Windows 10
- Webstorm terminal
- logged in to a docker container running Ubuntu 14.4
Thanks! :)
** UPDATE: --color=always restores color **
As #Charles Duffy suggested, adding --color=always in the yarn script preserved the formatting:
If anyone has some specialized knowledge to share about what's going on here, I'm in the market to hear it! Thanks!
Short(ish) answer: What's actually going on?
The below answer assumes the GNU implementation of ls.
There are a few possibilities at play:
Your interactive terminal's options may be modified by a shell alias. Output from type ls will indicate whether this is true.
You may have ls --color=auto enabled, either via an alias or via an equivalent environment variable; regardless, this checks whether it's writing directly to a TTY, and only enables color if so.
If output is not direct to a TTY (for instance, if output is being captured by yarn before it's printed), ls --color=auto will not colorize.
To fix this, you can explicitly pass ls --color=always, or its equivalent, simply ls --color. This covers both cases: If you had an alias in use passing --color=auto on your behalf, passing it explicitly means you no longer need the alias. By contrast, if yarn is capturing content rather than passing it straight to the TTY, then --color=always tells ls to ignore isatty() returning false and colorize anyhow.
Background on what the above means:
A "TTY" is, effectively, a terminal. It provides bells and whistles (literally, for the bells) specialized for providing a device that a user is actually typing at. This means it has control sequences for inspecting and modifying cursor location, and -- pertinently for our purposes -- for changing the color with which output is rendered.
A "FIFO" is a pipe -- it moves characters from point A to point B, first-in, first-out. In the case of prog-one | prog-two, the thing that connects those two is a FIFO. It just moves characters, and has no concept of cursor location or colorization or anything else.
If ls tried to put color sequences in its output when that output is intended for any destination other than a terminal, those sequences wouldn't make any sense -- indeed, the very format in which colorization markers need to be printed is determined by the TERM variable specifying the currently active terminal type.
If you run ls --color, then, you're promising ls that its output really will be rendered by a terminal, or (at least) otherwise something that understands the color sequences appropriate to the currently configured TERM.
I would like to start two different services in my Docker container and exit the container as soon as one of them exits. I looked at supervisor, but I can't find how to get it to quit as soon as one of the managed applications exits. It tries to restart them up to three times, as is the standard setting and then just sits there doing nothing. Is supervisor able to do this or is there any other tool for this? A bonus would be if there also was a way to let both managed programs write to stdout, tagged with their application name, e.g.:
[Program 1] Some output
[Program 2] Some other output
[Program 1] Output again
Since you asked if there was another tool... we designed and wrote a powerful replacement for supervisord that is designed specifically for Docker. It automatically terminates when all applications quit, as well as has special service settings to control this behavior, plus will redirect stdout with tagged syslog-compatible output lines as well. It's open source, and being used in production.
Here is a quick start for Docker: http://garywiz.github.io/chaperone/guide/chap-docker-simple.html
There is also a complete set of tested base-images which are a good example at: https://github.com/garywiz/chaperone-docker, but these might be overkill and the earlier quickstart may do the trick.
I found solutions to both of my requirements by reading through the docs some more.
Exit supervisord on application exit
This can be achieved by using a custom eventlistener. I had to add the following segment into my supervisord configuration file:
[eventlistener:shutdownevent]
command=/shutdownhandler.sh
events=PROCESS_STATE_EXITED
supervisord will start the referenced script and upon the given event being triggered (PROCESS_STATE_EXITED is triggered after the exit of one of the managed programs and it not restarting automatically) will send a line containing data about the event on the scripts stdin.
The referenced shutdownhandler-script contains:
#!/bin/bash
while :
do
echo -en "READY\n"
read line
kill $(cat /supervisord.pid)
echo -en "RESULT 2\nOK"
done
The script has to indicate being ready by sending "READY\n" on its stdout, after which it may receive an event data line on its stdin. For my use case upon receival of a line (meaning one of the managed programs has exited), a SIGTERM is sent to the supervisord process being found by the pid it leaves in its pid file (situated in the root directory by default). For technical completeness, I also included a positive answer for the eventlistener, though that one should never matter.
Tagged output on stdout
I did this by simply starting a tail process in the background before starting supervisord, tailing the programs output log and piping the lines through ts (from the moreutils package) to prepend a tag to it. This way it shows up via docker logs with an easy way to see which program actually wrote the line.
tail -fn0 /var/log/supervisor/program1.log | ts '[Program 1]' &