Why is my terminal output not identical when running a yarn script vs its bash equivalent? - docker

**NOTE: I've added updates in order, just keep reading, thanks. :) **
I've been very curious about this -- please see this screenshot of me running:
ls -lah build, and
yarn run assets, which runs ls -lah build.
Let me start by saying that this is a WIP build in webpack, so no need to tell me that a 31M bundle is less than optimal. :)
But why do I get the colors and the more detailed font with the native command and not when yarn executes the command? It may be relevant: this screen shot is:
- Windows 10
- Webstorm terminal
- logged in to a docker container running Ubuntu 14.4
Thanks! :)
** UPDATE: --color=always restores color **
As #Charles Duffy suggested, adding --color=always in the yarn script preserved the formatting:
If anyone has some specialized knowledge to share about what's going on here, I'm in the market to hear it! Thanks!

Short(ish) answer: What's actually going on?
The below answer assumes the GNU implementation of ls.
There are a few possibilities at play:
Your interactive terminal's options may be modified by a shell alias. Output from type ls will indicate whether this is true.
You may have ls --color=auto enabled, either via an alias or via an equivalent environment variable; regardless, this checks whether it's writing directly to a TTY, and only enables color if so.
If output is not direct to a TTY (for instance, if output is being captured by yarn before it's printed), ls --color=auto will not colorize.
To fix this, you can explicitly pass ls --color=always, or its equivalent, simply ls --color. This covers both cases: If you had an alias in use passing --color=auto on your behalf, passing it explicitly means you no longer need the alias. By contrast, if yarn is capturing content rather than passing it straight to the TTY, then --color=always tells ls to ignore isatty() returning false and colorize anyhow.
Background on what the above means:
A "TTY" is, effectively, a terminal. It provides bells and whistles (literally, for the bells) specialized for providing a device that a user is actually typing at. This means it has control sequences for inspecting and modifying cursor location, and -- pertinently for our purposes -- for changing the color with which output is rendered.
A "FIFO" is a pipe -- it moves characters from point A to point B, first-in, first-out. In the case of prog-one | prog-two, the thing that connects those two is a FIFO. It just moves characters, and has no concept of cursor location or colorization or anything else.
If ls tried to put color sequences in its output when that output is intended for any destination other than a terminal, those sequences wouldn't make any sense -- indeed, the very format in which colorization markers need to be printed is determined by the TERM variable specifying the currently active terminal type.
If you run ls --color, then, you're promising ls that its output really will be rendered by a terminal, or (at least) otherwise something that understands the color sequences appropriate to the currently configured TERM.

Related

My docker container keeps instantly closing when trying to run an image for bigcode-tools

I'm new to Docker, and I'm not sure how to quite deal with this situation.
So I'm trying to run a docker container in order to replicate some results from a research paper, specifically from here: https://github.com/danhper/bigcode-tools/blob/master/doc/tutorial.md
(image link: https://hub.docker.com/r/tuvistavie/bigcode-tools/).
I'm using a windows machine, and every time I try to run the docker image (via: docker run -p 80:80 tuvistavie/bigcode-tools), it instantly closes. I've tried running other images, such as the getting-started, but that image doesn't close instantly.
I've looked at some other potential workarounds, like using -dit, but since the instructions require setting an alias/doskey for a docker run command, using the alias and chaining it with other commands multiple times results in creating a queue for the docker container since the port is tied to the alias.
Like in the instructions from the GitHub link, I'm trying to set an alias/doskey to make api calls to pull data, but I am unable to get any data nor am I getting any errors when performing the calls on the command prompt.
Sorry for the long question, and thank you for your time!
Going in order of the instructions:
0. I can run this, it added the image to my Docker Desktop
1.
Since I'm using a windows machine, I had to use 'set' instead of 'export'
I'm not exactly sure what the $ is meant for in UNIX, and whether or not it has significant meaning, but from my understanding, the whole purpose is to create a directory named 'bigcode-workspace'
Instead of 'alias,' I needed to use doskey.
Since -dit prevented my image from instantly closing, I added that in as well, but I'm not 100% sure what it means. Running docker run (...) resulted in the docker image instantly closing.
When it came to using the doskey alias + another command, I've tried:
(doskey macro) (another command)
(doskey macro) ^& (another command)
(doskey macro) $T (another command)
This also seemed to be using github api call, so I also added a --token=(github_token), but that didn't change anything either
Because the later steps require expected data pulled from here, I am unable to progress any further.
Looks like this image is designed to be used as a command-line utility. So it should not be running continuously, but you run it via alias docker-bigcode for your tasks.
$BIGCODE_WORKSPACE is an environment variable expansion here. So on a Windows machine it's %BIGCODE_WORKSPACE%. You might want to set this variable in Settings->System->About->Advanced System Settings, because variables set with SET command will apply to the current command prompt session only. Or you can specify the path directly, without environment variable.
As for alias then I would just create a batch file with the following content:
docker run -p 6006:6006 -v %BIGCODE_WORKSPACE%:/bigcode-tools/workspace tuvistavie/bigcode-tools %*
This will run the specified command appending the batch file parameters at the end. You might need to add double quotes if BIGCODE_WORKSPACE path contains spaces.

How does BusyBox evade my redirection of stdout, and can I work around it?

I have a BusyBox based system, and another one with vanilla Ubuntu LTS.
I made a C++ program which takes main()'s argv[1] as a command name, to fork() and execl() that command in the child process. Right before, I did dup2() to redirect the child's standard output, similar to this, so the parent process can read() its output.
Then, the text read from the child is written to the console, with markers, to see that it was the parent who output this.
Now if I run this program with, as its arg, "dd --help", then two different things happen:
on the Ubuntu system, the output clearly comes from the parent process (and only)
on the BusyBox system, the parent process reads back nothing, and the output of dd writes directly to the console, apparently bypassing my (attempt at) redirection.
Since all the little commands on the BusyBox system are symlinks to the one BusyBox executable and I thought there could be a problem, I also tried making a child process out of "busybox dd --help".
That changed nothing, though.
But: if I do "busybox --help", all of the output is caught by the child process and nothing "spilled besides" it. (note I left out the "sub command" dd here, only --help for BusyBox itself)
What's the reason for this happening, and (how) can I get this to work as intended, on the BusyBox system, too?
Busybox is outputting its own output, e.g. when calling it with options like --help, on stdout. But its implemented commands, like "dd", are output on stderr - even if it's not errors, which I found rather unintuitive and hence didn't look down that alley at first.

How to run repo from a script inside a container in a jenkins job

I am unable to run repo non-interactively inside a container as part of a freestyle job.
It prompts for the user-name and email. I got round that by doing a git config --global inside the job.
But then it does the color test, and that hangs indefinitely.
Looking at the source code for repo I see this
if os.isatty(0) and os.isatty(1) and not self.manifest.IsMirror:
if opt.config_name or self._ShouldConfigureUser():
self._ConfigureUser()
self._ConfigureColor()
So, I ran the following inside the container:
python -C "import os; print os.isatty(0), os.isatty(1)"
and, sure enough, it printed out True True
Looking at the Jenkins log, it launches the container with --tty specified, and there seems no way to configure that option.
I can't find a bash option to force a script to be run in a non-interactive shell. If I put the above python line in a file and execute it with almost any combination of commands and options, it still prints out True True
The only way I see something different is if I use I/O redirection
bash <a.sh
which prints out False True - i.e. stdin is not a tty, and
bash <a.sh >a.log
which prints False False.
For a complex script, are there any problems using the bash <script approach?
Does anyone know any jenkins magic to prevent docker being launched using --tty?
I know that the --tty is the culprit. I built the container locally and ran the following
$ docker run repotest python -c "import os;print os.isatty(0), os.isatty(1)"
False False
$ docker run --tty repotest python -c "import os;print os.isatty(0), os.isatty(1)"
True True
Running Versions:
repo: 1.12.37 (per Ubuntu 16.04 apt-get)
Jenkins: 2.149
Cloudbees Docker Plugin: 1.7.3
Container base is ubuntu:xenial
I'm using the "Build inside a docker container" option.
To run bash script repo_script.sh "non-interactively", or more exactly speaking without having terminals associated with standard streams, you could run your script simply as
repo_script.sh < /dev/null 2>&1 | cat
assuming you want to see the output the way you would see it running simply as repo_script.sh. By piping the standard output and error to a different process the file descriptor appears as a pipe and not TTY to repo_script.sh. You could also direct output to a file, or even to /dev/null if you do not care about the output:
log_file=/dev/null
repo_script.sh < /dev/null > "${log_file}" 2>&1
Running the script as
bash < repo_script.sh | cat
might would work too, though it is very unorthodox and to my mind hackish way of running a script just to break the association of TTY to the standard input. From script engine point of view, it is different to read a script program from a file than from standard input (which typically, if it is a terminal, is not seekable), so there might be some subtle differences that could possibly bite you in unexpected ways. This way does not as clearly communicate your intention to the next person that need to understand your code, and may lead to partial hair loss in that person due to extraneous head scratching.
There is no need for any bash options, just using the output directions from within the interpreting shell as above described is an easy-to-comprehend, multi-platform compatible standard convention for changing the standard stream associations.
P.S. I think it should be enough for your repo script to just test if the standard input is a TTY. It looks to me like the author of that script did not think deeply enough there. There is simply no use waiting for input if you do not have terminal device associated with standard input, and you could determine that everything needs to run without user interaction from there or stop with an error if that is not possible.

Remove a certain amount of files with a sequence of characters in a directory

I stopped a process to troubleshoot something. Now, I would like to start the process where it left off in CentOS 6.4.
This script I stopped runs a perl script in a loop to process all of the files in /dev/shm/split/. These files were split into many parts from a larger file. An example of how they are named are as follows:
file.txt.aa
file.txt.ab
file.txt.ac
...and so on.
I have identified that the script left off at file.txt.fy. So, I would like to remove all of the files in /dev/shm/split/ that are from file.txt.aa through file.txt.fy.
I tried to create a whitelist for the rm command by doing:
ls /dev/shm/split/ > whitelist
cat whitelist | egrep -v 'file.txt.[aa-fz]' | tee whitelist.tmp
This did not do what I had intended.
Please help me! Thank you!
The problem with your command is that you cannot match two characters with the square bracket pattern in bash. You should use something like that instead:
ls file.txt.[a-e]? file.txt.f[a-y]
Basically decompose your range into two ranges, the first will match .aa to .ez, and the second .fa to .fy (included).
Note that I have used the ls command here. I always find it a good idea to first echo or ls the commands/files you're going to run when the operations you do are potentially destructive. When you're sure it produces the right output, go on and use rm instead of ls.

grep command that works on Ubuntu, but not on Fedora

I clean hacked Wordpress installations for clients on a regular basis, and I have a set of scripts that I have written to help me track down where the sites are hacked. One code snippet that I use on a regular basis worked fine on Ubuntu, but since switching to Fedora on Friday has quite behaving as expected. The command is this:
grep -Iri --exclude "*.js" "eval\s*(" * | grep -rivf ~/safeevals.txt >../foundevals.txt;
What it is supposed to happen (and did happen when I was using Ubuntu): grep through all non-binary files, excluding Javascript includes, for all occurances of the eval() function, then perform a negative match on a line by line basis against all known occurances of the eval() function in a vanilla installation of Wordpress (the patterns of which are in ~/safeevals.txt).
What is actually happening: The first part is working fine, as I ran it separately and it did find all instances of eval() in the installation. However, instead of greping through those results, after the pipe is it re-grepping through all of the files, returning a negative match of ~/safeevals.txt (ie. pretty much every line of every file in the installation).
Any idea why the second grep isn't acting on the piped data, or what I need to do to fix it? Thanks.
-Michael
Just tested on my Debian box: apparently, grep -r likes to assume a default argument of .. I am really wondering if that behaviour is valid. Anyway, I guess dropping the -r option from the second grep command will fix it.
Edit: rgrep defaulting to $PWD seems to be a recent change in grep, see this discussion on unix.stackexchange and the link there to the commit in the upstream grep code repository.

Resources