I am using the GNU parallel tool. I have an input file in.txt the looks like this:
export MY_ENV=$1 && echo hi: $MY_ENV
export MY_ENV=$1 && echo hi: $MY_ENV
export MY_ENV=$1 && echo hi: $MY_ENV
export MY_ENV=$1 && echo hi: $MY_ENV
export MY_ENV=$1 && echo hi: $MY_ENV
export MY_ENV=$1 && echo hi: $MY_ENV
I use this command (case 1) to invoke parallel:
parallel -j 4 -a in.txt --link ::: 11 22 33 44
which (as expected) results in this output:
hi: 11
hi: 22
hi: 33
hi: 44
hi: 11
hi: 22
However when i try to send the input via STDIN using the command below (case 2) I get different behavior. In other words this command:
cat in.txt | parallel -j 4 --link ::: 11 22 33 44
results in this error message:
/bin/bash: 11: command not found
/bin/bash: 22: command not found
/bin/bash: 33: command not found
/bin/bash: 44: command not found
Shouldn't the behavior be identical? How can I invoke the parallel program so that when the input is via STDIN I get the same output as in case 1 above?
cat in.txt | parallel -j 4 -a - --link ::: 11 22 33 44
or
cat in.txt | parallel -j 4 --link :::: - ::: 11 22 33 44
or
cat in.txt | parallel -j 4 :::: - :::+ 11 22 33 44
See details on https://doi.org/10.5281/zenodo.1146014 (section 4.2).
Related
On mac i can do
sudo filebyproc.d 2>/dev/null | grep " cat "
running
cat balbalbla
i've got
sudo filebyproc.d 2>/dev/null | grep " cat "
Password:
8 178 open:entry cat /dev/dtracehelper
10 964 open_nocancel:entry cat /usr/share/locale/UTF-8/LC_CTYPE
2 178 open:entry cat balbalbla
which is the list of all files open by cat process.
How to do the same (or similar) in QNX?
Use pidin -p <process> fds. <process> may be specified as either a pid or the name of the binary (eg. 'cat').
Ref: http://www.qnx.com/developers/docs/7.1/#com.qnx.doc.neutrino.utilities/topic/p/pidin.html
File1 contains
hello
hello
I need to write a grep command to print the filename if this file contains more than one "hello". Otherwise, I need grep to exit on failure.
So far I have
grep -c "hello" File1 | grep -v :0
but it outputs
2. How do I get the desired output, which should either be filename File1 or no output at all (from what I understand, no match is a non zero exit code for grep)
with GNU grep for -z:
grep -lz 'hello.*hello' file
e.g.:
$ seq 15 | grep -lz '3.*3'
(standard input)
$ echo $?
0
$ seq 5 | grep -lz '3.*3'
$ echo $?
1
Like this:
#!/bin/bash
count=$(grep -c "hello" "$1")
if ((count > 1)); then
echo "$1"
else
exit 1
fi
Usage:
chmod +x script.sh
./script.sh File1
Explanations:
((...)) is an arithmetic command, which returns an exit status of 0 if the expression is nonzero, or 1 if the expression is zero. Also used as a synonym for "let", if side effects (assignments) are needed. See http://mywiki.wooledge.org/ArithmeticExpression
Using perl in a shell:
perl -0 -le '
my $filename = $ARGV[0];
print $filename if grep { /hello\nhello/ } <>
' file
I'm using Docker client Version: 18.09.2.
When I run start a container interactively and run a date command, then pipe its output to hexdump for inspection, I'm seeing a trailing \n as expected:
$ docker run --rm -i -t alpine
/ # date | hexdump -c
0000000 T h u M a r 7 0 0 : 1 5
0000010 : 0 6 U T C 2 0 1 9 \n
000001d
However, when I pass the date command as an entrypoint directly and run the container, I get a \r \n every time there's a new line in the output.
$ docker run --rm -i -t --entrypoint=date alpine | hexdump -c
0000000 T h u M a r 7 0 0 : 1 6
0000010 : 1 9 U T C 2 0 1 9 \r \n
000001e
This is weird.
It totally doesn't happen when I omit -t (not allocating any TTY):
docker run --rm -i --entrypoint=date alpine | hexdump -c
0000000 T h u M a r 7 0 0 : 1 7
0000010 : 3 0 U T C 2 0 1 9 \n
000001d
What's happening here?
This sounds dangerous, as I use docker run command in my scripts, and if I forget to omit -t from my scripts, the output I'll collect from docker run command will have invisible/non-printible \r characters which can cause all sorts of issues.
tldr; This is a tty default behaviour and unrelated to docker. Per the ticket filed on github about your exact issue.
Quoting the relevant comments in that ticket:
Looks like this is indeed TTY by default translates newlines to CRLF
$ docker run -t --rm debian sh -c "echo -n '\n'" | od -c
0000000 \r \n
0000002
disabling "translate newline to carriage return-newline" with stty -onlcr correctly gives;
$ docker run -t --rm debian sh -c "stty -onlcr && echo -n '\n'" | od -c
0000000 \n
0000001
Default TTY options seem to be set by the kernel ... On my linux host it contains:
/*
* Defaults on "first" open.
*/
#define TTYDEF_IFLAG (BRKINT | ISTRIP | ICRNL | IMAXBEL | IXON | IXANY)
#define TTYDEF_OFLAG (OPOST | ONLCR | XTABS)
#define TTYDEF_LFLAG (ECHO | ICANON | ISIG | IEXTEN | ECHOE|ECHOKE|ECHOCTL)
#define TTYDEF_CFLAG (CREAD | CS7 | PARENB | HUPCL)
#define TTYDEF_SPEED (B9600)
ONLCR is indeed there.
When we go looking at the ONLCR flag documentation, we can see that:
[-]onlcr: translate newline to carriage return-newline
To again quote the github ticket:
Moral of the story, don't use -t unless you want a TTY.
TTY line endings are CRLF, this is not Docker's doing.
I have a basic docker-compose file file for wurstmeister/kafka
I'm trying to configure it to use SASL_PLAIN with SSL
However I keep getting this error no matter how many ways I try to specify my jaas file
This is the error I get
[2018-04-11 10:34:34,545] FATAL [KafkaServer id=1001] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.IllegalArgumentException: Could not find a 'KafkaServer' or 'sasl_ssl.KafkaServer' entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set
These are the vars I have. Last one is where I specify my jaas file
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_HOST_NAME: 10.10.10.1
KAFKA_PORT: 9092
KAFKA_ADVERTISED_PORT: 9093
KAFKA_ADVERTISED_HOST_NAME: 10.10.10.1
KAFKA_LISTENERS: PLAINTEXT://:9092,SASL_SSL://:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://10.10.10.1:9092,SASL_SSL://10.10.10.1:9093
KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SASL_SSL
KAFKA_SASL_ENABLED_MECHANISMS: PLAIN
SASL_MECHANISM_INTER_BROKER_PROTOCOL: PLAIN
KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: PLAIN
KAFKA_SSL_TRUSTSTORE_LOCATION: /kafka.server.truststore.jks
KAFKA_SSL_TRUSTSTORE_PASSWORD: password
KAFKA_SSL_KEYSTORE_LOCATION: /kafka.server.keystore.jks
KAFKA_SSL_KEYSTORE_PASSWORD: password
KAFKA_SSL_KEY_PASSWORD: password
KAFKA_OPTS: '-Djava.security.auth.login.config=/path/kafka_server_jaas.conf'
Also when I try to check the docker logs I see
/usr/bin/start-kafka.sh: line 96: KAFKA_OPTS=-Djava.security.auth.login.config: bad substitution
Any help is greatly appreciated!
equals '=' inside the last value is causing this issue.
KAFKA_OPTS: '-Djava.security.auth.login.config=/path/kafka_server_jaas.conf'
This is what I have got after debugging.
+ for VAR in $(env)
+ [[ KAFKA_OPTS=-
Djava.security.auth.login.config=/path/kafka_server_jaas.conf =~ ^KAFKA_ ]]
+ [[ ! KAFKA_OPTS=-
Djava.security.auth.login.config=/path/kafka_server_jaas.conf =~
^KAFKA_HOME ]]
++ echo KAFKA_OPTS=-
Djava.security.auth.login.config=/path/kafka_server_jaas.conf
++ sed -r 's/KAFKA_(.*)=.*/\1/g'
++ tr '[:upper:]' '[:lower:]'
++ tr _ .
+ kafka_name=opts=-djava.security.auth.login.config
++ echo KAFKA_OPTS=-
Djava.security.auth.login.config=/path/kafka_server_jaas.conf
++ sed -r 's/(.*)=.*/\1/g'
+ env_var=KAFKA_OPTS=-Djava.security.auth.login.config
+ grep -E -q '(^|^#)opts=-djava.security.auth.login.config='
/opt/kafka/config/server.properties
start-kafka.sh: line 96: KAFKA_OPTS=-Djava.security.auth.login.config: bad
substitution
and this is the piece of code that is performing this operation.
88 for VAR in $(env)
89 do
90 if [[ $VAR =~ ^KAFKA_ && ! $VAR =~ ^KAFKA_HOME ]]; then
91 kafka_name=$(echo "$VAR" | sed -r 's/KAFKA_(.*)=.*/\1/g' | tr '[:upper:]' '[:lower:]' | tr _ .)
92 env_var=$(echo "$VAR" | sed -r 's/(.*)=.*/\1/g')
93 if grep -E -q '(^|^#)'"$kafka_name=" "$KAFKA_HOME/config/server.properties"; then
94 sed -r -i 's#(^|^#)('"$kafka_name"')=(.*)#\2='"${!env_var}"'#g' "$KAFKA_HOME/config/server.properties" #note that no config values may contain an '#' char
95 else
96 echo "$kafka_name=${!env_var}" >> "$KAFKA_HOME/config/server.properties"
97 fi
98 fi
99
100 if [[ $VAR =~ ^LOG4J_ ]]; then
101 log4j_name=$(echo "$VAR" | sed -r 's/(LOG4J_.*)=.*/\1/g' | tr '[:upper:]' '[:lower:]' | tr _ .)
102 log4j_env=$(echo "$VAR" | sed -r 's/(.*)=.*/\1/g')
103 if grep -E -q '(^|^#)'"$log4j_name=" "$KAFKA_HOME/config/log4j.properties"; then
104 sed -r -i 's#(^|^#)('"$log4j_name"')=(.*)#\2='"${!log4j_env}"'#g' "$KAFKA_HOME/config/log4j.properties" #note that no config values may contain an'#' char
105 else
106 echo "$log4j_name=${!log4j_env}" >> "$KAFKA_HOME/config/log4j.properties"
107 fi
108 fi
109 done
Update: They have fixed it and it is merged now!
https://github.com/wurstmeister/kafka-docker/pull/321
There's a bug open now with wurstmeister/kafka but they have gotten back to me with a workaround as follows
I believe his is part of a larger namespace collision problem that
affects multiple elements such as Kubernetes deployments etc (as well
as other KAFKA_ service settings).
Given you are referencing an external file /kafka_server_jaas.conf,
i'm assuming you're OK adding/mounting extra files through; a
work-around is to specify a CUSTOM_INIT_SCRIPT environment var, which
should be a script similar to:
#!/bin/bash
export KAFKA_OPTS="-Djava.security.auth.login.config=/kafka_server_jaas.conf"
This is executed after the substitution part that is failing.
This could have been done inline, however there is currently a bug in
how we process the environment, where we need to specify the input
separator to make this work correctly.
Hopefully this works!
I have a list of number in a file with format: {integer}\n . So a possible list is:
3
12
53
23
18
32
1
4
i want to use grep to get the count of a specific number, but grep -c "1" file results 3 because it takes into account except the 1, the 12, 18 also. How can i correct this?
Although all the answers until now are logical, and i thought of them and tested before, actually nothing works:
username#domain2:~/code/***/project/random/r2$ cat out.txt
2
16
11
1
13
2
1
16
16
9
username#domain2:~/code/***/project/random/r2$ grep -Pc "^1$" out.txt
0
username#domain2:~/code/***/project/random/r2$ grep -Pc ^1$ out.txt
0
username#domain2:~/code/***/project/random/r2$ grep -c ^1$ out.txt
0
username#domain2:~/code/***/project/random/r2$ grep -c "^1$" out.txt
0
username#domain2:~/code/***/project/random/r2$ grep -xc "^1$" out.txt
0
username#domain2:~/code/***/project/random/r2$ grep -xc "1" out.txt
0
Use the -x flag:
grep -xc 1 file
This is what it means:
-x, --line-regexp
Select only those matches that exactly match the whole line.
There a some other ways you can do this besides grep
$ cat file
3 1 2 100
12 x x x
53
23
18
32
1
4
$ awk '{for(i=1;i<=NF;i++) if ($i=="1") c++}END{print c}' file
2
$ ruby -0777 -ne 'puts $_.scan(/\b1\b/).size' file
2
$ grep -o '\b1\b' file | wc -l
2
$ tr " " "\n" < file | grep -c "\b1\b"
2
Use this regex...
\D1\D
...or ^1$ with multiline mode on.
Tested with RegExr and they both work.
Use e.g. ^123$ to match "Beginning of line, 123, End of line"
grep -wc '23' filename.txt
It will count the number of exact matches of digit 23.