How to grep text with brackets only - grep

the log looks like:
2019/10/04 03:03:12.938 [error]: some exception happened
2019/10/04 03:03:12.938 [error]: some exception happened
2019/10/04 03:53:35.595 [info]: there is a benign error in the code
grep "error" log.txt will give me all three lines but I only want the top two. How do I do that?


You may either grep all lines containing [error]:
grep '\[error]:' file
Or, you may use awk to check if the third field is equal to [error]:
awk '$3 == "[error]:"' file
See the online demo
#!/bin/bash
log="2019/10/04 03:03:12.938 [error]: some exception happened
2019/10/04 03:03:12.938 [error]: some exception happened
2019/10/04 03:53:35.595 [info]: there is a benign error in the code"
grep '\[error]:' <<< "$log"
# => 2019/10/04 03:03:12.938 [error]: some exception happened
# 2019/10/04 03:03:12.938 [error]: some exception happened
awk '$3 == "[error]:"' <<< "$log"
# => 2019/10/04 03:03:12.938 [error]: some exception happened
# 2019/10/04 03:03:12.938 [error]: some exception happened


Try using the following pattern:
^.*\[error\]:.*$
For example:
grep "^.*\[error\]:.*$"


Thanks for the answers above. However, the issue is that the log contains color code for [error] and [info]. In the console it shows as [error] while in text it is actually [^[[31merror^[[0m]
To turn off the color code
cat xxx.log | sed 's/\x1b\[[0-9;]*m//g'
To grep [error]:
cat xxx.log | sed 's/\x1b\[[0-9;]*m//g' | grep "\[error\]"

Related

PoET engine error when setting up Docker Sawtooth Network Environment

I'm trying to set up the Docker Sawtooth Network Environment using PoET Simulator (CFT) following these steps: https://sawtooth.hyperledger.org/docs/core/releases/latest/app_developers_guide/docker_test_network.html.
The intkey set transaction works properly in the PBFT network. I can get the key value using inktey show in all REST API containers and a new block is created.
But, with PoET, I don't have any response in log terminal and no block is created. What happens when i try to get key value in any node:
root#e9b57e11feb6:/# intkey set --url http://sawtooth-rest-api-default-0:8008 MyKey 999
{
"link": "http://sawtooth-rest-api-default-0:8008/batch_statuses?id=35f975022d853deddf0b7329ca8d10e608d3a3fa3e5f2318164e6de738c705e11aa335d709dfda12c50dc807346e8dace2e204cb14ad32699cb2a87b2a6e6f1b"
}
root#e9b57e11feb6:/# intkey show --url http://sawtooth-rest-api-default-1:8008 MyKey
Error: No such key: MyKey
When I started the network with docker-compose up, the following error message appeared:
sawtooth-poet-engine-0 | [2020-12-12 14:28:55.147 ERROR zmq_driver] Uncaught driver exception
sawtooth-poet-engine-0 | Traceback (most recent call last):
sawtooth-poet-engine-0 | File "/usr/lib/python3/dist-packages/sawtooth_sdk/consensus/zmq_driver.py", line 88, in _driver_loop
sawtooth-poet-engine-0 | result = self._process(message)
sawtooth-poet-engine-0 | File "/usr/lib/python3/dist-packages/sawtooth_sdk/consensus/zmq_driver.py", line 237, in _process
sawtooth-poet-engine-0 | &apos;Received unexpected message type: {}&apos;.format(type_tag))
sawtooth-poet-engine-0 | sawtooth_sdk.consensus.exceptions.ReceiveError: Received unexpected message type: 700
I found these messages in the end of /var/log/sawtooth/poet-engine-debug.log file in the sawtooth-poet-engine-0 container:
[14:28:37.840 [MainThread] engine DEBUG] Received message: CONSENSUS_NOTIFY_BLOCK_NEW
[14:28:37.840 [MainThread] engine INFO] Received Block(block_num: 1, block_id: a4299924b77cc32934ac6a470636312b24c9153327b5b7e2e878640f85c0442d5f3dca4bfc4dd4b6575ef047c63e01da3228cbac8d4b6d68c149b9d2589720b1, previous_id: e268a0b21a0d33b0e57a162deb41dd55af7a88fee69382d4bfa3f26f93be7afc485ea8e73f00764cf8e99cbe3efa3c0c42a4357183e227388b1cf51c33737e5b, signer_id: 02d69ef8bd879297899bac65fcde686c74fefeb7010a58db99a9eb24ed014f39db, payload: b&apos;{"SerializedCertificate": "{\\"block_hash\\": \\"b\&apos;\\\\\\\\xf6#>>V\\\\\\\\xd25%\\\\\\\\xd1{\\\\\\\\x87\\\\\\\\r\\\\\\\\xda\\\\\\\\x91t\\\\\\\\xc2\\\\\\\\x97\\\\\\\\x8c\\\\\\\\xf1\\\\\\\\x08C5\\\\\\\\x92\\\\\\\\x15\\\\\\\\x83^H}\\\\\\\\x02\\\\\\\\xb7\\\\\\\\x85T\&apos;\\", \\"duration\\": 11.998389297959184, \\"local_mean\\": 5.0, \\"nonce\\": \\"8e1661a9178b8faacf94b67cd8efcbb964ddce04cdd8a3c09e78532100726480\\", \\"previous_certificate_id\\": \\"0000000000000000\\", \\"request_time\\": 1607783305.7010527, \\"validator_address\\": \\"020dd2bd7c5992708b9f48c2fa72e78ddd61d5cd3608c21cb27957dc576887f614\\"}", "Signature": "7844aa644f3a40f5fe63e2208648415b0212f4fd8af1fcbf3b71a17f97fcfd5d097aac398aea1c2640b08c0b8396353bd88df5166f2f7d85c2cb93ad0fe29695"}&apos;, summary: f6233e3e56d23525d17b870dda9174c2978cf10843359215835e487d02b78554)
[14:28:37.928 [MainThread] poet_block_verifier ERROR] Block a4299924 rejected: Received block from an unregistered validator 02d69ef8...014f39db
[14:28:37.928 [MainThread] engine INFO] Failed consensus check: a4299924b77cc32934ac6a470636312b24c9153327b5b7e2e878640f85c0442d5f3dca4bfc4dd4b6575ef047c63e01da3228cbac8d4b6d68c149b9d2589720b1
[14:28:55.147 [Thread-2] zmq_driver ERROR] Uncaught driver exception
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/sawtooth_sdk/consensus/zmq_driver.py", line 88, in _driver_loop
result = self._process(message)
File "/usr/lib/python3/dist-packages/sawtooth_sdk/consensus/zmq_driver.py", line 237, in _process
&apos;Received unexpected message type: {}&apos;.format(type_tag))
sawtooth_sdk.consensus.exceptions.ReceiveError: Received unexpected message type: 700
Edit: I'm currenly working on Ubuntu 18.04

So, poet-engine doesn't implement PING Message (MessageType=700)

Try not to use hyperledger/sawtooth-poet-engine:chime docker image for poet-engine-# containers in sawtooth-default-poet.yaml file.
Change it to hyperledger/sawtooth-poet-engine:nightly to all containers if you are building a network.
If you feel uncomfortable with the nightly version try others(https://hub.docker.com/r/hyperledger/sawtooth-poet-engine/tags) but it seems outdated.
Don't forget to use this command to clear all of the previous block data in docker volume when necessary.
docker volume rm $(docker volume ls -q)

How can I print 10 lines above and until a matching word below for each matching line?

I need to parse log file so I output the errors(with the stack trace below), and 10 lines above each error.
For example:
2017-10-29 00:00:10,440 INFO ...
2017-10-29 00:00:10,473 WARN ...
2017-10-29 00:00:10,504 INFO ...
2017-10-29 00:00:10,547 INFO ...
2017-10-29 00:00:10,610 INFO ...
2017-10-29 00:00:11,176 WARN ...
2017-10-29 00:00:11,894 WARN ..
2017-10-29 00:00:11,900 INFO ...
2017-10-29 00:00:11,900 INFO ...
2017-10-29 00:00:12,632 WARN ...
2017-10-29 00:00:12,946 ERROR...
...(stack trace)...
...(stack trace)...
...(stack trace)...
2017-10-29 00:00:12,946 WARN
I need to output 10 lines above the ERROR until the the date(2017-10-29) below(not including the line of the date)
Thought about doing it with grep -n -B10 "ERROR"(for the 10 lines above) and sed '/ERROR/,/29/p'(for the stack trace) but how do I combine the two?

With grep + head pipeline:
grep -B10 'ERROR' g1 | head -n -1

This might work for you (GNU sed):
sed -n ':a;N;/ERROR/bb;s/[^\n]\+/&/11;Ta;D;:b;p;n;/2017-10-29$/!bb' file
Gather up at most 10 lines in the pattern space then use these lines as moving window through the file. When the string ERROR is encountered print the window and then any further lines until (but not including) the string 2017-10-29 is matched. Repeat if necessary.

Try this one
grep -no -B10 ' [0-9:,]* ERROR.*' infile
Need perhaps to substitute ' ' by [[:blank:]]

here is a way using awk:
awk ' {a[++b]=$0}
/^([0-9]{2,4}-?){3}/ {f=0}
/ERROR/ {f=1; for(i=NR-10;i<NR;i++) print a[i]}
f' file
we store each line to array. When matching a date log line, we unset the flag. When matching ERROR we set the flag and we print last 10 lines of the array. And when flag is on, we print (default action so we wrote just f)
This should print expected lines for all existing ERRORs in file.
note: the date regexp used is not strict but seems enough for the case.

Limit of argument size of a command (for ex: in grep "$some_argument_size" filename)

I'm running a shell script and line# 15 is:
grep "$var1$var2" somefilewithonly10000lines_letsay
and getting an error:
line 15: /bin/grep: Argument list too long
What's the limit for the grep command's pattern? -- which when I reach, gives me the above error message.

THe following will tell the limit. Thanks Cyrus.
$ getconf ARG_MAX
More info: http://www.in-ulm.de/~mascheck/various/argmax/

CMake how to remove the path from compile errors

When using cmake if there is a compile error the error is output with the entire path to the file containing the error. This path can be very long (see example below) so often it word wraps and makes it difficult to read.
Example Output:
/home/nick/projects/projectA/src/environment/base/terrain/base/TestFile.h:21:37: error: ‘TestFile’ does not name a type
/home/nick/projects/projectA/src/environment/base/terrain/base/TestFile.h:21:54: error: expected unqualified-id before ‘test’
/home/nick/projects/projectA/src/environment/base/terrain/base/TestFile.h:21:54: error: expected ‘)’ before ‘test’
Is there a way to remove the path from the error?
Something like this:
TestFile.h:21:37: error: ‘TestFile’ does not name a type
TestFile.h:21:54: error: expected unqualified-id before ‘test’
TestFile.h:21:54: error: expected ‘)’ before ‘test’
Thanks

I don't know if it is possible within cmake, but you can always redirect stderr to stdout and filter the output with a short sed script. At least the common project path can be filtered
make 2>&1 | sed 's/\/home\/nick\/projects\/projectA\/src\///g'

Boot script terminates and gives error (no error logger present)

I made an Erlang application, that shall be started on booting of the operating system. The boot script is stored in /etc/init.d. It looks like this:
#!/bin/sh
cd $ROOT/lib/di
INET_ADDR=$(ifconfig eth0 | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1}')
NODE_NAME=$(echo di#$INET_ADDR)
erl -pa $PWD/ebin -pa $PWD/deps/*/ebin -name $NODE_NAME -boot di $1 -setcookie agfeo
The script tries to determine the IP address of the machine, in order to give the node an unique name. When the machine boots, the script gets executed automatically. On the terminal I get the following output:
(no error logger present) error: "Error in process <0.1.0> with exit value:
{badarg,[{erlang,list_to_atom,[[<<2 bytes>>,<<5 bytes>>,46,98,111,111,116]]},
{init,get_boot,2},{init,do_boot,3}]}"
{"init terminating in do_boot",{badarg,[{erlang,list_to_atom,[[<<2 bytes>>,<<5 bytes>>,46,98,111,111,116]]},
{init,get_boot,2},{init,do_boot,3}]}}
init terminating in do_boot ()
This is what the shell prints out, when the script is loaded automatically.
When I call the script manually, my application gets started normally, without any problems.
Could anybody please explain, what the error message above means?

If we look at the stack trace the last function executed is init:get_boot/2 and the last instruction is erlang:list_to_atom([<<2 bytes>>, <<5 bytes>>, ".boot"]). In the init:get_boot/2 there are three lines with list_to_atom, so error should be one of the following:
'cannot get bootfile';
'bootfile format error';
I believe the error is 'cannot get bootfile'.

Resources