I have a txt file that has only information about location (location.txt)
Another large txt file (all.txt) has a lot of information like id , and a location.txt is subset of all.txt ( some records common in both )
I want to search the location.txt in another file with grep (all.txt)
and print all common records ( but all information like all.txt )
I try to grep by :
grep -f location.txt all.txt
the problem grep just give me the last location not all locations
how can I print all location?
I'm assuming you mean to use one of the files as a set of patterns for grep. If this is the case, you seem to be looking for a way to print all lines in one file not found in the other and this is what you want:
grep -vFf file_with_patterns other_file
Explanation
-F means to interpret the pattern(s) literally, giving no particular meaning to regex metacharacters (like * and +, for example)
-f means read regex patterns from the file named as argument (file_with_patterns in this case).
Am given a list if ID which I need to trace back a name in a file
file: ID contains
1
2
3
4
5
6
The ID are contained in a Large 2 GB file called result.txt
ABC=John,dhds,72828,73737,3939,92929
CDE=John,uubad,32424,ajdaio,343533
FG1=Peter,iasisaio,097282,iosoido
WER=Ann,97391279,89719379,7391739
result,**id=1**,iuhdihdio,ihwoihdoih,iuqhwiuh,ABC
result2,**id=2**,9729179,hdqihi,hidqi,82828,CDE
result3,**id=3**,biasi,8u9829,90u209w,jswjso,FG1
So I cat the ID file into a variable
I then use this variable in a loop to grep out the values to link back to the name using grep and cut -d from results.txt and output to a variable
so variable contains ABS CDE FG1
In the same loop I pass the output of the grep to perform another grep on results.txt, to get the name
ie regrets file for ABC CDE FG1
I do get the answer but takes a long time is their a more efficient way?
Thanks
Making some assumptions about your requirement... ID's that are not found in the big file will not be shown in the output; the desired output is in the format shown below.
Here are mock input files - f1 for the id's and f2 for the large file:
[mathguy#localhost test]$ cat f1
1
2
3
4
5
6
[mathguy#localhost test]$ cat f2
ABC=John,dhds,72828,73737,3939,92929
CDE=John,uubad,32424,ajdaio,343533
FG1=Peter,iasisaio,097282,iosoido
WER=Ann,97391279,89719379,7391739
result,**id=1**,iuhdihdio,ihwoihdoih,iuqhwiuh,ABC
result2,**id=2**,9729179,hdqihi,hidqi,82828,CDE
result3,**id=3**,biasi,8u9829,90u209w,jswjso,FG1
Proposed solution and output:
[mathguy#localhost test]$ sed 's/.*/\*\*id=&\*\*/' f1 | grep -Ff - f2 | \
> sed -E 's/^.*\*\*id=([[:digit:]]*)\*\*.*,([^,]*)$/\1 \2/'
1 ABC
2 CDE
3 FG1
The hard work here is done by grep -F which might be just fast enough for your needs. There is some prep work and some clean-up work done by sed, but those are both on small datasets.
First we take the id's from the input file and we output strings in the format **id=<number>**. The output is presented as the fixed-character patterns to grep -F via the option -f (take the patterns from file, in this case from stdin, invoked as -; that is, from the output of sed).
After we find the needed lines from the big file, the final sed just extracts the id and the name from each line.
Note: this assumes that each id is only found once in the big file. (Actually the command will work regardless; but if there are duplicate lines for an id, your business users will have to tell you how to handle. What if you get contradictory names for the same id? Etc.)
Say I have two files:
1.json
{"foo":"bar"}\n
2.json
{"foo":"bar"}
when using checksum routines, is there a way to ignore the trailing whitespace?
maybe something like this:
md5sum < <(cat file | trim_somehow)
You can use sed or xargs.
xargs is much simpler, but be careful with it. I'm not sure if it is safe to use in this context. Read the comments below this answer https://stackoverflow.com/a/12973694/4330274. (There are a lot of answers to your question in that post).
md5sum < <(cat file | xargs) will delete trailing/leading whitespaces (Also, as stated by dave_thompson_085 on the comments down below, it will compress each whiltespace sequence to one whitespace and it will remove quotation marks and backslashes) from the file before passing it to the md5sum utility.
Note: xargs appends a new line to the end of the input.
I recommend using sed for this purpose. It is much safer. Read this answer https://stackoverflow.com/a/3232433/4330274
I am wanting to search for files that contain 'even:suspendcount>0' AND 'even:holdcount>0'. These 2 strings of text must be somewhere in the file, not necessarily on the same line. The problem I am running into is that my search results are not pulling back files that contain 1 sting of text on say line #5 and the other on line #10. It is only pulling back files if they are on the same line number. How would I search for files that contains multiple strings of text just somewhere in the file, they do not have to be on the same line.
Using grep
To use grep to get files that have both strings in either order:
grep -lZ 'even:suspendcount>0' * | xargs --null grep -l 'even:holdcount>0'
How it works:
grep -lZ 'even:suspendcount>0' *
This returns a nul-separated list of the names of files which contain the string even:suspendcount>0.
xargs --null grep -l 'even:holdcount>0'
Of the files selected by the first step, this returns the names of files which also contain even:holdcount>0
Because we are using nul-separation when passing the file names from one process to the next, this approach is safe even for difficult file names.
Using awk
This prints the file name of any file that contains both strings:
awk 'BEGINFILE{f=0;g=0} /even:suspendcount>0/{f=1} /even:holdcount>0/{g=1} f && g{print FILENAME; nextfile}' *
How it works:
BEGINFILE{f=0;g=0}
As we start reading a new file, variables f and g are set to zero (false).
/even:suspendcount>0/{f=1}
If we encounter a line containing even:suspendcount>0, then set variable f to 1.
/even:holdcount>0/{g=1}
Similarly, f we encounter a line containing even:holdcount>0, then set variable g to 1.
f && g{print FILENAME; nextfile}
If both f and g are true (nonzero), then print the filename and skip to the next file.
A grep pattern is line-oriented, i.e. in your case it should be 'even:suspendcount>0' OR 'even:holdcount>0' (namely grep -E 'even:(suspend|hold)count>0').
I want to read all the keys from a dictionary which in plist using plistbuddy and push the value into an array.
I knew that the below utility help to read plist.
/usr/libexec/PlistBuddy
So far I'm hard coding the key to get value.
Eg: theKey = "selva"
So i write below code to get value of above key
val=$(function_stringFields "${theFile}" "${theKey}" "${index}")
(function_stringFields is a custom method.)
Is there anyway to achieve the same.
plist_file=/Applications/iTunes.app/Contents/Info.plist
keys=(`/usr/libexec/PlistBuddy -c Print "$plist_file" | perl -lne 'print $1 if /^ (\S*) =/'`)
This will create an array keys that holds all the top-level keys. All it's doing is looking for output that starts with four spaces and then a key and then "=".
The key to getting this done is the print option for plistbuddy.
Without arguments it will print out all lines.
Here is a small script that will load an array of keys and and array of values.
Success of this will really depend on what kind of tree you may have built in the plist. If it is flat you will be happier. This example also does not account for spaces in the values (they would be removed).
LINES=`/usr/libexec/PlistBuddy FrameworkList.plist -c print | grep = | tr -d ' '`
COUNTER=0
for PLIST_ITEMS in $LINES;
do
KEY[${COUNTER}]=`echo $PLIST_ITEMS | cut -d= -f1`
VALUE[${COUNTER}]=`echo $PLIST_ITEMS | cut -d= -f2`
COUNTER=${COUNTER}+1
done