I often compute checksums of files downloaded from the Internet, using shasum family of commands, without paying attention to the mode for reading. In particular, sha1sum usually defaults to text mode for applications.
What is the difference between reading in text mode and binary mode, when checking checksums using sha1sum?
~/Downloads$ sha1sum --help
Usage: sha1sum [OPTION]... [FILE]...
Print or check SHA1 (160-bit) checksums.
With no FILE, or when FILE is -, read standard input.
-b, --binary read in binary mode
-c, --check read SHA1 sums from the FILEs and check them
--tag create a BSD-style checksum
-t, --text read in text mode (default)
-z, --zero end each output line with NUL, not newline,
and disable file name escaping
The following five options are useful only when verifying checksums:
--ignore-missing don't fail or report status for missing files
--quiet don't print OK for each successfully verified file
--status don't output anything, status code shows success
--strict exit non-zero for improperly formatted checksum lines
-w, --warn warn about improperly formatted checksum lines
--help display this help and exit
--version output version information and exit
The sums are computed as described in FIPS-180-1. When checking, the input
should be a former output of this program. The default mode is to print a
line with checksum, a space, a character indicating input mode ('*' for binary,
' ' for text or where binary is insignificant), and name for each FILE.
GNU coreutils online help: <https://www.gnu.org/software/coreutils/>
Full documentation <https://www.gnu.org/software/coreutils/sha1sum>
or available locally via: info '(coreutils) sha1sum invocation'
None.
At least judging by the answer to the similar question on the difference between text and binary in md5sum.
It appears it flags are for standards compliance or something
Related
I want to modify the checks that the code analyzer program clang-tidy is doing, but it seems like the content of the configuration file .clang-tidy is being ignored.
I create the file by calling clang-tidy with the flag -dump-config and redirect the output to the file .clang-tidy.
Then I call sed to replace the value 800 with the value 700, which corresponds to the option with key google-readability-function-size.StatementThreshold. The specific option is not important to me, this is just for testing.
I verify that the value has indeed been changed.
Lastly, I rerun clang-tidy to see if it has accepted the new configuration, but it remains unchanged.
# generate config
clang-tidy -dump-config > .clang-tidy
# change config
sed -i 's/800/700/' .clang-tidy
# verify change
grep '700' .clang-tidy
# use config, does not work
clang-tidy -config '' -dump-config
The CheckOption remains at the default value, the content of the config file has been ignored:
CheckOptions:
# some lines omitted for brevity
- key: google-readability-function-size.StatementThreshold
value: '800'
Running clang-tidy -config '' -dump-config -explain-config shows that the configuration file has at least been found, i.e. many clang-analyzer specific checks are enabled in the detected config file, but the check google-readability-function-size.StatementThreshold is not listed.
I also tried passing the config directly as command line parameter with the command clang-tidy -config="{CheckOptions: [ {key: google-readability-function-size.StatementThreshold, value: 700} ]}" -dump-config, but got the same result.
The command clang-tidy --version gives the following output, running on Ubuntu 20.04:
LLVM (http://llvm.org/):
LLVM version 10.0.0
Optimized build.
Default target: x86_64-pc-linux-gnu
Host CPU: haswell
To see the change, you need to enable the check:
Checks: 'google-readability-function-size'
You can see it changed in the effective configuration with:
clang-tidy --dump-config
Another pitfall to be aware of is that errors parsing the values will be silently discarded.
I am trying to check whether variable GROUP exist in SAS data set file or not from the UNIX command but unfortunately it's showing that GROUP variable does not exist in the data set,However GROUP variable is present in SAS data set.
In my command for case sensitive and whole word match I am using i and w options of grep command respectively. But still UNIX command is not giving the expected result.I s there any way to fix this issue?
Below is the command which I am using:
sasfile="sasdata"
rwords="GROUP"
cat $sasfile | grep -iqw "$rwords"
Thank you
As mentioned in earlier comment
SAS data sets are stored in disk files using a proprietary format.
There may be encodings and storage methodologies that do not yield the
information you seek in a plain text examination of said disk file.
Running SAS code in a SAS session is the definitive way to glean information about a data set.
What will that code look like ?
Proc CONTENTS
Data step or macro code that uses VARNAME function
... many other ways ...
In UNIX SAS can use stdio.
From "SAS(R) 9.2 Companion for UNIX Environments", STDIO System Option: UNIX
Details
This option tells SAS to take its input from standard input (stdin),
to write its log to standard error (stderr), and to write its output
to standard output (stdout).
This option is designed for running SAS
in batch mode or from a shell script. If you specify this option
interactively, SAS starts a line mode session.
The STDIO option
overrides the DMS, DMSEXP, and EXPLORER system options. The STDIO
option does not affect the assignment of the Stdio, Stdin, and Stderr
filerefs. See Filerefs Assigned by SAS in UNIX Environments for more
information.
For example, in the following SAS command, the file
myinput is used as the source program, and files myoutput and mylog
are used for the procedure output and log respectively.
sas -stdio < myinput > myoutput 2> mylog
If you are using the C shell, you should
use parentheses:
(sas -stdio < myinput > myoutput ) >& output_log
With -stdio you want a short SAS program that can indicate if a variable is present in a data set, or perhaps emit a list of variables in a data set for further shell processing. A Proc CONTENTS step is short and sweet.
So looking for your proverbial needle in a haystack
sasfile=<path to data set file>/<dataset>.sas7bdat
needle=GROUP
echo "Proc CONTENTS data=""$sasfile""" | sas -stdio | grep $needle
The default CONTENTS output might contain yield some false matches. So you could also try
echo "Proc CONTENTS noprint data=""$sasfile"" out=list;data _null_;set list;file print;put name;"
| sas -stdio
| grep -i "GROUP"
You could try:
sasfile="sasdata"
rwords="GROUP"
grep -iw "$rwords" "$sasfile"
The only difference between these and your original commands is that I omitted cat and grep's quiet flag -q.
Sample input in sasdata:
fasd group
fdsfds fdsfdsa
fdsfd as GROUP afdsfdsa
Output:
fasd group
fdsfd as GROUP afdsfdsa
The -q flag of grep will suppress standard output but echo $? can retrieve the return value of grep. Using the same input file as before:
grep -iqw "$rwords" "$sasfile" # No stout
echo $? # Prints 0, means grep succeeded
grep -iqw "word" "$sasfile" # No stout
echo $? # Prints 1, means grep failed
I have used spiffsimg to create a single file containing multiple lua files:
# ./spiffsimg -f lua.img -c 262144 -r lua.script
f 4227 init.lua
f 413 cfg.lua
f 2233 setupWifi.lua
f 7498 configServer.lua
f 558 cfgForm.htm
f 4255 setupConfig.lua
f 14192 main.lua
#
I then use esptool.py to flash the NodeMCU firmware and the file containing the lua files to the esp8266 (NodeMCU dev kit):
c:\esptool-master>c:\Python27\python esptool.py -p COM7 write_flash -fs 32m -fm dio 0x00000 nodemcu-dev-9-modules-2016-07-18-12-06-36-integer.bin 0x78000 lua.img
esptool.py v1.0.2-dev
Connecting...
Running Cesanta flasher stub...
Flash params set to 0x0240
Writing 446464 # 0x0... 446464 (100 %)
Wrote 446464 bytes at 0x0 in 38.9 seconds (91.9 kbit/s)...
Writing 262144 # 0x78000... 262144 (100 %)
Wrote 262144 bytes at 0x78000 in 22.8 seconds (91.9 kbit/s)...
Leaving...
I then run ESPLorer to check the status and get:
PORT OPEN 115200
Communication with MCU..Got answer! AutoDetect firmware...
Can't autodetect firmware, because proper answer not received.
NodeMCU custom build by frightanic.com
branch: dev
commit: b21b3e08aad633ccfd5fd29066400a06bb699ae2
SSL: true
modules: file,gpio,http,net,node,rtctime,tmr,uart,wifi
build built on: 2016-07-18 12:05
powered by Lua 5.1.4 on SDK 1.5.4(baaeaebb)
lua: cannot open init.lua
>
----------------------------
No files found.
----------------------------
>
Total : 3455015 bytes
Used : 0 bytes
Remain: 3455015 bytes
The NodeMCU firmware flashed correctly, but the lua files can't be located.
I have tried flashing to other locations (0x84000, 0x7c000), but I am just guessing at these locations based on reading threads on github.
I used the NodeMCU file.fscfg() routine to get the flash address and size. If I only flash the NodeMCU firmware I get the following:
print (file.fscfg())
524288 3653632
534288 is 0x80000, so I tried flashing only the spiffsimg file (lua.img) to 0x8000, then ran the same print statement and got:
print (file.fscfg())
786432 3391488
The flash address incremented by the exact number of bytes in the lua.img - which I don't understand, why would the flash address change? Is the first number returned by file.fscfg not the starting flash address, but the ending flash address?
What is the correct address for flashing an image file, contain lua files, that was created by spiffsimg?
The version of spiffsimg found here will provide the correct address for flashing an image file that contains lua files.
Do not use this version of spiffsimg as it is out of date.
To install the spiffsimg utility, you need to download and install the entire nodemcu-firmware package (into a linux environment, use make to install - note: make on my debian linux box generated an error, but i was able to go to the ../tools/spiffsimg subdirectory and run make on the Makefile found in that directory to create the utility).
The spiffsimg instructions found here are quite clear, with one exception: the file name you specify, with the -f parameter, needs to include the characters %x. The %x will be replaced with the address that the image file should be flashed to.
For example, the command
spiffsimage -f %x-luaFiles.img -S 4MB -U 465783 -r lua.script
will create a file, in the local directory, with a name like: 80000-luaFiles.img. Which means you should install that image file at address 0x80000 on the ESP8266.
I've never done that myself but I'm reasonably confident the correct answer can be extracted from the docs.
-f specifies the filename for the disk image. '%x' will be replaced
by the calculated offset of the file system.
And a bit further down
The disk image file is placed into the bin directory and it is named
0x<offset>-<size>.bin where the offset is the location where it
should be flashed, and the size is the size of the flash part.
However, there's a slight mismatch between the two statements. We may have a bug in the docs. If "'%x' will be replaced..." then I'd expected the final name won't contain 0x anymore.
Furthermore, it is possible to define a fixed SPIFFS position when you build the firmware.
#define SPIFFS_FIXED_LOCATION 0x100000
This specifies that the SPIFFS filesystem starts at 1Mb from the start of the flash. Unless
otherwise specified, it will run to the end of the flash (excluding
the 16k of space reserved by the SDK).
When input is read from terminal, GNU Parallel always displays a warning:
parallel: Warning: Input is read from the terminal. Only experts do this on purpose. Press CTRL-D to exit.
But sometimes I do want to read from terminal (e.g., when I'm copy & pasting stuff from elsewhere entry by entry). Is it possible to turn off this warning? I couldn't find such an option in man parallel or man parallel_tutorial.
Note that I don't want a cheap solution like 2>/dev/null, since warning messages from other programs will be turned off, too. For instance, consider the following simple script:
#!/bin/bash
function print12 () {
echo "printing $1 to stdout"
echo "printing $1 to stderr" >/dev/stderr
}
export -f print12
SHELL=/bin/bash parallel -k print12 2>/dev/null
Messages printed to stderr will all be suppressed.
Just realized that I can do a cat or some read </dev/tty to achieve my desired effect. But let's just focus on the original question.
It cannot be turned off. But see it as a praise: Since you are doing it on purpose, you are an expert (at least in the eyes of GNU Parallel).
As it is just a warning, you are free to paste your arguments and have them run: The warning does not stop GNU Parallel from reading your input.
If you really do not like the warning:
cat | parallel ...
I've worked extensively with ROOT, which has it's own format for data files, but for various reasons we would like to switch to HDF5 files. Unfortunately we'd still require some way of translating files between formats. Does anyone know of any existing libraries which do this?
You might check out rootpy, which has a facility for converting ROOT files into HDF5 via PyTables: http://www.rootpy.org/commands/root2hdf5.html
If this issue is still of interest to you, recently there have been large improvements to rootpy's root2hdf5 script and the root_numpy package (which root2hdf5 uses to convert TTrees into NumPy structured arrays):
root2hdf5 -h
usage: root2hdf5 [-h] [-n ENTRIES] [-f] [--ext EXT] [-c {0,1,2,3,4,5,6,7,8,9}]
[-l {zlib,lzo,bzip2,blosc}] [--script SCRIPT] [-q]
files [files ...]
positional arguments:
files
optional arguments:
-h, --help show this help message and exit
-n ENTRIES, --entries ENTRIES
number of entries to read at once (default: 100000.0)
-f, --force overwrite existing output files (default: False)
--ext EXT output file extension (default: h5)
-c {0,1,2,3,4,5,6,7,8,9}, --complevel {0,1,2,3,4,5,6,7,8,9}
compression level (default: 5)
-l {zlib,lzo,bzip2,blosc}, --complib {zlib,lzo,bzip2,blosc}
compression algorithm (default: zlib)
--script SCRIPT Python script containing a function with the same name
that will be called on each tree and must return a tree or
list of trees that will be converted instead of the
original tree (default: None)
-q, --quiet suppress all warnings (default: False)
As of when I last checked (a few months ago) root2hdf5 had a limitation that it could not handle TBranches which were arrays. For this reason I wrote a bash script: root2hdf (sorry for non-creative name).
It takes a ROOT file and the path to the TTree in the file as input arguments and generates source code & compiles to an executable which can be run on ROOT files, converting them into HDF5 datasets.
It also has a limitation that it cannot handle compound TBranch types, but I don't know that root2hdf5 does either.