Related
I'd like to write a bit of code that looks like this:
while ( `ls -1 ${JOB_PREFIX}_${job_counter}_*.out | wc -l` > 0 )
echo "Do something here."
end
But every time there is no ls -1 ${JOB_PREFIX}_${job_counter}*.csh it gives an annoying ls: No match.
Is it possible to suppress this error message but still pipe STDOUT to wc ? I went through the existing questions on this but most of the answers are either not working on csh or tries to combine STDOUT and STDERR with |& which I do not want.
The cleanest way would have been to mute the ls itself, but it is not possible.
You can suppress your entire script, meaning:
If your loops is in a file called myscript.cs and you call your file with some args myscript.cs -arg1 blabla -arg2 bla, add >& to your shell command to redirect stderr to wherever you want. e.g.
myscript.cs -arg1 blabla -arg2 bla >& /dev/null
It's not answering your answer directly, but it should solve your problem.
Edit
Since the comment added that the script should be redirected to another script, you can split your line to two lines:
if `ls -1 ${JOB_PREFIX}_${job_counter}_*.out >& /dev/null` then
while < your while here >
end
endif
csh and tcsh cannot redirect stderr by itself, one of many reasons you should not use it for scripting.
If you're okay with spawning another shell, you can change JOB_COUNTER into an environment variable and do:
while ( `sh -c '2>/dev/null ls -1 ${JOB_PREFIX}_${JOB_COUNTER}_*.out | wc -l'` > 0 )
echo "Do something here."
end
If you want to use a test as a condition in a while loop instead of an expression, you have to get creative.
#!/bin/tcsh -f
set i = 4
while ( 1 )
if ( $i <= 0 ) break
echo hi
# i = ($i - 1)
end
In your case, this might look something like: I changed the wc -l to a grep -q . so that we can use the exit status instead of the result printed to stderr.
#!/bin/tcsh -f
set dir_exit_status = 0
while ( $dir_exit_status = 0 )
( ls -1 ${JOB_PREFIX}_${job_counter}_*.out | grep -q '.') >& /dev/null
set dir_exit_status = $status
end
I have the following piece of code, which works as expected. It ensures that 2 processes are always spawned, and if any process fails, the script comes to a halt.
I have worked with GNU parallel earlier on simple one line scripts and they have worked really well.I'm sure the one below too can be made simpler.
The sleeper function in reality is MUCH more complex than one shown below.
The objective is that GNU parallel will call sleeper function in parallel and also do error handling
`sleeper(){
stat=$1
sleep 5
echo "Status is $1"
return $1
}
PROCS=2
errfile="errorfile"
rm "$errfile"
while read LINE && [ ! -f "$errfile" ]
do
while [ ! -f "$errfile" ]
do
NUM=$(jobs | wc -l)
if [ $NUM -lt $PROCS ]; then
(sleeper $LINE || echo "bad exit status" > "$errfile") &
break
else
sleep 2
fi
done
done<sleep_file
wait`
Thanks
What you are looking for is --halt (requires version 20150622):
sleeper(){
stat=$1
sleep 5
echo "Status is $1"
return $1
}
export -f sleeper
parallel -j2 --halt now,fail=1 -v sleeper ::: 0 0 0 1 0 1 0
If you do not want the sleeper to get killed (maybe you want it to finish so it cleans up), then use --halt soon,fail=1 to let the running jobs complete without starting new ones.
I've got a small script called "onewhich". Its purpose is to behave like which, except that it will only give the FIRST occurrence of any executables specified as options, as found in the order they'd appear in the path.
So for example, if my path is /opt/bin:/usr/bin:/bin, and I have both /opt/bin/runme and /usr/bin/runme, then the command onewhich runme would return /opt/bin/runme.
But if I also have a /usr/bin/doit, then the command onewhich doit runme would return /usr/bin/doit instead.
The idea is to walk through the path, check for each executable specified, and if it exists, show it and exit.
Here's the script so far.
#!/bin/sh
for what in "$#"; do
for loc in `echo "${PATH}" | awk -vRS=: 1`; do
if [ -f "${loc}/${what}" ]; then
echo "${loc}/${what}"
exit 0
fi
done
done
exit 1
The problem is, I want to be better about PATH directories with special characters. Every second shell question here on StackOverflow talks about how bad it is to parse paths with tools like awk and sed. There's even a bash faq entry about it. (Proviso: I'm not using bash for this, but the recommendation is still valid.)
So I tried rewriting the script to separate paths in a pipe, like this"
#!/bin/sh
for what in "$#"; do
echo "${PATH}" | awk -vRS=: 1 | while read loc ; do
if [ -f "${loc}/${what}" ]; then
echo "${loc}/${what}"
exit 0
fi
done
done
exit 1
I'm not sure if this gives me any real advantage (since $loc is still inside quotes), but it also doesn't work because for some reason, the exit 0 seems to be ignored. Or ... it exits something (the sub-shell with the while loop that terminates the pipe, maybe), but the script exits with a value of 1 every time.
What's a better way to step through directories in ${PATH} without the risk that special characters will confuse things?
Alternately, am I reinventing the wheel? Is there maybe a way to do this that's built in to existing shell tools?
This needs to run in both Linux and FreeBSD, which is why I'm writing it in Bourne instead of bash.
Thanks.
This doesn't directly answer your question, but does eliminate the need to parse PATH at all:
onewhich () {
for what in "$#"; do
which "$what" 2>/dev/null && break
done
}
This just calls which on each command on the input list until it finds a match.
To parse PATH, you can simply set `IFS=':'.
if [ "${IFS:-x}" = "${IFS-x}" ]; then
# Only preserve the value of IFS if it is currently set
OLDIFS=$IFS
fi
IFS=":"
for f in $PATH; do # Do not quote $PATH, to allow word splitting
echo $f
done
if [ "${OLDIFS:-x}" = "${OLDIFS-x}" ]; then
IFS=$OLDIFS
fi
The above will fail if any of the directories in PATH actually contain colons.
Your first method looks to me as if it should work. In practical terms, if it's really the $PATH you'll be searching, it's unlikely you'll have spaces and newlines embedded in directories there. If you do, it's probably time to refactor.
But still, I don't think you're at risk from the possibility of bad names clobbering your loop, since you're wrapping variables in quotes. At worst, I suspect you might miss the odd valid executable, but I can't see how the script would generate errors. (I don't see how the script would miss valid executables, and I haven't tested - I'm just saying I don't see problems at first glance.)
As for your second question, about the loop, I think you've hit the nail on the head. When you run a pipe like this | that | while condition; do things; done, the while loop runs in its own shell at the end of the pipe. Exiting that shell may terminate the actions of the pipe, but that only brings you back to the parent shell, which has its own thread of execution that terminates with exit 1.
As for a better way to do this, I would consider which.
#!/bin/sh
for what in "$#"; do
which "$what"
done | head -1
And if you really want the exit values as well:
#!/bin/sh
for what in "$#"; do
which "$what" && exit 0
done
exit 1
The second might even be fewer resources, as it doesn't have to open a file handle and pipe through head.
You can also split your path using IFS. For example, if you wanted to wrap your loops the other way around, you could do this:
#!/bin/sh
IFS=":"
for loc in $PATH; do
for what in "$#"; do
if [ -x "$loc"/"$what" ]; then
echo "$loc"/"$what"
exit 0
fi
done
done
exit 1
Note that under normal circumstances, you might want to save the old value of $IFS, but you seem to be doing things in a stand-alone script, so the "new" value gets thrown out when the script exits.
All the above code is untested. YMMV.
Another way to get around the need to parse PATH at all is to run the builtin type command in new shell with a stripped environment (i. e. there simply are no functions or aliases to look up; cf. env -i sh -c 'type cmd 2>/dev/null).
# using `cmd` instead of $(cmd) for portability
onewhich() {
ec=0 # exit code
for cmd in "$#"; do
command -p env -i PATH="$PATH" sh -c '
export LC_ALL=C LANG=C
cmd="$1"
path="`type "$cmd" 2>/dev/null`"
if [ X"$path" = "X" ]; then
printf "%s\n" "error: command \"${cmd}\" not found in PATH" 1>&2
exit 1
else
case "$path" in
*\ /*)
path="/${path#*/}"
printf "%s\n" "$path";;
*)
printf "%s\n" "error: no disk file: $path" 1>&2
exit 1;;
esac
exit 0
fi
' _ "$cmd"
[ $? != 0 ] && ec=1
done
[ $ec != 0 ] && return 1
}
onewhich awk ls sed
onewhich builtin
onewhich if
Since which on success returns two full command paths if two commands are specified as arguments, exit 0 in the first onewhich script above aborts the program prematurely. In addition, if two commands are specified as arguments to which, the exit code of which is set to 1 even if only one command lookup failed (cf. which awk sedxyz ls; echo $?). To mimic this behaviour of the which command it is necessary to toggle on/off two variables (cnt and nomatches below).
onewhich() (
IFS=":"
nomatches=0
for cmd in "$#"; do
cnt=0
for loc in $PATH ; do
if [ $cnt = 0 ] && [ -x "$loc"/"$cmd" ]; then
echo "$loc"/"$cmd"
cnt=1
fi
done
[ $cnt = 0 ] && nomatches=1
done
[ $nomatches = 1 ] && exit 1 || exit 0 # exit 1: at least one cmd was not in PATH
)
onewhich awk ls sed
onewhich awk lsxyz sed
onewhich builtin
onewhich if
I have a CSV file that shows the statistics for links on a half an hour basis. The link name only appears on the 00:00 line.
link1,0:00,0,0,0,0
,00:30,0,0,0,0
,01:00,0,0,0,0
,01:30,0,0,0,0
,02:00,0,0,0,0
,02:30,0,0,0,0
,03:00,0,0,0,0
,03:30,0,0,0,0
,23:30,0,0,0,0
....
....
link2,00:00,0,0,0,0
How do I copy the link name to every other line until the link name is different, using sed or awk?
With awk, just keep track of the last seen non-empty link name, and always use that.
awk -F, -v OFS=, '$1 != "" { link=$1 } { $1 = link; print $0 }'
Omitting the ellipses, this gives:
link1,0:00,0,0,0,0
link1,00:30,0,0,0,0
link1,01:00,0,0,0,0
link1,01:30,0,0,0,0
link1,02:00,0,0,0,0
link1,02:30,0,0,0,0
link1,03:00,0,0,0,0
link1,03:30,0,0,0,0
link1,23:30,0,0,0,0
link2,00:00,0,0,0,0
This is a simpler job with awk, but if you want to use sed:
sed -e '/^[^,]/{h;s/,.*//;x};/^,/{G;s/^\(.*\)\n\(.*\)/\2\1/}'
Bellow a commented version in sed script file format that can be run with sed -f script:
# For lines not beginning with a ',', saves what precedes a ',' in the hold space and print the original line.
/^[^,]/{
h
s/,.*//
x}
# For lines beginning with a ',', put what has been save in the hold space at the beginning of the pattern space and print.
/^,/{
G
s/^\(.*\)\n\(.*\)/\2\1/}
You can do that in pure bash shell without needing to start a new process, which should be faster than using awk or sed:
IFS=","
while read v1 v2; do
if [[ $v1 != "" ]]; then
link=$v1;
fi
printf "%s,%s\n" "$link" "$v2"
done < file
It is typical to have something like this in your cshrc file for setting the path:
set path = ( . $otherpath $path )
but, the path gets duplicated when you source your cshrc file multiple times, how do you prevent the duplication?
EDIT: This is one unclean way of doing it:
set localpaths = ( . $otherpaths )
echo ${path} | egrep -i "$localpaths" >& /dev/null
if ($status != 0) then
set path = ( . $otherpaths $path )
endif
Im surprised no one used the tr ":" "\n" | grep -x techique to search if a given folder already exists in $PATH. Any reason not to?
In 1 line:
if ! $(echo "$PATH" | tr ":" "\n" | grep -qx "$dir") ; then PATH=$PATH:$dir ; fi
Here is a function ive made myself to add several folders at once to $PATH (use "aaa:bbb:ccc" notation as argument), checking each one for duplicates before adding:
append_path()
{
local SAVED_IFS="$IFS"
local dir
IFS=:
for dir in $1 ; do
if ! $( echo "$PATH" | tr ":" "\n" | grep -qx "$dir" ) ; then
PATH=$PATH:$dir
fi
done
IFS="$SAVED_IFS"
}
It can be called in a script like this:
append_path "/test:$HOME/bin:/example/my dir/space is not an issue"
It has the following advantages:
No bashisms or any shell-specific syntax. It run perfectly with !#/bin/sh (ive tested with dash)
Multiple folders can be added at once
No sorting, preserves folder order
Deals perfectly with spaces in folder names
A single test works no matter if $folder is at begginning, end, middle, or is the only folder in $PATH (thus avoiding testing x:*, *:x, :x:, x, as many of the solutions here implicitly do)
Works (and preserve) if $PATH begins or ends with ":", or has "::" in it (meaning current folder)
No awk or sed needed.
EPA friendly ;) Original IFS value is preserved, and all other variables are local to the function scope.
Hope that helps!
ok, not in csh, but this is how I append $HOME/bin to my path in bash...
case $PATH in
*:$HOME/bin | *:$HOME/bin:* ) ;;
*) export PATH=$PATH:$HOME/bin
esac
season to taste...
you can use the following Perl script to prune paths of duplicates.
#!/usr/bin/perl
#
# ^^ ensure this is pointing to the correct location.
#
# Title: SLimPath
# Author: David "Shoe Lace" Pyke <eselle#users.sourceforge.net >
# : Tim Nelson
# Purpose: To create a slim version of my envirnoment path so as to eliminate
# duplicate entries and ensure that the "." path was last.
# Date Created: April 1st 1999
# Revision History:
# 01/04/99: initial tests.. didn't wok verywell at all
# : retreived path throught '$ENV' call
# 07/04/99: After an email from Tim Nelson <wayland#ne.com.au> got it to
# work.
# : used 'push' to add to array
# : used 'join' to create a delimited string from a list/array.
# 16/02/00: fixed cmd-line options to look/work better
# 25/02/00: made verbosity level-oriented
#
#
use Getopt::Std;
sub printlevel;
$initial_str = "";
$debug_mode = "";
$delim_chr = ":";
$opt_v = 1;
getopts("v:hd:l:e:s:");
OPTS: {
$opt_h && do {
print "\n$0 [-v level] [-d level] [-l delim] ( -e varname | -s strname | -h )";
print "\nWhere:";
print "\n -h This help";
print "\n -d Debug level";
print "\n -l Delimiter (between path vars)";
print "\n -e Specify environment variable (NB: don't include \$ sign)";
print "\n -s String (ie. $0 -s \$PATH:/looser/bin/)";
print "\n -v Verbosity (0 = quiet, 1 = normal, 2 = verbose)";
print "\n";
exit;
};
$opt_d && do {
printlevel 1, "You selected debug level $opt_d\n";
$debug_mode = $opt_d;
};
$opt_l && do {
printlevel 1, "You are going to delimit the string with \"$opt_l\"\n";
$delim_chr = $opt_l;
};
$opt_e && do {
if($opt_s) { die "Cannot specify BOTH env var and string\n"; }
printlevel 1, "Using Environment variable \"$opt_e\"\n";
$initial_str = $ENV{$opt_e};
};
$opt_s && do {
printlevel 1, "Using String \"$opt_s\"\n";
$initial_str = $opt_s;
};
}
if( ($#ARGV != 1) and !$opt_e and !$opt_s){
die "Nothing to work with -- try $0 -h\n";
}
$what = shift #ARGV;
# Split path using the delimiter
#dirs = split(/$delim_chr/, $initial_str);
$dest;
#newpath = ();
LOOP: foreach (#dirs){
# Ensure the directory exists and is a directory
if(! -e ) { printlevel 1, "$_ does not exist\n"; next; }
# If the directory is ., set $dot and go around again
if($_ eq '.') { $dot = 1; next; }
# if ($_ ne `realpath $_`){
# printlevel 2, "$_ becomes ".`realpath $_`."\n";
# }
undef $dest;
#$_=Stdlib::realpath($_,$dest);
# Check for duplicates and dot path
foreach $adir (#newpath) { if($_ eq $adir) {
printlevel 2, "Duplicate: $_\n";
next LOOP;
}}
push #newpath, $_;
}
# Join creates a string from a list/array delimited by the first expression
print join($delim_chr, #newpath) . ($dot ? $delim_chr.".\n" : "\n");
printlevel 1, "Thank you for using $0\n";
exit;
sub printlevel {
my($level, $string) = #_;
if($opt_v >= $level) {
print STDERR $string;
}
}
i hope thats useful.
I've been using the following (Bourne/Korn/POSIX/Bash) script for most of a decade:
: "#(#)$Id: clnpath.sh,v 1.6 1999/06/08 23:34:07 jleffler Exp $"
#
# Print minimal version of $PATH, possibly removing some items
case $# in
0) chop=""; path=${PATH:?};;
1) chop=""; path=$1;;
2) chop=$2; path=$1;;
*) echo "Usage: `basename $0 .sh` [$PATH [remove:list]]" >&2
exit 1;;
esac
# Beware of the quotes in the assignment to chop!
echo "$path" |
${AWK:-awk} -F: '#
BEGIN { # Sort out which path components to omit
chop="'"$chop"'";
if (chop != "") nr = split(chop, remove); else nr = 0;
for (i = 1; i <= nr; i++)
omit[remove[i]] = 1;
}
{
for (i = 1; i <= NF; i++)
{
x=$i;
if (x == "") x = ".";
if (omit[x] == 0 && path[x]++ == 0)
{
output = output pad x;
pad = ":";
}
}
print output;
}'
In Korn shell, I use:
export PATH=$(clnpath /new/bin:/other/bin:$PATH /old/bin:/extra/bin)
This leaves me with PATH containing the new and other bin directories at the front, plus one copy of each directory name in the main path value, except that the old and extra bin directories have bin removed.
You would have to adapt this to C shell (sorry - but I'm a great believer in the truths enunciated at C Shell Programming Considered Harmful). Primarily, you won't have to fiddle with the colon separator, so life is actually easier.
Well, if you don't care what order your paths are in, you could do something like:
set path=(`echo $path | tr ' ' '\n' | sort | uniq | tr '\n' ' '`)
That will sort your paths and remove any extra paths that are the same. If you have . in your path, you may want to remove it with a grep -v and re-add it at the end.
Here is a long one-liner without sorting:
set path = ( echo $path | tr ' ' '\n' | perl -e 'while (<>) { print $_ unless $s{$_}++; }' | tr '\n' ' ')
dr_peper,
I usually prefer to stick to scripting capabilities of the shell I am living in. Makes it more portable. So, I liked your solution using csh scripting. I just extended it to work on per dir in the localdirs to make it work for myself.
foreach dir ( $localdirs )
echo ${path} | egrep -i "$dir" >& /dev/null
if ($status != 0) then
set path = ( $dir $path )
endif
end
Using sed(1) to remove duplicates.
$ PATH=$(echo $PATH | sed -e 's/$/:/;s/^/:/;s/:/::/g;:a;s#\(:[^:]\{1,\}:\)\(.*\)\1#\1\2#g;ta;s/::*/:/g;s/^://;s/:$//;')
This will remove the duplicates after the first instance, which may or may not be what you want, e.g.:
$ NEWPATH=/bin:/usr/bin:/bin:/usr/local/bin:/usr/local/bin:/bin
$ echo $NEWPATH | sed -e 's/$/:/; s/^/:/; s/:/::/g; :a; s#\(:[^:]\{1,\}:\)\(.*\)\1#\1\2#g; t a; s/::*/:/g; s/^://; s/:$//;'
/bin:/usr/bin:/usr/local/bin
$
Enjoy!
Here's what I use - perhaps someone else will find it useful:
#!/bin/csh
# ABSTRACT
# /bin/csh function-like aliases for manipulating environment
# variables containing paths.
#
# BUGS
# - These *MUST* be single line aliases to avoid parsing problems apparently related
# to if-then-else
# - Aliases currently perform tests in inefficient in order to avoid parsing problems
# - Extremely fragile - use bash instead!!
#
# AUTHOR
# J. P. Abelanet - 11/11/10
# Function-like alias to add a path to the front of an environment variable
# containing colon (':') delimited paths, without path duplication
#
# Usage: prepend_path ENVVARIABLE /path/to/prepend
alias prepend_path \
'set arg2="\!:2"; if ($?\!:1 == 0) setenv \!:1 "$arg2"; if ($?\!:1 && $\!:1 !~ {,*:}"$arg2"{:*,}) setenv \!:1 "$arg2":"$\!:1";'
# Function-like alias to add a path to the back of any environment variable
# containing colon (':') delimited paths, without path duplication
#
# Usage: append_path ENVVARIABLE /path/to/append
alias append_path \
'set arg2="\!:2"; if ($?\!:1 == 0) setenv \!:1 "$arg2"; if ($?\!:1 && $\!:1 !~ {,*:}"$arg2"{:*,}) setenv \!:1 "$\!:1":"$arg2";'
When setting path (lowercase, the csh variable) rather than PATH (the environment variable) in csh, you can use set -f and set -l, which will only keep one occurrence of each list element (preferring to keep either the first or last, respectively).
https://nature.berkeley.edu/~casterln/tcsh/Builtin_commands.html#set
So something like this
cat foo.csh # or .tcshrc or whatever:
set -f path = (/bin /usr/bin . ) # initial value
set -f path = ($path /mycode /hercode /usr/bin ) # add things, both new and duplicates
Will not keep extending PATH with duplicates every time you source it:
% source foo.csh
% echo $PATH
% /bin:/usr/bin:.:/mycode:/hercode
% source foo.csh
% echo $PATH
% /bin:/usr/bin:.:/mycode:/hercode
set -f there ensures that only the first occurrence of each PATH element is kept.
I always set my path from scratch in .cshrc.
That is I start off with a basic path, something like:
set path = (. ~/bin /bin /usr/bin /usr/ucb /usr/bin/X11)
(depending on the system).
And then do:
set path = ($otherPath $path)
to add more stuff
I have the same need as the original question.
Building on your previous answers, I have used in Korn/POSIX/Bash:
export PATH=$(perl -e 'print join ":", grep {!$h{$_}++} split ":", "'$otherpath:$PATH\")
I had difficulties to translate it directly in csh (csh escape rules are insane). I have used (as suggested by dr_pepper):
set path = ( `echo $otherpath $path | tr ' ' '\n' | perl -ne 'print $_ unless $h{$_}++' | tr '\n' ' '`)
Do you have ideas to simplify it more (reduce the number of pipes) ?