I want to parallelize an image processing step which uses two programs at the same time. My code works fine for a single image but when I try to parallelize it, it fails.
The two programs I am using are fx and getkey from USGS Integrated Software for Imagers and Spectrometers. I use fx to perform an arithmetic operation on my input image (which is 'f1' in the code below) and writes it to a new file (which is the 'to' parameter). getkey outputs the value of a requested keyword, which is a number in this case.
In the following code, I am subtracting the output of getkey from my input image, f1, and writing the result to a new file, which is defined by the 'to' parameter. This code works as I expect it to:
fx f1=W1660432760_1_overclocks_average_lwps5.cub to=testing_fx2.cub equation=f1-$(getkey from=W1660432760_1_overclocks_average_lwps5_stats.txt grpname=results keyword=average)
The problem comes when I try to parallelize it. The following code gives an error, saying 'Unable to open [{2}].'
parallel fx f1={1} to={1.}_minus_avg.cub equation=f1-$(getkey from={2} grpname=results keyword=average) ::: $(find *lwps5.cub) ::: $(find *stats.txt)
The result I am expecting is an output image with pixel values that are smaller by the getkey value compared to the input image.
If the two inputs should be combined in all ways:
parallel fx f1={1} to={1.}_minus_avg.cub 'equation=f1-$(getkey from={2} grpname=results keyword=average)' ::: *lwps5.cub ::: *stats.txt
If the two inputs should be linked:
parallel fx f1={1} to={1.}_minus_avg.cub 'equation=f1-$(getkey from={2} grpname=results keyword=average)' ::: *lwps5.cub :::+ *stats.txt
If neither of these solve you issue, then make a shell function that takes 2 arguments:
doit() {
arg1="$1"
arg2="$2"
# Do all your stuff with getkey and fx
}
export -f doit
# all combinations
parallel doit ::: *lwps5.cub ::: *stats.txt
# or linked
parallel doit ::: *lwps5.cub :::+ *stats.txt
Related
I have more than 50 file pairs with names in the following format: AA-7R-76L1.clean.R1.fastq.gz, AA-7R-76L1.clean.R2.fastq.gz
I tried to use parallel in the following way:
parallel --plus echo {%R..fastq.gz} ::: *.fastq.gz |parallel 'repair.sh in1={}.R1.fastq.gz in2={}.R2.fastq.gz out1={}.repd.R1.fastq.gz out2={}.repd.R2.fastq.gz outs={}.singletons.fastq.gz repair'
--plus echo should dynamically replace R1.fastq.gz, R2.fastq.gz to capture the sample name i.e.HB-7R-25L0.clean. It should then feed it to repair.sh
The error I get is, the first section extracts the entire filename and does not capture the sample name. Thus in1 and in2 becomes AA-7R-76L1.clean.R1.fastq.gz.R1.fastq.gz and AA-7R-76L1.clean.R2.fastq.gz.R2.fastq.gz
What is the error here?
Something like:
$ parallel --plus --dry-run 'repair.sh in1={} in2={/R1/R2} out1={/R1/fixed.R1} out2={/R1/fixed.R2} outs={%.R1.fastq.gz}_singletons.fastq repair' ::: *R1.fastq.gz
(Assuming R1 and R2 is not part of the *-part of the name).
I understand that Julia has a complete set of low level tools for interfacing with binary files on one hand and some powerfull utilities such as readdlm to load text files containing rectangular data into Array structures on the other hand.
What I cannot discover in the standard library docs, however, is how to easily get input from less structured text files. In particular, what would be the Julia equivalent of the c++ idiom
some_input_stream >> a_variable_int_perhaps;
Given this is such a common usage scenario I am surprised something like this does not feature prominently in the standard library...
You can use readuntil http://docs.julialang.org/en/latest/stdlib/io-network/#Base.readuntil
shell> cat test.txt
1 2 3 4
julia> i,j = open("test.txt") do f
parse(Int, readuntil(f," ")), parse(Int, readuntil(f," "))
end
(1,2)
EDIT: To address comments
To get the last integer in an irregularly formatted ascii file you could use split if you know the character preceding the integer (I've use a blank space here)
shell> cat test.txt
1.0, two five:$#!() + 4
last line 3
julia> i = open("test.txt") do f
parse(Int, split(readline(f), " ")[end])
end
4
As far as code length is concerned, the above examples are completely self contained and the file is opened and closed in an exception safe manner (i.e. wrapped in a try-finally block). To do the same in C++ would be quite verbose.
I want to make use of Maxima as the backend to solve some computations used in my LaTeX input file.
I did the following steps.
Step 1
Download and install Maxima.
Step 2
Create a batch file named cas.bat (for example) as follows.
rem cas.bat
echo off
set PATH=%PATH%;"C:\Program Files (x86)\Maxima-5.31.2\bin"
maxima --very-quiet -r %1 > solution.tex
Save the batch in the same directory in which your input file below exists. It is just for the sake of simplicity.
Step 3
Create the input file named main.tex (for example) as follows.
% main.tex
\documentclass[preview,border=12pt,12pt]{standalone}
\usepackage{amsmath}
\def\f(#1){(#1)^2-5*(#1)+6}
\begin{document}
\section{Problem}
Evaluate $\f(x)$ for $x=\frac 1 2$.
\section{Solution}
\immediate\write18{cas "x: 1/2;tex(\f(x));"}
\input{solution}
\end{document}
Step 4
Compile the input file with pdflatex -shell-escape main and you will get a nice output as follows.
!
Step 5
Done.
Questions
Apparently the output of Maxima is as follows. I don't know how to make it cleaner.
solution.tex
1
-
2
$${{15}\over{4}}$$
false
Now, my question are
how to remove such texts?
how to obtain just \frac{15}{4} without $$...$$?
(1) To suppress output, terminate input expressions with dollar sign (i.e. $) instead of semicolon (i.e. ;).
(2) To get just the TeX-ified expression sans the environment delimiters (i.e. $$), call tex1 instead of tex. Note that tex1 returns a string, which you have to print yourself (while tex prints it for you).
Combining these ideas with the stuff you showed, I think your program could look like this:
"x: 1/2$ print(tex1(\f(x)))$"
I think you might find the Maxima mailing list helpful. I'm pretty sure there have been several attempts to create a system such as the one you describe. You can also look at the documentation.
I couldn't find any way to completely clean up Maxima's output within Maxima itself. It always echoes the input line, and always writes some whitespace after the output. The following is an example of a perl script that accomplishes the cleanup.
#!/usr/bin/perl
use strict;
my $var = $ARGV[0];
my $expr = $ARGV[1];
sub do_maxima_to_tex {
my $m = shift;
my $c = "maxima --batch-string='exptdispflag:false; print(tex1($m))\$'";
my $e = `$c`;
my #x = split(/\(%i\d+\)/,$e); # output contains stuff like (%i1)
my $f = pop #x; # remove everything before the echo of the last input
while ($f=~/\A /) {$f=~s/\A .*\n//} # remove echo of input, which may be more than one line
$f =~ s/\\\n//g; # maxima breaks latex tokens in the middle at end of line; fix this
$f =~ s/\n/ /g; # if multiple lines, get it into one line
$f =~ s/\s+\Z//; # get rid of final whitespace
return $f;
}
my $e1 = do_maxima_to_tex("diff($expr,$var,1)");
my $e2 = do_maxima_to_tex("diff($expr,$var,2)");
print <<TEX;
The first derivative is \$$e1\$. Differentiating a second time,
we get \$$e2\$.
TEX
If you name this script a.pl, then doing
a.pl z 3*z^4
outputs this:
The first derivative is $12\,z^3$. Differentiating a second time,
we get $36\,z^2$.
For the OP's application, a script like this one could be what is invoked by the write18 in the latex file.
If you really want to use LaTeX then the maxiplot package is the answer. It provides a maxima environment inside of which you enter Maxima commands. When you process your LaTeX file a Maxima batch file is generated. Process this file with Maxima and process your LaTeX file again to typeset the equations generated by Maxima.
If you would rather have 2D math input with live typesetting then use TeXmacs. It is a cross-platform document authoring environment (a word processor on steroids if you like) that includes plugins for Maxima, Mathematica and many more scientific computing tools. If you need to or are not satisfied with the typesetting, you can export your document to LaTeX.
I know this is a very old post. Excellent answers for the question asked by OP. I was using --very-quiet -r options on the command line for a long time like OP, but in maxima version 5.43.2 they behave differently. See maxima command line v5.43 is behaving differently than v5.41. I am answering this question with a cross reference because when incorporating these answers in your solutions, make sure the changes in behavior of those command line flags are also incorporated.
Here are 4 sample rows of the text file of interest
EnumerateKey,explorer.exe,HKCR\\Directory\\shellex\\ContextMenuHandlers,NOMORE
CreateSec,explorer.exe,\\WINDOWS\\system32\\verclsid.exe,SUCCESS
QueryKey,AcroRd32.exe,HKCU\\Control Panel\\International,BUFOVRFLOW
QueryValue,AcroRd32.exe,HKCU\\Software\\Microsoft\\Windows\\CurrentVersion\\Policies\\Explorer\\NoRecentDocsHistory,NOTFOUND
I would like to augment the rows by appending K fields/columns (for example K=3 below) which contain the elements of the path found in $3 but parsed by \\
Here are the desired output for the 4 lines.
EnumerateKey,explorer.exe,HKCR\\Directory\\shellex\\ContextMenuHandlers,NOMORE, Directory, shellex, ContextMenuHandlers
CreateSec,explorer.exe,\\WINDOWS\\system32\\verclsid.exe,SUCCESS, WINDOWS, system32, verclsid.exe
QueryKey,AcroRd32.exe,HKCU\\Control Panel\\International,BUFOVRFLOW, Control Panel, International,
QueryValue,AcroRd32.exe,HKCU\\Software\\Microsoft\\Windows\\CurrentVersion\\Policies\\Explorer\\NoRecentDocsHistory,NOTFOUND, Software, Microsoft, Windows
After some more study, here are 2 nuances:
Some of the paths begin with HK**, others don't. However, in both cases I only care about the path that starts after the initial \\. This difference is captured between line 1 and 2. Therefore I believe the parsing must be anchored at \\ rather than simply $3 if possible. (Am I using that terminology correctly?)
Second, the depth of the path varies. In order to keep consistency in column/fields I'm willing to lose some information (line 4) as well as have empty fields for the short paths (line 3) in order to maintain this.
Here's one way using GNU awk:
awk 'BEGIN { FS=OFS="," } { split($3,a,"\\\\\\\\"); print $0, a[2], a[3], a[4] }' file
Results:
EnumerateKey,explorer.exe,HKCR\\Directory\\shellex\\ContextMenuHandlers,NOMORE,Directory,shellex,ContextMenuHandlers
CreateSec,explorer.exe,\\WINDOWS\\system32\\verclsid.exe,SUCCESS,WINDOWS,system32,verclsid.exe
QueryKey,AcroRd32.exe,HKCU\\Control Panel\\International,BUFOVRFLOW,Control Panel,International,
QueryValue,AcroRd32.exe,HKCU\\Software\\Microsoft\\Windows\\CurrentVersion\\Policies\\Explorer\\NoRecentDocsHistory,NOTFOUND,Software,Microsoft,Windows
With some ugly Perl:
perl -lane '{$l=$_;s/.*?\\\\([^,]*),.*/$1/;#v=split(/\\\\/,$_); print "$l,".join(",",#v[0,1,2]);}' input
I invoked Maxima tex1 from within a batch script as follows:
maxima --very-quiet -r "tex1(solve(8*x^2+7*x+5));" | grep -v false > output.txt
and I got the output.txt as follows:
\left[ x=-{{\sqrt{111}\,i+7}\over{16}} , x={{\sqrt{111}\,i-7}\over{16}} \righ\
t]
that is not valid as a (La)TeX input file.
How to prevent Maxima tex1 from wrapping its output?
Sorry for the late reply.
Instead of
tex1(solve(8*x^2+7*x+5));
write:
?princ(tex1(solve(8*x^2+7*x+5)))$
The problem is that the string returned by tex1 is being printed by the display formatter (the same function which would print the string if you were using Maxima in an interactive session). The display formatter breaks strings at linel characters (default = 79) and inserts a backslash. Instead for your purposes you want to evade the display formatter entirely, so you print the string with ?princ (a Lisp function to just print the string) and terminate the input with "$" instead of ";" to tell Maxima not to call the display formatter.
Note that the hard-coded constant 70 in MYPRINC doesn't come into play here. MYPRINC is not called in the example given.
This is, unfortunately, hard coded into Maxima. A way to solve this problem is to edit the function myprinc located in the file maxima/src/mactex.lisp. There is a cond form that has a 70. written there, it should read linel instead of 70. If you recompile maxima after making this change then the following will work:
maxima --very-quiet -r "linel: 1000$ tex1(solve(8*x^2+7*x+5));" | grep -v false > output.txt
Anyway, I'll send a patch to the Maxima list ASAP so that future versions of the program won't have this shortcoming.