How to issue Message Before Build--or seq problems - nix

I'm trying to add helpful messages for arbitrary builds. If the build fails the user can, for example, install the package with different arguments.
My interface idea is to provide a function, build-with-message, that would be called with something like this:
build-with-message
''Building ${pkg.name}. Alternative invocations are: ..''
pkg
My implementation is based on builtins.seq
build-with-message = msg : pkg :
seq
(self.runCommand "issue-message" {} ''mkdir $out; echo ${msg}'')
pkg;
When I build a package with build-with-message I never see the message. My hunch is that seq evaluates the runCommand far enough to see that a set is returned and moves on to building the package. I tried with deepSeq as well, but a deepSeq build fails on runCommand. I also tried calling out some attributes from the runCommand, e.g.
(self.runCommand "issue-message" {} ''mkdir $out; echo ${msg}'').drvPath
(self.runCommand "issue-message" {} ''mkdir $out; echo ${msg}'').out
My thought being that calling for one of these would prompt the rest of the build. Perhaps I'm not calling the right attribute, but in any case the ones I've tried don't work.
So:
Is there a way to force the runCommand to build in the above scenario?
Is there already some builtin that just lets me issue messages on top of arbitrary builds?

Here's me answering my own question again, consider this a warning.
Solution:
I've in-lined some numbered comments to help with the explanation.
build-with-message = msg : pkg :
let runMsg /*1*/ = self.runCommand "issue-message"
{ version = toString currentTime; /*2*/ } ''
cat <<EOF
${msg}
EOF
echo 0 > $out /*3*/
'';
in seq (import runMsg /*4*/) pkg; /*5*/
Explanation:
runMsg is the derivation that issues the message.
Adding a version based on the current time ensures that the build of runMsg will not be in /nix/store. Otherwise, each unique message will only be issued for the first build.
After the message is printed, a 0 is saved to file as the output of the derivation.
The import loads runMsg--a derivation, and therefore serialized as the path $out. Import expects a nix expression, which in this case is just the number 0 (a valid nix expression).
Now, since the runMsg output will not be available until after it has been built, the seq command will build it (issuing the message) and then build pkg.
Discussion:
I take note of Robert Hensing's comment to my question--this may not be something Nix was not intended for. I'm not arguing against that. Moving on.
Notice that issuing a message like so will add a file to your nix store for every message issued. I don't know if the message build will be garbage collected while pkg is still installed, so there's the possibility of polluting the nix store if such a pattern is overused.
I also think it's really interesting that the result of the runMsg build was to install a nix expression. I suppose this opens the door to doing useful things.

Related

Finding all Nix build time dependencies, including bootstrapping ones

I'm trying to set up a Nix cache containing all the store paths needed to build a simple derivation. The goal is for this to work on an empty store, with no cache misses so I don't have to hit cache.nixos.org at all. I'm having trouble because Nix seems to download a bunch of extra bootstrapping stuff, apparently to help with fetching.
For example, consider this derivation:
# empty_store_test_simple.nix
with { inherit (import <nixpkgs> {}) fetchFromGitHub; };
(fetchFromGitHub {
owner = "codedownio";
repo = "templates";
rev = "ba68b83d25d2b74f5475521ac00de3bbb884c983";
sha256 = "sha256-LNTi1ZBEsThmGWK53U9Na1j5DKHljcS42/PRXj97p6s=";
})
If I build this on an empty store with --dry-run, I see the following:
λ nix build --impure --store ~/experimental-store --substituters https://cache.nixos.org/ --dry-run -f ./empty_store_test_simple.nix
this derivation will be built:
/nix/store/jr55kq3kk6va95rvjcdyn5jmh059007p-source.drv
these 48 paths will be fetched (24.52 MiB download, 121.41 MiB unpacked):
/nix/store/02bfycjg1607gpcnsg8l13lc45qa8qj3-libssh2-1.10.0
/nix/store/0fi0432kdh46x9kbngnmz2y7z0q68cdz-xz-5.2.5-bin
/nix/store/0rizskpri8d8qawx6qjqcnvlxcvzr1bm-keyutils-1.6.3-lib
/nix/store/1l4r0r4ab3v3a3ppir4jwiah3icalk9d-zlib-1.2.11
/nix/store/1xyz8jwyg9rya2f7gs549c7n2ah378v6-stdenv-linux
/nix/store/3h4a92kysiw3s3rvbsa6a2nys3lf8f8v-libkrb5-1.18
/nix/store/3ibnw61rlgj2lj5hycy2dn3ybpq7wapm-libev-4.33
/nix/store/5qbrz5fimkbywws73vaim8allyh8kjy5-nghttp2-1.43.0-bin
/nix/store/67x6kxbanrqafx3hg7pb3bc83i3d1v3f-gzip-1.11
/nix/store/6irxz4fbf1d1ac7wvdjf8cqb3sgmnvg8-zlib-1.2.11-dev
/nix/store/71pachqc22wlvf3xjhwjh2rqbl6l3ngg-diffutils-3.8
/nix/store/9mp06ni69a44dmrjhn28mn15brdry52w-gnused-4.8
/nix/store/9ppi191zsi7zvynkm8vy2bi22lci9iwg-bzip2-1.0.6.0.2
/nix/store/c9f15p1kwm0mw5p13wsnvd1ixrhbhb12-gcc-10.3.0-lib
/nix/store/d1n274a607fmqdgr7888nq19hdsj7av0-openssl-1.1.1l-bin
/nix/store/d8p27w4d21xs6svkaf3ij60lsw243rn2-openssl-1.1.1l-dev
/nix/store/fdbwa5jrijn0yzwl8l4xdxa0l5daf5j6-curl-7.79.1
/nix/store/fvprxgcxf4px865gdjd81fbwnxcjrg41-coreutils-9.0
/nix/store/gf6j3k1flnhayvpnwnhikkg0s5dxrn1i-openssl-1.1.1l
/nix/store/gmnh4jfjhx83aggwgwzcnrwmpmqr8fwf-gnutar-1.34
/nix/store/gmzhclix3kzhir5jmmwakwhpg6j5zwf1-acl-2.3.1
/nix/store/h0b8ajwz9lvw3a3vqrf41cxrhlx9dz7p-nghttp2-1.43.0-lib
/nix/store/h7srws2r1nalsih91lrm0hfhhar14jzm-libkrb5-1.18-dev
/nix/store/h97sr1q1rpv1ry83031q51jbkba7q0m4-bzip2-1.0.6.0.2-bin
/nix/store/ihscadskdrvwc9dvbirff51lr70cphjj-curl-7.79.1-bin
/nix/store/ikvp5db9hygc14da45lvxi1c9b4ylna9-pcre-8.44
/nix/store/ilszk5f0zcv8lifkixg47ja1f2lsgxkd-nghttp2-1.43.0-dev
/nix/store/k0qa3rjifblr2vrgx4g54a59zxlfhg90-xz-5.2.5
/nix/store/kd14wd2wfmb56zpv5y71yq2lqs11l06k-attr-2.5.1
/nix/store/ksqy6mszsld4z3w8ybxa2vkjf5cqxw3f-c-ares-1.17.2
/nix/store/l0wlqpbsvh1pgvhcdhw7qkka3d31si7k-bash-5.1-p8
/nix/store/lhambyc1v2c7qzzr5sq7p449xs1j6pg8-gnugrep-3.7
/nix/store/lypy3bif096j0qc1divwa87gdvv3r575-curl-7.79.1-dev
/nix/store/p12km8psjlmvbmi52wb9r6gfykqxcdnd-libssh2-1.10.0-dev
/nix/store/pkpynsyxm8c38z4m8ngv52c7v8vhkr2h-unzip-6.0
/nix/store/prq96vz3ywk955nnxlr7s892wf5qvbr0-mirrors-list
/nix/store/psqacrv7k5fxz6mdiawc28sxcdchb4c9-ed-1.17
/nix/store/qbdsd82q5fyr0v31cvfxda0n0h7jh03g-libunistring-0.9.10
/nix/store/qzr7r4w5gm5m20afn2wz4vlv7ah4sr89-gnumake-4.3
/nix/store/r5niwjr8r8qags2bzv9z583r9vajxag3-patchelf-0.13
/nix/store/rnx655nq2qs53yb5arv2gapa91r1wsbn-findutils-4.8.0
/nix/store/scz4zbxirykss3hh5iahgl39wk9wpaps-libidn2-2.3.2
/nix/store/sqn31ly001033hsz0dpxwcsay5qdbk2w-gawk-5.1.1
/nix/store/vslsa0l17xjcrdgm2knwj0z5hlvf73m7-perl-5.34.0
/nix/store/x6pz7c0ffcd6kxzc8m1rflvqmdbjiihh-nghttp2-1.43.0
/nix/store/yj11v0gdjqli4nzax4x48xjnh9y36b2q-curl-7.79.1-man
/nix/store/z56jcx3j1gfyk4sv7g8iaan0ssbdkhz1-glibc-2.33-56
/nix/store/zjm4xv4nr872mdhvv3j22bzb08rgf1hk-patch-2.7.6
However, I can't find all these paths by using the means I would expect:
λ nix repl empty_store_test_simple.nix
nix-repl> :b with import <nixpkgs> {}; closureInfo { rootPaths = [inputDerivation]; }
This derivation produced the following outputs:
out -> /nix/store/1iqm7sr2rr6i5njnfxlqqyzi567mb4cz-closure-info
λ cat /nix/store/1iqm7sr2rr6i5njnfxlqqyzi567mb4cz-closure-info/store-paths
/nix/store/icvdinlsgl6y2kxk9wzkj82a53jgpdlm-source
I would expect to see ~48 paths here, but I only see 1! How can I get all the build-time dependencies indicated by the dry run? I've seen this kind of issue in the past when IFD is present, could there be some going on in Nixpkgs?
If you build your derivation and store it in your own Nix cache that should work; Nix shouldn't need to get the build-time dependencies of that thing, it can just download the result (i.e. the source code you're fetching) from your cache.
If you want to get the build-time dependencies anyway, try:
nix-store -qR $(nix-instantiate test.nix)
I think this will get you one step closer but it's not a complete solution. You'd probably need to build all the derivations in this list or something.

Validate URL in Informix 4GL program

In my Informix 4GL program, I have an input field where the user can insert a URL and the feed is later being sent over to the web via a script.
How can I validate the URL at the time of input, to ensure that it's a live link? Can I make a call and see if I get back any errors?
I4GL checking the URL
There is no built-in function to do that (URLs didn't exist when I4GL was invented, amongst other things).
If you can devise a C method to do that, you can arrange to call that method through the C interface. You'll write the method in native C, and then write an I4GL-callable C interface function using the normal rules. When you build the program with I4GL c-code, you'll link the extra C functions too. If you build the program with I4GL-RDS (p-code), you'll need to build a custom runner with the extra function(s) exposed. All of this is standard technique for I4GL.
In general terms, the C interface code you'll need will look vaguely like this:
#include <fglsys.h>
// Standard interface for I4GL-callable C functions
extern int i4gl_validate_url(int nargs);
// Using obsolescent interface functions
int i4gl_validate_url(int nargs)
{
if (nargs != 1)
fgl_fatal(__FILE__, __LINE__, -1318);
char url[4096];
popstring(url, sizeof(url));
int r = validate_url(url); // Your C function
retint(r);
return 1;
}
You can and should check the manuals but that code, using the 'old style' function names, should compile correctly. The code can be called in I4GL like this:
DEFINE url CHAR(256)
DEFINE rc INTEGER
LET url = "http://www.google.com/"
LET rc = i4gl_validate_url(url)
IF rc != 0 THEN
ERROR "Invalid URL"
ELSE
MESSAGE "URL is OK"
END IF
Or along those general lines. Exactly what values you return depends on your decisions about how to return a status from validate_url(). If need so be, you can return multiple values from the interface function (e.g. error number and text of error message). Etc. This is about the simplest possible design for calling some C code to validate a URL from within an I4GL program.
Modern C interface functions
The function names in the interface library were all changed in the mid-00's, though the old names still exist as macros. The old names were:
popstring(char *buffer, int buflen)
retint(int retval)
fgl_fatal(const char *file, int line, int errnum)
You can find the revised documentation at IBM Informix 4GL v7.50.xC3: Publication library in PDF in the 4GL Reference Manual, and you need Appendix C "Using C with IBM Informix 4GL".
The new names start ibm_lib4gl_:
ibm_libi4gl_popMInt()
ibm_libi4gl_popString()
As to the error reporting function, there is one — it exists — but I don't have access to documentation for it any more. It'll be in the fglsys.h header. It takes an error number as one argument; there's the file name and a line number as the other arguments. And it will, presumably, be ibm_lib4gl_… and there'll be probably be Fatal or perhaps fatal (or maybe Err or err) in the rest of the name.
I4GL running a script that checks the URL
Wouldn't it be easier to write a shell script to get the status code? That might work if I can return the status code or any existing results back to the program into a variable? Can I do that?
Quite possibly. If you want the contents of the URL as a string, though, you'll might end up wanting to call C. It is certainly worth thinking about whether calling a shell script from within I4GL is doable. If so, it will be a lot simpler (RUN "script", IIRC, where the literal string would probably be replaced by a built-up string containing the command and the URL). I believe there are file I/O functions in I4GL now, too, so if you can get the script to write a file (trivial), you can read the data from the file without needing custom C. For a long time, you needed custom C to do that.
I just need to validate the URL before storing it into the database. I was thinking about:
#!/bin/bash
read -p "URL to check: " url
if curl --output /dev/null --silent --head --fail "$url"; then
printf '%s\n' "$url exist"
else
printf '%s\n' "$url does not exist"
fi
but I just need the output instead of /dev/null to be into a variable. I believe the only option is to dump the output into a temp file and read from there.
Instead of having I4GL run the code to validate the URL, have I4GL run a script to validate the URL. Use the exit status of the script and dump the output of curl into /dev/null.
FUNCTION check_url(url)
DEFINE url VARCHAR(255)
DEFINE command_line VARCHAR(255)
DEFINE exit_status INTEGER
LET command_line = "check_url ", url
RUN command_line RETURNING exit_status
RETURN exit_status
END FUNCTION {check_url}
Your calling code can analyze exit_status to see whether it worked. A value of 0 indicates success; non-zero indicates a problem of some sort, which can be deemed 'URL does not work'.
Make sure the check_url script (a) exits with status zero on success and non-zero on any sort of failure, and (b) doesn't write anything to standard output (or standard error) by default. The writing to standard error or output will screw up screen layouts, etc, and you do not want that. (You can obviously have options to the script that enable standard output, or you can invoke the script with options to suppress standard output and standard error, or redirect the outputs to /dev/null; however, when used by the I4GL program, it should be silent.)
Your 'script' (check_url) could be as simple as:
#!/bin/bash
exec curl --output /dev/null --silent --head --fail "${1:-http://www.example.com/"
This passes the first argument to curl, or the non-existent example.com URL if no argument is given, and replaces itself with curl, which generates a zero/non-zero exit status as required. You might add 2>/dev/null to the end of the command line to ensure that error messages are not seen. (Note that it will be hell debugging this if anything goes wrong; make sure you've got provision for debugging.)
The exec is a minor optimization; you could omit it with almost no difference in result. (I could devise a scheme that would probably spot the difference; it involves signalling the curl process, though — kill -9 9999 or similar, where the 9999 is the PID of the curl process — and isn't of practical significance.)
Given that the script is just one line of code that invokes another program, it would be possible to embed all that in the I4GL program. However, having an external shell script (or Perl script, or …) has merits of flexibility; you can edit it to log attempts, for example, without changing the I4GL code at all. One more file to distribute, but better flexibility — keep a separate script, even though it could all be embedded in the I4GL.
As Jonathan said "URLs didn't exist when I4GL was invented, amongst other things". What you will find is that the products that have grown to superceed Informix-4gl such as FourJs Genero will cater for new technologies and other things invented after I4GL.
Using FourJs Genero, the code below will do what you are after using the Informix 4gl syntax you are familiar with
IMPORT com
MAIN
-- Should succeed and display 1
DISPLAY validate_url("http://www.google.com")
DISPLAY validate_url("http://www.4js.com/online_documentation/fjs-fgl-manual-html/index.html#c_fgl_nf.html") -- link to some of the features added to I4GL by Genero
-- Should fail and display 0
DISPLAY validate_url("http://www.google.com/testing")
DISPLAY validate_url("http://www.google2.com")
END MAIN
FUNCTION validate_url(url)
DEFINE url STRING
DEFINE req com.HttpRequest
DEFINE resp com.HttpResponse
-- Returns TRUE if http request to a URL returns 200
TRY
LET req = com.HttpRequest.create(url)
CALL req.doRequest()
LET resp = req.getResponse()
IF resp.getStatusCode() = 200 THEN
RETURN TRUE
END IF
-- May want to handle other HTTP status codes
CATCH
-- May want to capture case if not connected to internet etc
END TRY
RETURN FALSE
END FUNCTION

FAKE Fsc task is writing build products to wrong directory

I'm just learning F#, and setting up a FAKE build harness for a hello-world-like application. (Though the phrase "Hell world" does occasionally come to mind... :-) I'm using a Mac and emacs (generally trying to avoid GUI IDEs by preference).
After a bit of fiddling about with documentation, here's how I'm invoking the F# compiler via FAKE:
let buildDir = #"./build-app/" // Where application build products go
Target "CompileApp" (fun _ -> // Compile application source code
!! #"src/app/**/*.fs" // Look for F# source files
|> Seq.toList // Convert FileIncludes to string list
|> Fsc (fun p -> // which is what the Fsc task wants
{p with //
FscTarget = Exe //
Platform = AnyCpu //
Output = (buildDir + "hello-fsharp.exe") }) // *** Writing to . instead of buildDir?
) //
That uses !! to make a FileIncludes of all the sources in the usual way, then uses Seq.toList to change that to a string list of filenames, which is then handed off to the Fsc task. Simple enough, and it even seems to work:
...
Starting Target: CompileApp (==> SetVersions)
FSC with args:[|"-o"; "./build-app/hello-fsharp.exe"; "--target:exe"; "--platform:anycpu";
"/Users/sgr/Documents/laboratory/hello-fsharp/src/app/hello-fsharp.fs"|]
Finished Target: CompileApp
...
However, despite what the console output above says, the actual build products go to the top-level directory, not the build directory. The message above looks like the -o argument is being passed to the compiler with an appropriate filename, but the executable gets put in . instead of ./build-app/.
So, 2 questions:
Is this a reasonable way to be invoking the F# compiler in a FAKE build harness?
What am I misunderstanding that is causing the build products to go to the wrong place?
This, or a very similar problem, was reported in FAKE issue #521 and seems to have been fixed in FAKE pull request #601, which see.
Explanation of the Problem
As is apparently well-known to everyone but me, the F# compiler as implemented in FSharp.Compiler.Service has a practice of skipping its first argument. See FSharp.Compiler.Service/tests/service/FscTests.fs around line 127, where we see the following nicely informative comment:
// fsc parser skips the first argument by default;
// perhaps this shouldn't happen in library code.
Whether it should or should not happen, it's what does happen. Since the -o came first in the arguments generated by FscHelper, it was dutifully ignored (along with its argument, apparently). Thus the assembly went to the default place, not the place specified.
Solutions
The temporary workaround was to specify --out:destinationFile in the OtherParams field of the FscParams setter in addition to the Output field; the latter is the sacrificial lamb to be ignored while the former gets the job done.
The longer term solution is to fix the arguments generated by FscHelper to have an extra throwaway argument at the front; then these 2 problems will annihilate in a puff of greasy black smoke. (It's kind of balletic in its beauty, when you think about it.) This is exactly what was just merged into the master by #forki23:
// Always prepend "fsc.exe" since fsc compiler skips the first argument
let optsArr = Array.append [|"fsc.exe"|] optsArr
So that solution should be in the newest version of FAKE (3.11.0).
The answers to my 2 questions are thus:
Yes, this appears to be a reasonable way to invoke the F# compiler.
I didn't misunderstand anything; it was just a bug and a fix is in the pipeline.
More to the point: the actual misunderstanding was that I should have checked the FAKE issues and pull requests to see if anybody else had reported this sort of thing, and that's what I'll do next time.

Comparing generated executables for equivilance

I need to compare 2 executables and/or shared objects, compiled using the same compiler/flags and verify that they have not changed. We work in a regulated environment, so it would be really useful for testing purposes to isolate exactly what parts of the executable has changed.
Using MD5Sums/Hashes doesn't work due to the headers containing information about the file.
Does anyone know of a program or way to verify that 2 files are executionally the same even if they were built at a different time?
An interesting question. I have a similar problem on linux. Intrusion detection systems like OSSEC or tripwire may generate false positives if the hashsum of an executable changes all of a sudden. This may be nothing worse than the Linux "prelink" program patching the executable file for faster startups.
In order to compare two binaries (in the ELF format), one can use the "readelf" executable and then "diff" to compare outputs. I'm sure there are refined solutions, but without further ado, a poor man's comparator in Perl:
#!/usr/bin/perl -w
$exe = $ARGV[0];
if (!$exe) {
die "Please give name of executable\n"
}
if (! -f $exe) {
die "Executable $exe not found or not a file\n";
}
if (! (`file '$exe'` =~ /\bELF\b.*?\bexecutable\b/)) {
die "file command says '$exe' is not an ELF executable\n";
}
# Identify sections in ELF
#lines = pipeIt("readelf --wide --section-headers '$exe'");
#sections = ();
for my $line (#lines) {
if ($line =~ /^\s*\[\s*(\d+)\s*\]\s+(\S+)/) {
my $secnum = $1;
my $secnam = $2;
print "Found section $1 named $2\n";
push #sections, $secnam;
}
}
# Dump file header
#lines = pipeIt("readelf --file-header --wide '$exe'");
print #lines;
# Dump all interesting section headers
#lines = pipeIt("readelf --all --wide '$exe'");
print #lines;
# Dump individual sections as hexdump
for my $section (#sections) {
#lines = pipeIt("readelf --hex-dump='$section' --wide '$exe'");
print #lines;
}
sub pipeIt {
my($cmd) = #_;
my $fh;
open ($fh,"$cmd |") or die "Could not open pipe from command '$cmd': $!\n";
my #lines = <$fh>;
close $fh or die "Could not close pipe to command '$cmd': $!\n";
return #lines;
}
Now you can run for example, on machine 1:
./checkexe.pl /usr/bin/curl > curl_machine1
And on machine 2:
./checkexe.pl /usr/bin/curl > curl_machine2
After having copypasted, SFTP-ed or NSF-ed (you don't use FTP, do you?) the files into the same filetree, compare the files:
diff --side-by-side --width=200 curl_machine1 curl_machine2 | less
In my case, differences exist in section ".gnu.conflict", ".gnu.liblist", ".got.plt" and ".dynbss", which might be ok for a "prelink" intervention, but in the code section, ".text", which would be a Bad Sign.
To follow up, here is what I came up with finally:
Instead of comparing the final executables & shared objects, we compared the .o files output before linking. We assumed that the linking process was sufficiently reproducible that this would be fine.
It works in some of our cases, where we have two builds were we've made some small change that shouldn't effect the final code (Code pretty-printer) but doesn't help us if we do not have the build intermediary output.
You can compare the contents of RO and RW initialized sections by generating a binary file from the ELF file.
objcopy <elf_file> -O binary <binary_file>
Use the generated binary files to compare if they are identical, using diff, for example.
In my opinion, this is enough to grantee you are generating the same executable.
A few years back I had to do the same thing. We had to prove that we could rebuild the executable from source when given only a revision number, revision control repository, build tools, and build configuration. Note: If any of these change you may see a difference.
I remember there is some timestamps in the executable. The trick is to realize that the file is not just a bunch of bytes, that can not be interpreted. The file has sections, most will not change, but there will be a section for time of build (or some such thing).
I don't remember all the details, but the commands you will need are { objcopy, objdump, nm }, I think objdump would be the first to try.
Hope this helps.

Capturing output from WshShell.Exec using Windows Script Host

I wrote the following two functions, and call the second ("callAndWait") from JavaScript running inside Windows Script Host. My overall intent is to call one command line program from another. That is, I'm running the initial scripting using cscript, and then trying to run something else (Ant) from that script.
function readAllFromAny(oExec)
{
if (!oExec.StdOut.AtEndOfStream)
return oExec.StdOut.ReadLine();
if (!oExec.StdErr.AtEndOfStream)
return "STDERR: " + oExec.StdErr.ReadLine();
return -1;
}
// Execute a command line function....
function callAndWait(execStr) {
var oExec = WshShell.Exec(execStr);
while (oExec.Status == 0)
{
WScript.Sleep(100);
var output;
while ( (output = readAllFromAny(oExec)) != -1) {
WScript.StdOut.WriteLine(output);
}
}
}
Unfortunately, when I run my program, I don't get immediate feedback about what the called program is doing. Instead, the output seems to come in fits and starts, sometimes waiting until the original program has finished, and sometimes it appears to have deadlocked. What I really want to do is have the spawned process actually share the same StdOut as the calling process, but I don't see a way to do that. Just setting oExec.StdOut = WScript.StdOut doesn't work.
Is there an alternate way to spawn processes that will share the StdOut & StdErr of the launching process? I tried using "WshShell.Run(), but that gives me a "permission denied" error. That's problematic, because I don't want to have to tell my clients to change how their Windows environment is configured just to run my program.
What can I do?
You cannot read from StdErr and StdOut in the script engine in this way, as there is no non-blocking IO as Code Master Bob says. If the called process fills up the buffer (about 4KB) on StdErr while you are attempting to read from StdOut, or vice-versa, then you will deadlock/hang. You will starve while waiting for StdOut and it will block waiting for you to read from StdErr.
The practical solution is to redirect StdErr to StdOut like this:
sCommandLine = """c:\Path\To\prog.exe"" Argument1 argument2"
Dim oExec
Set oExec = WshShell.Exec("CMD /S /C "" " & sCommandLine & " 2>&1 """)
In other words, what gets passed to CreateProcess is this:
CMD /S /C " "c:\Path\To\prog.exe" Argument1 argument2 2>&1 "
This invokes CMD.EXE, which interprets the command line. /S /C invokes a special parsing rule so that the first and last quote are stripped off, and the remainder used as-is and executed by CMD.EXE. So CMD.EXE executes this:
"c:\Path\To\prog.exe" Argument1 argument2 2>&1
The incantation 2>&1 redirects prog.exe's StdErr to StdOut. CMD.EXE will propagate the exit code.
You can now succeed by reading from StdOut and ignoring StdErr.
The downside is that the StdErr and StdOut output get mixed together. As long as they are recognisable you can probably work with this.
Another technique which might help in this situation is to redirect the standard error stream of the command to accompany the standard output.
Do this by adding "%comspec% /c" to the front and "2>&1" to the end of the execStr string.
That is, change the command you run from:
zzz
to:
%comspec% /c zzz 2>&1
The "2>&1" is a redirect instruction which causes the StdErr output (file descriptor 2) to be written to the StdOut stream (file descriptor 1).
You need to include the "%comspec% /c" part because it is the command interpreter which understands about the command line redirect. See http://technet.microsoft.com/en-us/library/ee156605.aspx
Using "%comspec%" instead of "cmd" gives portability to a wider range of Windows versions.
If your command contains quoted string arguments, it may be tricky to get them right:
the specification for how cmd handles quotes after "/c" seems to be incomplete.
With this, your script needs only to read the StdOut stream, and will receive both standard output and standard error.
I used this with "net stop wuauserv", which writes to StdOut on success (if the service is running)
and StdErr on failure (if the service is already stopped).
First, your loop is broken in that it always tries to read from oExec.StdOut first. If there is no actual output then it will hang until there is. You wont see any StdErr output until StdOut.atEndOfStream becomes true (probably when the child terminates). Unfortunately, there is no concept of non-blocking I/O in the script engine. That means calling read and having it return immediately if there is no data in the buffer. Thus there is probably no way to get this loop to work as you want. Second, WShell.Run does not provide any properties or methods to access the standard I/O of the child process. It creates the child in a separate window, totally isolated from the parent except for the return code. However, if all you want is to be able to SEE the output from the child then this might be acceptable. You will also be able to interact with the child (input) but only through the new window (see SendKeys).
As for using ReadAll(), this would be even worse since it collects all the input from the stream before returning so you wouldn't see anything at all until the stream was closed. I have no idea why the example places the ReadAll in a loop which builds a string, a single if (!WScript.StdIn.AtEndOfStream) should be sufficient to avoid exceptions.
Another alternative might be to use the process creation methods in WMI. How standard I/O is handled is not clear and there doesn't appear to be any way to allocate specific streams as StdIn/Out/Err. The only hope would be that the child would inherit these from the parent but that's what you want, isn't it? (This comment based upon an idea and a little bit of research but no actual testing.)
Basically, the scripting system is not designed for complicated interprocess communication/synchronisation.
Note: Tests confirming the above were performed on Windows XP Sp2 using Script version 5.6. Reference to current (5.8) manuals suggests no change.
Yes, the Exec function seems to be broken when it comes to terminal output.
I have been using a similar function function ConsumeStd(e) {WScript.StdOut.Write(e.StdOut.ReadAll());WScript.StdErr.Write(e.StdErr.ReadAll());} that I call in a loop similar to yours. Not sure if checking for EOF and reading line by line is better or worse.
You might have hit the deadlock issue described on this Microsoft Support site.
One suggestion is to always read both from stdout and stderr.
You could change readAllFromAny to:
function readAllFromAny(oExec)
{
var output = "";
if (!oExec.StdOut.AtEndOfStream)
output = output + oExec.StdOut.ReadLine();
if (!oExec.StdErr.AtEndOfStream)
output = output + "STDERR: " + oExec.StdErr.ReadLine();
return output ? output : -1;
}

Resources