I have a situation where disk becomes full and my program hangs because of fflush being used on stdout. I have put down a small code to mimic the problem. We have to redirect this programs stdout to a file in a disk whose size is full already.
while(1){
cout << "a big data to be written here";
int ret = fflush(stdout);
if(ret != 0){
cerr << "get error : " << strerror(errno) << endl;
exit(1);
}
}
And this code hangs forever. I tried to use fcntl with O_NONBLOCK for stdout. Even that doesn`t work somehow. Please note I can not use write system call here though that avoids this kinda hang problem when disk is full. As my system widely uses library calls in many places if I use write system call only in this place it would create output in mixed manner. Can anyone suggest how to avoid hanging ?
I have tried fsync, fdatasync also. Same hanging with those functions too.
Update: fcntl fixed this problem even with cout and fflush combination.
You are mixing C++ stream I/O and C stdio functions. Instead of using fflush(stdout), use cout.flush() if needed. Instead of checking the return code from fflush, you should instead check cout.good() or use cout.rdstate(). I assume the cout operations are failing but fflush is not the part seeing the failure.
Related
This question came up into my mind when I was thinking about ways of server logging yesterday.
Normally, we open a terminal connected to local computer or remote server, run an executable, and print (printf, cout) some debug/log information in the terminal.
But for those processes/executables/scripts running on the server which are not connected to a terminal, what are the standard input and output?
For example:
Suppose I have a crontab task, running a program on the server many times a day. If I write something like cout << "blablabla" << endl; in the program. What's gonna happen? Where those output will flow into?
Another example I came up and wanted to know is, if I write a CGI program (use C or C++) for let's say a Apache web server, what is the standard input and output of my CGI program ? (According to this C++ CGI tutorial, I guess the standard input and output of the CGI program are in some ways redirected to the Apache server. Because it's using cout to output the html contents, not by return. )
I've read this What is “standard input”? before asking, which told me standard input isn't necessary to be tied to keyboard while standard output isn't necessary to be tied to a terminal/console/screen.
OS is Linux.
The standard input and standard output (and standard error) streams can point to basically any I/O device. This is commonly a terminal, but it can also be a file, a pipe, a network socket, a printer, etc. What exactly those streams direct their I/O to is usually determined by the process that launches your process, be that a shell or a daemon like cron or apache, but a process can redirect those streams itself it it would like.
I'll use Linux as an example, but the concepts are similar on most other OSes. On Linux, the standard input and standard output stream are represented by file descriptors 0 and 1. The macros STDIN_FILENO and STDOUT_FILENO are just for convenience and clarity. A file descriptor is just a number that matches up to some file description that the OS kernel maintains that tells it how to write to that device. That means that from a user-space process's perspective, you write to pretty much anything the same way: write(some_file_descriptor, some_string, some_string_length) (higher-level I/O functions like printf or cout are just wrappers around one or more calls to write). To the process, it doesn't matter what type of device some_file_descriptor represents. The OS kernel will figure that out for you and pass your data to the appropriate device driver.
The standard way to launch a new process is to call fork to duplicate the parent process, and then later to call one of the exec family of functions in the child process to start executing some new program. In between, it will often close the standard streams it inherited from its parent and open new ones to redirect the child process's output somewhere new. For instance, to have the child pipe its output back to the parent, you could do something like this in C++:
int main()
{
// create a pipe for the child process to use for its
// standard output stream
int pipefds[2];
pipe(pipefds);
// spawn a child process that's a copy of this process
pid_t pid = fork();
if (pid == 0)
{
// we're now in the child process
// we won't be reading from this pipe, so close its read end
close(pipefds[0]);
// we won't be reading anything
close(STDIN_FILENO);
// close the stdout stream we inherited from our parent
close(STDOUT_FILENO);
// make stdout's file descriptor refer to the write end of our pipe
dup2(pipefds[1], STDOUT_FILENO);
// we don't need the old file descriptor anymore.
// stdout points to this pipe now
close(pipefds[1]);
// replace this process's code with another program
execlp("ls", "ls", nullptr);
} else {
// we're still in the parent process
// we won't be writing to this pipe, so close its write end
close(pipefds[1]);
// now we can read from the pipe that the
// child is using for its standard output stream
std::string read_from_child;
ssize_t count;
constexpr size_t BUF_SIZE = 100;
char buf[BUF_SIZE];
while((count = read(pipefds[0], buf, BUF_SIZE)) > 0) {
std::cout << "Read " << count << " bytes from child process\n";
read_from_child.append(buf, count);
}
std::cout << "Read output from child:\n" << read_from_child << '\n';
return EXIT_SUCCESS;
}
}
Note: I've omitted error handling for clarity
This example creates a child process and redirects its output to a pipe. The program run in the child process (ls) can treat the standard output stream just as it would if it were referencing a terminal (though ls changes some behaviors if it detects its standard output isn't a terminal).
This sort of redirection can also be done from a terminal. When you run a command you can use the redirection operators to tell your shell to redirect that commands standard streams to some other location than the terminal. For instance, here's a convoluted way to copy a file from one machine to another using an sh-like shell:
gzip < some_file | ssh some_server 'zcat > some_file'
This does the following:
create a pipe
run gzip redirecting its standard input stream to read from "some_file" and redirecting its standard output stream to write to the pipe
run ssh and redirect its standard input stream to read from the pipe
on the server, run zcat with its standard input redirected from the data read from the ssh connection and its standard output redirected to write to "some_file"
I want read on stdout to be non-blocking. I was using the newlib-lpc library in WINarm to do this. But even though it is set as non-Blocking, the code stops at read every time and waits for a character to be received.
Here is my read line:
read(fileno(stdout), &inchar, 1);
Here is the code to set stdout to non-blocking using newlib-lpc
blocking=BLOCKING_IO_NO;
ioctl( fileno(stdout), BLOCKING_SETUP, &blocking);
When that didn't work, I decided to forget using the library and do it myself with this line
fcntl(fileno(stdout), F_SETFL,fcntl(fileno(stdout), F_GETFL) |O_NONBLOCK);
But that didn't work either.
Could anyone give me some advice?
Thanks!
edit:
Someone asked if I really meant stdout. Yes, I am resurrecting some old code, and the stdout has been set as the socket for UART communication between my board and the computer.
sp.baud = 115200uL;
sp.length = UART_WORD_LEN_8;
sp.parity = UART_PARITY_NONE;
sp.stop = UART_STOP_BITS_1;
ioctl( fileno(stdout), UART_SETUP, &sp);
irq.FIQ = 0;
irq.pri = (INT_PRIORITY)10;
ioctl( fileno(stdout), INTERRUPT_SETUP, &irq);
edit2: They used stdout this way so that printf works. When it tries to print to stdout, it gets redirected to the UART serial stream.
edit3: The error I get when calling
ioctl( fileno(stdout), BLOCKING_SETUP, &blocking);
is
errno88 function not implemented
In a first phase, i collect a list of constraints. Then, i would like to store this "session", i.e. All the constraints but all the associated variables as well in a file so that I can, in a second phase, read back the constraints and assert them, or even negate some of them before asserting.
What is the best way (fast and reliable) to store such a "session" in a file, and read it back ? Would the Z3_parse_smtlib2_file() API be the right way ? I have tried the Z3_open_log() API, but I don't find the API to read the log file generated by Z3_open_log(). And what about z3_log_replay(). This API does not seem to be exposed yet.
Thanks in advance.
AG
The log file created by Z3_open_log() can be replayed with Z3.exe (stand alone interpreter, not the lib) through the command line option /log myfile. As of today, I haven't seen any API in Z3 library that allows such a replay. For the time being, I have understood that the replay is deemed for debug analysis.
However, you can hack the library (just expose the z3_replayer class in z3_replayer.h) and use it to replay any log file, it is quite easy. The source code of my little feasibility-proof is given below, and is working fine as far as I know. I think it is very nice to be able to do that because sometimes I need to replay a session for debugging purpose. It is good to be able to replay it from a file, rather than from my whole program which is a bit heavy.
Any feedback would be very welcome. Also I would be interested to know whether this functionality could be integrated in the lib, or not.
AG.
#include <fstream>
#include <iostream>
#include "api/z3_replayer.h"
int main(int argc, char * argv[])
{
const char * filename = argv[1];
std::ifstream in(filename);
if (in.bad() || in.fail()) {
std::cerr << "Error: failed to open file: " << filename << "\n";
exit(EXIT_FAILURE);
}
z3_replayer r(in);
r.parse();
Z3_context ctx = reinterpret_cast<Z3_context>(r.get_obj(0));
check(ctx,Z3_L_TRUE); // this function is taken from the c examples
return 0;
}
This is probably not of major importance, however I have noticed during testing that the performance of the print statement and also stdout is much faster in the Dart-Editor than from the command-line. From the command-line the performance of print takes around 36% longer than using stdout from the command-line. However, running the program from within the editor, using stdout takes around 900% longer than using the print statement in the editor, but both are considerably faster than from the command-line. ie. Print from a program running in the editor takes around 2.65% of the time it takes from the command-line.
Some relative timings based on average performance from my test :
Running program from command line (5000 iterations) :
print 1700 milliseconds.
stdout 1245 milliseconds.
Running program within Dart-Editor (5000 iterations) :
print 45 milliseconds
stdout 447 milliseconds.
Can someone explain to me the reason for these differences – in particular why performance in the Dart-Editor is so much faster? Also, is it acceptable practice to use stdout and what are the pros and cons versus using print?
Why is the Dart Editor faster?
Because the output handling by the command line is just really slow, and this blocks the output stream, and subsequently the call to print/stdout.
You can test this for yourself - test the following java program (with your own paths, of course):
public static void main(String[] args) {
try {
// the dart file does print and stdout in a loop
Process p = Runtime.getRuntime().exec("C:\\eclipse\\dart-sdk\\bin\\dart.exe D:\\DEVELOP\\Dart\\Console_Playground\\bin\\console_playground.dart");
BufferedReader in = new BufferedReader(new InputStreamReader(p.getInputStream()));
StringBuffer buf = new StringBuffer();
String line;
while((line = in.readLine()) != null) {
buf.append(line + "\r\n");
}
System.out.print(buf.toString());
} catch (IOException e) {
e.printStackTrace();
}
}
On my machine, this is even slightly faster than the Dart Editor (which probably does something like buffering the input and rendering it periodically, but I don't really know).
You will also see that adding a Thread.sleep(1); into the loop will severely impact the performance of the dart program, because the stream is blocked.
Should stdout be used?
I think that's highly subjective. I, for one, do whatever lets me write code more quickly. When i just want to dump a variable, i use print(myvar);. But with stdout, you can do neat stuff like this: stdout.addStream(new File(r"D:\test.csv").openRead());. Of course, if performance is an issue, it depends on how your application will be used - for example, called by another program (where print is faster) vs. command line (where stdout is faster, for some reason).
Why is stdout faster in command line?
I have no idea, sorry. It's the only environment I tested where print() is slower, so I'd guess it has something to do with how the console handles incoming data.
I wrote the following two functions, and call the second ("callAndWait") from JavaScript running inside Windows Script Host. My overall intent is to call one command line program from another. That is, I'm running the initial scripting using cscript, and then trying to run something else (Ant) from that script.
function readAllFromAny(oExec)
{
if (!oExec.StdOut.AtEndOfStream)
return oExec.StdOut.ReadLine();
if (!oExec.StdErr.AtEndOfStream)
return "STDERR: " + oExec.StdErr.ReadLine();
return -1;
}
// Execute a command line function....
function callAndWait(execStr) {
var oExec = WshShell.Exec(execStr);
while (oExec.Status == 0)
{
WScript.Sleep(100);
var output;
while ( (output = readAllFromAny(oExec)) != -1) {
WScript.StdOut.WriteLine(output);
}
}
}
Unfortunately, when I run my program, I don't get immediate feedback about what the called program is doing. Instead, the output seems to come in fits and starts, sometimes waiting until the original program has finished, and sometimes it appears to have deadlocked. What I really want to do is have the spawned process actually share the same StdOut as the calling process, but I don't see a way to do that. Just setting oExec.StdOut = WScript.StdOut doesn't work.
Is there an alternate way to spawn processes that will share the StdOut & StdErr of the launching process? I tried using "WshShell.Run(), but that gives me a "permission denied" error. That's problematic, because I don't want to have to tell my clients to change how their Windows environment is configured just to run my program.
What can I do?
You cannot read from StdErr and StdOut in the script engine in this way, as there is no non-blocking IO as Code Master Bob says. If the called process fills up the buffer (about 4KB) on StdErr while you are attempting to read from StdOut, or vice-versa, then you will deadlock/hang. You will starve while waiting for StdOut and it will block waiting for you to read from StdErr.
The practical solution is to redirect StdErr to StdOut like this:
sCommandLine = """c:\Path\To\prog.exe"" Argument1 argument2"
Dim oExec
Set oExec = WshShell.Exec("CMD /S /C "" " & sCommandLine & " 2>&1 """)
In other words, what gets passed to CreateProcess is this:
CMD /S /C " "c:\Path\To\prog.exe" Argument1 argument2 2>&1 "
This invokes CMD.EXE, which interprets the command line. /S /C invokes a special parsing rule so that the first and last quote are stripped off, and the remainder used as-is and executed by CMD.EXE. So CMD.EXE executes this:
"c:\Path\To\prog.exe" Argument1 argument2 2>&1
The incantation 2>&1 redirects prog.exe's StdErr to StdOut. CMD.EXE will propagate the exit code.
You can now succeed by reading from StdOut and ignoring StdErr.
The downside is that the StdErr and StdOut output get mixed together. As long as they are recognisable you can probably work with this.
Another technique which might help in this situation is to redirect the standard error stream of the command to accompany the standard output.
Do this by adding "%comspec% /c" to the front and "2>&1" to the end of the execStr string.
That is, change the command you run from:
zzz
to:
%comspec% /c zzz 2>&1
The "2>&1" is a redirect instruction which causes the StdErr output (file descriptor 2) to be written to the StdOut stream (file descriptor 1).
You need to include the "%comspec% /c" part because it is the command interpreter which understands about the command line redirect. See http://technet.microsoft.com/en-us/library/ee156605.aspx
Using "%comspec%" instead of "cmd" gives portability to a wider range of Windows versions.
If your command contains quoted string arguments, it may be tricky to get them right:
the specification for how cmd handles quotes after "/c" seems to be incomplete.
With this, your script needs only to read the StdOut stream, and will receive both standard output and standard error.
I used this with "net stop wuauserv", which writes to StdOut on success (if the service is running)
and StdErr on failure (if the service is already stopped).
First, your loop is broken in that it always tries to read from oExec.StdOut first. If there is no actual output then it will hang until there is. You wont see any StdErr output until StdOut.atEndOfStream becomes true (probably when the child terminates). Unfortunately, there is no concept of non-blocking I/O in the script engine. That means calling read and having it return immediately if there is no data in the buffer. Thus there is probably no way to get this loop to work as you want. Second, WShell.Run does not provide any properties or methods to access the standard I/O of the child process. It creates the child in a separate window, totally isolated from the parent except for the return code. However, if all you want is to be able to SEE the output from the child then this might be acceptable. You will also be able to interact with the child (input) but only through the new window (see SendKeys).
As for using ReadAll(), this would be even worse since it collects all the input from the stream before returning so you wouldn't see anything at all until the stream was closed. I have no idea why the example places the ReadAll in a loop which builds a string, a single if (!WScript.StdIn.AtEndOfStream) should be sufficient to avoid exceptions.
Another alternative might be to use the process creation methods in WMI. How standard I/O is handled is not clear and there doesn't appear to be any way to allocate specific streams as StdIn/Out/Err. The only hope would be that the child would inherit these from the parent but that's what you want, isn't it? (This comment based upon an idea and a little bit of research but no actual testing.)
Basically, the scripting system is not designed for complicated interprocess communication/synchronisation.
Note: Tests confirming the above were performed on Windows XP Sp2 using Script version 5.6. Reference to current (5.8) manuals suggests no change.
Yes, the Exec function seems to be broken when it comes to terminal output.
I have been using a similar function function ConsumeStd(e) {WScript.StdOut.Write(e.StdOut.ReadAll());WScript.StdErr.Write(e.StdErr.ReadAll());} that I call in a loop similar to yours. Not sure if checking for EOF and reading line by line is better or worse.
You might have hit the deadlock issue described on this Microsoft Support site.
One suggestion is to always read both from stdout and stderr.
You could change readAllFromAny to:
function readAllFromAny(oExec)
{
var output = "";
if (!oExec.StdOut.AtEndOfStream)
output = output + oExec.StdOut.ReadLine();
if (!oExec.StdErr.AtEndOfStream)
output = output + "STDERR: " + oExec.StdErr.ReadLine();
return output ? output : -1;
}