Redirecting standard I/O from within a program

You've come to this page because you've asked a question similar to

I know how one can use, for example, the redirection syntax in the 32-bit Command Interpreter to redirect a command's standard I/O before it is run. But how do I redirect the standard input/standard output/standard error of my program from within the program itself as it is running? Indeed, how do command interpreters themselves go about doing this before they invoke commands?

This is the Frequently Given Answer to that question.

APIs

There are three APIs at which the notion of "standard I/O" applies. How one redirects standard I/O depends, in part, from which levels one wishes to affect — what APIs one's program is using in order to perform I/O in the first place. These APIs are:

Redirecting C streams

The Standard C library provides a function that one can use to cause the stdin, stdout, and stderr C streams to refer to other files: freopen(). It is used like this example of redirecting stdout:

FILE * f = freopen("newfile", "w", stdout) ;

For an explanation of the various parameters, see § 7.19.5.4 of the C standard, or your C/C++ implementation's Standard C library documentation.

The disadvantages to using freopen() on its own are severalfold:

These problems are addressed by redirecting at a lower layer, that the C library's I/O streams are layered on top of. The difficulty is that what they are layered on top of varies from implementation to implementation. On some implementations, they are layered on top of the POSIX API. This is (of course) the case for almost all C and C++ implementations for Unix and Linux systems. On other implementations, however, they are layered directly over the system API (when the POSIX API is not, in fact, the system API). This is the case for some C and C++ implementations for Win32, for example.

If one redirects at a lower layer, then one doesn't use freopen(). It's wholly unnecessary, and indeed it's counterproductive in that it negates the effect of the redirection at the lower layer. What one generally has to use are implementation-specific functions that allow one to tweak the linkage between the Standard C library's streams and the I/O API that they are layered on top of — more on which later — during or after performing the redirection at the lower layer.

Microsoft KnowledgeBase article 58667 states the falsehood that redirection using freopen() is reversible, by calling freopen() again. It is not. The Microsoft article is written based upon the overly simplistic notion that all programs always initially have their standard inputs, outputs, and errors attached to a console. They of course do not. It is also written based upon the outright false notion that opening a stream afresh yields the same stream as the original open stream. It of course does not — especially so in the cases of files (which could have been deleted in the interim, and at the very least will have different file pointers), sockets, and pipes. Reversibility requires some Standard C streams equivalent to the POSIX dup2() API function, to allow a stream to be saved to a copy and restored. Standard C provides no such thing.

Redirecting POSIX file descriptors

The POSIX API provides a function that one can use to cause file descriptors 0, 1, and 2 (standard input, standard output, and standard error) to refer to other files: dup2(). It is used like this example of redirecting standard error:

int fd = open(…) ; // … or a call to pipe() or socket()
if (0 > fd) … // Handle errors.
int r1 = dup2(fd, 2) ;
int r2 = close(fd) ;

For an explanation of the various parameters, see the POSIX IEEE 1003.1:2004 definition of dup2(), or your C/C++ implementation's POSIX API library documentation.

Unlike the case with freopen(), this mechanism is reversible, should one want to reverse it. One simply calls dup() to obtain a copy of the original file descriptor before performing the redirection. To reverse the redirection, one simply calls dup2() to copy the saved file descriptor back to the original standard file descriptor (and then of course close() to close the now-superfluous copy).

The POSIX API on Unix and its imitators

On operating systems such as Unix, Linux, GNU Hurd, and so forth, where the POSIX API is the system API, or at least is the lowest level API that one can call without breaking the interface contract between applications softwares and the operating system, this does everything that one needs to do. This is what Unix shells do in order to redirect the standard inputs/outputs/errors of processes that they fork().

There is nothing that one needs to do further in order to have the Standard C library C streams redirected. stdin, stdout, and stderr are at C library initialization associated with the POSIX file descriptors 0, 1, and 2, and any I/O through the C streams will just be directed to wherever those file descriptors actually lead.

There is only the one caveat that one need be aware of:

There are several things that you'll find people doing, as a result of cargo-cult programming practices handed on from programmer to programmer, that are in fact unnecessary:

One doesn't need as many eggs as that in the pudding. The procedure is simple, and is as outlined afore.

The POSIX API on OS/2 and PC/MS/DR-DOS

On operating systems such as OS/2 and PC/MS/DR-DOS, whilst the POSIX API is not the system API, the POSIX API layer in the implementation's library is thin. The operating system's file handles have the same semantics as POSIX file descriptors in the relevant areas:

As a consequence, the mapping from the one to the other is usually a direct one, with a file descriptor in the POSIX API being turned directly into a file handle at the system API. And calling dup2() will end up calling DosDupHandle() (OS/2) or invoking INT 21h/AX=45h (PC/MS/DR-DOS).

As such, the same is true for OS/2 and PC/MS/DR-DOS as for Unix and its imitators when it comes to redirecting I/O via the POSIX API. One just calls dup2() and closes the new file descriptor, as aforegiven, with the same simple caveat relating to buffering in the C streams.

And even though the POSIX API is an artefact of the implementation's library on such systems, because it's such a thin layer over the system API one does not have the worries about mixing code using multiple runtime libraries in a single process, or about handles not being inherited by spawned child processes.

The POSIX API on Win32

Things are very different with the POSIX API when it is layered on top of Win32. Win32 file handles do not support the semantics needed by the POSIX API for file descriptors:

To support the POSIX API, therefore, a Win32 C/C++ library has to provide a complex shim layer, that for starters maps POSIX file descriptor numbers onto Win32 file handles through a table maintained privately within the library. It is, in contrast to the situation for Unix, OS/2, and PC/MS/DR-DOS, a comparatively thick layer on top of the underlying Win32 API.

As a consequence, and for much the same reasons, redirecting POSIX file descriptors in a Win32 C or C++ implementation suffers from much the same problems as redirecting the Standard C streams using freopen():

It suffers from specific additional problems of its own:

As with the case of the Standard C streams, these problems are addressed by redirecting at a lower layer, that the POSIX API's file descriptors is layered on top of, namely the Win32 API itself.

And as with the Standard C streams, what one generally has to use are implementation-specific functions that allow one to tweak the linkage between the POSIX API's file descriptors and the Win32 API that they are layered on top of — more on which later — during or after performing the redirection at the Win32 layer.

Redirecting System API file handles

In the cases of OS/2 and PC/MS/DR-DOS, redirection at the system API is effectively achieved by redirection at the POSIX API. System API file handles are directly interchangeable with POSIX API file descriptors, and one can call either dup2() or the equivalent underlying system API file handle duplication mechanism with equal effect.

In the case of Unix and its imitators, the system API is the POSIX API, and redirection at the system API is, again, effectively covered under redirection at the POSIX API.

This leaves Win32 as the sole case where redirection at the system API is a distinct case to be covered in its own right.

The Win32 System API

If one wants to avoid the aforementioned problems with freopen() and dup2() on Win32 C and C++ implementations, one has to redirect standard I/O at the Win32 system API level, also using implementation-specific functions that allow one to tweak the linkage between the Win32 API and the POSIX API and C streams that are layered on top of it.

Win32 has no concept of standard file handles. On Unix and its imitators, one can hardwire the standard file descriptor numbers 0, 1, and 2 (or the STDIN_FILENO, STDOUT_FILENO, and STDERR_FILENO macros from the <unistd.h> header) into an application. On OS/2 and PC/MS/DR-DOS, one can hardwire the file handle numbers 0, 1, and 2 into an application. There is no equivalent to this on Win32. There are no standard file handle numbers that one can hardwire into applications.

Instead, Win32 has what amounts to a small three-entry table of file handles, initialized by a process' parent in the call to CreateProcess() and queriable and modifiable within the process itself via GetStdHandle() and SetStdHandle(). Essentially, Win32 provides a process with a way of saying "Here are three file handle numbers, if anyone asks.".

Ironically, had this mechanism supported more than three handles, it would have made life a lot easier for C and C++ implementations wanting to provide a POSIX API over Win32. It would have provided a process-wide, compiler-neutral, language-neutral, inheritable from parent process to child process, mechanism for mapping from POSIX file descriptors to Win32 file handles. But since the Win32 table only supports three entries, C and C++ libraries have to roll their own tables, which are compiler-specific, not process-wide (when more than one runtime library is used within a process), and not easily inheritable from parent processes to child processes. Updating these tables involves calling C library functions that are not standardized, and that vary from implementation to implementation.

So when redirecting Win32 file handles, one has to separately and additionally inform the C library of the new Win32 file handles, using its implementation-specific mechanisms for modifying its internal POSIX file descriptor→Win32 file handle mapping table, and (if it is an implementation that doesn't layer the POSIX API between C streams and Win32) also modifying what Win32 file handles its C stream objects use.

With OpenWatcom C/C++, for example, this involves the implementation's _open_osfhandle() function:

HANDLE FileHandle = CreateFile(…) ;
BOOL r = SetStdHandle(STD_OUTPUT_HANDLE, FileHandle) ;
if (!r) … // Handle errors.
int fd = _open_osfhandle(FileHandle, O_WRONLY|O_TEXT) ;
if (0 > fd) … // Handle errors.
int r1 = dup2(fd, 1) ;
int r2 = close(fd) ;

The call to _open_osfhandle() returns a new POSIX file descriptor, with which one proceeds to redirect the POSIX standard file descriptors in the aforegiven fashion for POSIX, just as if one had obtained a new file descriptor using the POSIX open(), pipe(), or socket() API functions.

After the call to SetStdHandle(), any code in the process calling GetStdHandle(STD_OUTPUT_HANDLE) will obtain the new Win32 file handle to use. After the call to dup2(), any code in the process using the POSIX API with standard file descriptor number 1 or using the stdout C stream (since OpenWatcom layers the C streams on top of the POSIX API) will automatically use the new Win32 file handle.

This mechanism is, like redirecting POSIX file descriptors, reversible, should one want to reverse it. One simply calls GetStdHandle(STD_OUTPUT_HANDLE) to obtain the original file handle and dup() to obtain a copy of the original file descriptor, before performing the redirection. To reverse the redirection, one simply:

  1. calls SetStdHandle(STD_OUTPUT_HANDLE, …) to copy the saved Win32 file handle back to the Win32 process-wide mapping table;

  2. calls dup2() to copy the saved file descriptor back to the original standard file descriptor;

  3. calls close() to close the now-superfluous saved file descriptor.

Again, the fact that Win32 does not provide a general-purpose mechanism with GetStdHandle() and SetStdHandle(), that can handle an arbitrary number of POSIX file descriptor→Win32 file handle mappings, is the underlying cause of the one problem that remains with redirecting Win32 file handles:

If only the designers of Win32 had thought to provide a proper, general, API (instead of limited specialist functions that can only handle three mappings) which C/C++ library implementors could have used instead of having to roll their own, implementation-specific and runtime-library-instance-specific, mechanisms.


© Copyright 2010 Jonathan de Boyne Pollard. "Moral" rights asserted.
Permission is hereby granted to copy and to distribute this web page in its original, unmodified form as long as its last modification datestamp is preserved.