So I was working on improving a DOS emulator, when I found that something seemingly trivial wasn’t working right when COMMAND.COM was asked to do the following:
echo AB> foo.txt
echo CD>> foo.txt
Instead of ABCD, foo.txt contained ABBC.
I verified that yes, the right data was being passed to fwrite(), with the big caveat that what COMMAND.COM was doing wasn’t quite as straightforward as one might think:
- Open foo.txt
- Write ‘AB’
- Close foo.txt
- Open foo.txt
- Seek one byte backward from the end of the file
- Read one byte
- Write ‘CD’
- Close foo.txt
The reason for the complexity is that COMMAND.COM tries to deal with a case that the file ends with a Ctrl-Z character (which wasn’t the case for me), and if so, the Ctrl-Z needs to be deleted. Somehow the seek/read/write sequence was confusing things. But why?
Sitting down with a debugger, I could just see how the C run-time library (Open Watcom) could be fixed to avoid this problem. But I could not shake a nagging feeling that such a basic bug would have had to be discovered and fixed years ago.
So I proceeded to write a simple test program which I could try with other compilers.
To my great surprise, the venerable Microsoft Visual C++ 6.0 as well as IBM C/C++ 3.6 for Windows both only wrote ‘AB’ to the output file! The ‘CD’ never got written at all.
I added further logging to determine that in both cases, the second fwrite() reported that it wrote zero bytes. But that’s where things got a bit weird.
For the Microsoft runtime, ferror() was set but errno was zero. For the IBM runtime, ferror() was clear but errno was set to 41. Which according to IBM’s errno.h header means EPUTANDGET… and what does that error even mean?
At this point, I knew I was doing something wrong. But what? For once, stackoverflow actually had the right answer! Amazing, that almost never happens.
Why Oh Why?
Of course one has to wonder… why is it like this? Having basic file I/O functions behave in this non-obvious way (either quietly failing or not writing the expected data, depending on the sequence of other function calls) is clearly sub-optimal.
It is obvious that it would not be rocket science for the C library to keep a record of whether the most recent I/O was a read or a write, and perform the appropriate flush or seek when switching directions. Indeed it’s clear that for example the IBM C runtime keeps track internally, and issues a very specific error when the correct sequencing is violated.
The closest thing to an answer that I’ve been able to find is that “it’s always been this way”.
With a caveat that “always” means since circa 1979, not always always. Looking at the 1978 edition of K&R, it’s obvious why: The original K&R library only supported the read ("r"), write ("w"), and append ("a") modes for fopen(), with append being effectively a write. There was no update mode, ("r+") and hence reads and writes could not be mixed at all! That is very likely part of the puzzle.
By the time the oldest preserved ANSI C draft rolled out, the behavior was already set in stone. Consider how little things have changed over the years:
When a file is opened with update mode (
'+'as the second or third character in the mode argument), both input and output may be performed on the associated stream. However, output may not be directly followed by input without an intervening call to thefflushfunction or to a file positioning function (fseek,fsetpos, orrewind), and input may not be directly followed by output without an intervening call to a file positioning function, unless the input operation encounters end-of-file. Opening a file with update mode may open or create a binary stream in some implementations.ANSI X3J11 C draft, 1988
The ANSI C Rationale contains the following text:
A change of input/output direction on an update file is only allowed following a
fsetpos,fseek,rewind, orfflushoperation, since these are precisely the functions
which assure that the I/O buffer has been flushed.
The implication is that when the buffer I/O contains data, it’s not safe to switch read/write direction.
The published ANSI C89/ISO C90 is near identical to the draft Standard and does not bear repeating here. In C99, “may not” was replaced with “shall not” but little else changed:
When a file is opened with update mode (
'+'as the second or third character in the above list of mode argument values), both input and output may be performed on the associated stream. However, output shall not be directly followed by input without an intervening call to thefflushfunction or to a file positioning function (fseek,fsetpos, orrewind), and input shall not be directly followed by output without an intervening call to a file positioning function, unless the input operation encounters end-of-file. Opening (or creating) a text file with update mode may instead open (or create) a binary stream in some implementations.ISO C99, 1999
Fast forward another (almost) quarter century, and we have this:
When a file is opened with update mode (’+’ as the second or third character in the previously described list of mode argument values), both input and output may be performed on the associated stream. However, output shall not be directly followed by input without an intervening call to the
fflushfunction or to a file positioning function (fseek,fsetpos, orrewind), and input shall not be directly followed by output without an intervening call to a file positioning function, unless the input operation encounters end-of-file. Opening (or creating) a text file with update mode may instead open (or create) a binary stream in some implementations.ISO C23, 2024
As far as Standard C is concerned, this has provably not changed since 1988 until present.
But of course the ANSI X3J11 Committee did not invent the C library. It worked on the basis of earlier documents, namely the elusive 1984 /usr/group Standard in case of the library.
While I couldn’t find a copy of the /usr/group Standard, the /usr/group committee likewise didn’t create the C library but rather tried to standardize existing implementations. Which means that the answer might lie in old UNIX manuals.
Even System V is too new and we have to look further back. The AT&T UNIX System III manual contains the following text in the fread manual page:
When a file is opened for update, both input and output may be done on
the resulting stream. However, output may not be directly followed by
input without an interveningfseekorrewind, and input may not be directly
followed by output without an interveningfseek,rewind, or an input opera-
tion which encounters end of file.AT&T UNIX System III manual, 1980
Hmm, that text from 1980 is rather similar to what ended up in ANSI C89. Sure, there was no fsetpos() yet (an ANSI C invention), and the text is oddly missing any mention of fflush(), even though flushing almost certainly made it OK to switch from writing to reading even then.
But it’s obvious that the restriction on switching between reading and writing on C library streams has been there for a very, very long time.
7th Edition UNIX (1979), even in the updated documentation from 1983, does not mention update mode for fopen() and hence does not offer any advice on switching read/write directions.
Current Practice
At least Linux (glibc) and FreeBSD allow free intermixing of reads and writes. The FreeBSD man page for fopen() states:
Reads and writes may be intermixed on read/write streams in any order, and do not require an intermediate seek as in previous versions of stdio. This is not portable to other systems, however; ISO/IEC 9899:1990 (“ISO C90”) and IEEE Std 1003.1 (“POSIX.1”) both require that a file positioning function intervene between output and input, unless an input operation encounters end-of-file.
In contrast, Microsoft’s library documentation (as of 2024) mirrors ISO C and states that flushing or seeking is required when changing read/write direction.
On the one hand, transparently handling the direction switching in the library is not outrageously difficult. On the other hand, doing so encourages programmers to write non-conforming C code which will fail in rather interesting ways on other implementations. As always, there are tradeoffs.
Old Source
Looking at historic source code proved quite interesting.
In 32V UNIX from 1979, fopen clearly opens files for either reading or writing, but not both (and any mode other than ‘w’ or ‘a’ means implicitly ‘r’!).
V6 UNIX from 1975 is too old to even have fopen(). System III from 1980 on the other hand supports update mode, and opening streams for update sets an explicit _IORW flag (and, as mentioned above, the System III documentation demands extra care when switching I/O direction).
Things get confusing with V7 UNIX from 1979. Although the documentation does not show any update mode option for fopen(), the actual implementation supports it. In fact the V7 code from 1979 is nearly identical to what was in System III a year later. Why? I don’t know.
And then there’s the 2BSD code, again from 1979. While the BSD fopen() has no provision for indicating update mode with the ‘+’ character, it allows specifying open modes like "rw", setting both the _IOREAD and _IOWRT flags. In fact the 2BSD man page for fopen explicitly lists "rw" and "ra" as supported open modes which allow both reading and writing, but there is nothing said about whether mixing fread() and fwrite() freely is allowed. There is also an explanatory README file with a note from November 1978 describing the change to allow mixed read and write access.
A 1977 paper by Dennis M. Ritchie A New Input-Output Package is quite clear that when fopen() was first conceived, a stream would support either reading or writing, but not both. It is also clear that users found this too restrictive and by 1979, there were at least two different implementations (AT&T and BSD) which allowed mixed read/write streams.
Notably in the BSD implementation, fopen() was modified to allow both reading and writing but fread() and fwrite() were not. It is not clear to me if the BSD code was robust enough to allow free mixing of reads and writes. The AT&T documentation has always been clear that it’s not allowed.
And as far as Standard C and POSIX are concerned, that has not changed until today. To write portable code, it is necessary to take some action when changing read/write direction. A dummy call such as
fseek( f, 0, SEEK_CUR );
is entirely sufficient to get the stream into a state where switching between reading and writing is safe.
I suppose oddities like this just happen when you have nearly nearly 50 years of history behind you.
