[flang] Debugging of ACCESS='STREAM' I/O

[klausler]

Summary

Corrects the runtime implementation of I/O on files with
the access mode ACCESS='STREAM'. This is a collection
of edge-case tweaks to ensure that the distinctions between
stream and direct/sequential files, unformatted or formatted,
are respected where appropriate.
Moves NextInField() from io-stmt.h to io-stmt.cpp --
it was getting too big to keep in a header.

[Meinersbur]

This patch causes the test flang-Unit::ExternalIOTests.TestWriteAfterNonAvancingInput to fail under Windows (https://lab.llvm.org/buildbot/#/builders/172/builds/7664)

[ RUN      ] ExternalIOTests.TestWriteAfterNonAvancingInput
C:\Users\buildbot-worker\minipc-ryzen-win\flang-x86_64-windows\llvm-project\flang\unittests\Runtime\ExternalIOTest.cpp(649): error: Expected equality of these values:
  resultRecord
    Which is: "ABCDEFGHXYZ\r        "
  expectedRecord
    Which is: "ABCDEFGHXYZ         "
Record after non advancing read followed by wrote

[awarzynski]

@klausler Thanks for this patch! I was hoping that I could find a fix for the failure on Windows myself, but no luck. I reverted instead - our Windows buildbot is one of our fastest and reliable CI machines. Also, while this does not work on Windows, our pre-commit CI is unnecessarily noisy.

Please, could take a look and re-submit?

[klausler]

In D118834#3293689, @awarzynski wrote:

@klausler Thanks for this patch! I was hoping that I could find a fix for the failure on Windows myself, but no luck. I reverted instead - our Windows buildbot is one of our fastest and reliable CI machines. Also, while this does not work on Windows, our pre-commit CI is unnecessarily noisy.

Please, could take a look and re-submit?

It seems that this patch exposed a problem; I suspect file truncation on Windows. I may resubmit later.

[Meinersbur]

Thanks for reverting. The stray \r makes me suspect something removed an \n but left over the second byte from Windows CRLF line endings.

If needed, I can look for the problem on Windows.

[klausler]

In D118834#3294452, @Meinersbur wrote:

Thanks for reverting. The stray \r makes me suspect something removed an \n but left over the second byte from Windows CRLF line endings.

If needed, I can look for the problem on Windows.

The f18 runtime doesn't emit \r (yet). So I don't know from whence it came, if it did indeed come from the f18 runtime.

[Meinersbur]

The runtime uses a temporary file in scratch mode that the unittest uses, which is opened in text mode. Under Windows, when writing \n to a file opened in text mode, it will actually write \r\n. However, the internal position counter frameOffsetInFile_ will not take the additional character into account. On Rewind, it will truncate the file to the frameOffsetInFile_ size, keeping the first \r but truncating the second character \n away.

This makes the test pass:

diff --git a/flang/runtime/file.cpp b/flang/runtime/file.cpp
index 4e36978cfb23..787845f78353 100644
--- a/flang/runtime/file.cpp
+++ b/flang/runtime/file.cpp
@@ -46,7 +46,7 @@ static int openfile_mkstemp(IoErrorHandler &handler) {
     return -1;
   }
   int fd{::_open(
-      tempFileName, _O_CREAT | _O_TEMPORARY | _O_RDWR, _S_IREAD | _S_IWRITE)};
+      tempFileName, _O_CREAT | _O_BINARY | _O_TEMPORARY | _O_RDWR, _S_IREAD | _S_IWRITE)};
 #else
   char path[]{"/tmp/Fortran-Scratch-XXXXXX"};
   int fd{::mkstemp(path)};

However, I don't think this is an universal solution, as stdout and stderr are always in text mode.

[klausler]

In D118834#3295507, @Meinersbur wrote:

The runtime uses a temporary file in scratch mode that the unittest uses, which is opened in text mode. Under Windows, when writing \n to a file opened in text mode, it will actually write \r\n. However, the internal position counter frameOffsetInFile_ will not take the additional character into account. On Rewind, it will truncate the file to the frameOffsetInFile_ size, keeping the first \r but truncating the second character \n away.

This makes the test pass:

diff --git a/flang/runtime/file.cpp b/flang/runtime/file.cpp
index 4e36978cfb23..787845f78353 100644
--- a/flang/runtime/file.cpp
+++ b/flang/runtime/file.cpp
@@ -46,7 +46,7 @@ static int openfile_mkstemp(IoErrorHandler &handler) {
     return -1;
   }
   int fd{::_open(
-      tempFileName, _O_CREAT | _O_TEMPORARY | _O_RDWR, _S_IREAD | _S_IWRITE)};
+      tempFileName, _O_CREAT | _O_BINARY | _O_TEMPORARY | _O_RDWR, _S_IREAD | _S_IWRITE)};
 #else
   char path[]{"/tmp/Fortran-Scratch-XXXXXX"};
   int fd{::mkstemp(path)};

However, I don't think this is an universal solution, as stdout and stderr are always in text mode.

Thanks, that's a helpful clue. Unfortunately in Fortran we can't always know whether a file will be formatted or unformatted at the time that the file is opened, so the runtime will probably have to always use binary mode and emit explicit carriage returns. Stdout and stderr being in text mode is not much of a problem -- they don't have to be positionable and are always formatted. I will get back to this in a few weeks after I dig out from under a pile of more important things.

flang/include/flang/Runtime/iostat.h

enum Iostat {
 
   // The remaining error codes are not exported.
   IostatGenericError = 1001, // see IOMSG= for details
   IostatRecordWriteOverrun,
   IostatRecordReadOverrun,
   IostatInternalWriteOverrun,
   IostatErrorInFormat,
   IostatErrorInKeyword,
-  IostatEndfileNonSequential,
+  IostatEndfileDirect,
   IostatEndfileUnwritable,
   IostatOpenBadRecl,
   IostatOpenUnknownSize,
   IostatOpenBadAppend,
   IostatWriteToReadOnly,
   IostatReadFromWriteOnly,
   IostatBackspaceNonSequential,
   IostatBackspaceAtFirstRecord,
   IostatRewindNonSequential,
   IostatWriteAfterEndfile,
 };
 
 const char *IostatErrorString(int);
 
 } // namespace Fortran::runtime::io
 #endif // FORTRAN_RUNTIME_IOSTAT_H_

flang/runtime/connection.h

 
 namespace Fortran::runtime::io {
 
 class IoStatementState;
 
 enum class Direction { Output, Input };
 enum class Access { Sequential, Direct, Stream };
 
-inline bool IsRecordFile(Access a) { return a != Access::Stream; }
-
 // These characteristics of a connection are immutable after being
 // established in an OPEN statement.
 struct ConnectionAttributes {
   Access access{Access::Sequential}; // ACCESS='SEQUENTIAL', 'DIRECT', 'STREAM'
   std::optional<bool> isUnformatted; // FORM='UNFORMATTED' if true
   bool isUTF8{false}; // ENCODING='UTF-8'
   std::optional<std::int64_t> openRecl; // RECL= on OPEN
+
+  bool IsRecordFile() const {
+    // Formatted stream files are viewed as having records, at least on input
+    return access != Access::Stream || !isUnformatted.value_or(true);
+  }
 };
 
 struct ConnectionState : public ConnectionAttributes {
   bool IsAtEOF() const; // true when read has hit EOF or endfile record
   bool IsAfterEndfile() const; // true after ENDFILE until repositioned
   std::size_t RemainingSpaceInRecord() const;
   bool NeedAdvance(std::size_t) const;
   void HandleAbsolutePosition(std::int64_t);

flang/runtime/edit-input.cpp

 
 namespace Fortran::runtime::io {
 
 static bool EditBOZInput(IoStatementState &io, const DataEdit &edit, void *n,
     int base, int totalBitSize) {
   std::optional<int> remaining;
   std::optional<char32_t> next{io.PrepareInput(edit, remaining)};
   common::UnsignedInt128 value{0};
-  for (; next; next = io.NextInField(remaining)) {
+  for (; next; next = io.NextInField(remaining, edit)) {
     char32_t ch{*next};
     if (ch == ' ' || ch == '\t') {
       continue;
     }
     int digit{0};
     if (ch >= '0' && ch <= '1') {
       digit = ch - '0';
     } else if (base >= 8 && ch >= '2' && ch <= '7') {
static bool ScanNumericPrefix(IoStatementState &io, const DataEdit &edit,
     std::optional<char32_t> &next, std::optional<int> &remaining) {
   next = io.PrepareInput(edit, remaining);
   bool negative{false};
   if (next) {
     negative = *next == '-';
     if (negative || *next == '+') {
       io.GotChar();
       io.SkipSpaces(remaining);
-      next = io.NextInField(remaining, GetDecimalPoint(edit));
+      next = io.NextInField(remaining, edit);
     }
   }
   return negative;
 }
 
 bool EditIntegerInput(
     IoStatementState &io, const DataEdit &edit, void *n, int kind) {
   RUNTIME_CHECK(io.GetIoErrorHandler(), kind >= 1 && !(kind & (kind - 1)));
io.GetIoErrorHandler().SignalError(IostatErrorInFormat,
         edit.descriptor);
     return false;
   }
   std::optional<int> remaining;
   std::optional<char32_t> next;
   bool negate{ScanNumericPrefix(io, edit, next, remaining)};
   common::UnsignedInt128 value{0};
   bool any{negate};
-  for (; next; next = io.NextInField(remaining)) {
+  for (; next; next = io.NextInField(remaining, edit)) {
     char32_t ch{*next};
     if (ch == ' ' || ch == '\t') {
       if (edit.modes.editingFlags & blankZero) {
         ch = '0'; // BZ mode - treat blank as if it were zero
       } else {
         continue;
       }
     }
static int ScanRealInput(char *buffer, int bufferSize, IoStatementState &io,
   }
   char32_t decimal{GetDecimalPoint(edit)};
   char32_t first{*next >= 'a' && *next <= 'z' ? *next + 'A' - 'a' : *next};
   if (first == 'N' || first == 'I') {
     // NaN or infinity - convert to upper case
     // Subtle: a blank field of digits could be followed by 'E' or 'D',
     for (; next &&
          ((*next >= 'a' && *next <= 'z') || (*next >= 'A' && *next <= 'Z'));
-         next = io.NextInField(remaining)) {
+         next = io.NextInField(remaining, edit)) {
       if (*next >= 'a' && *next <= 'z') {
         Put(*next - 'a' + 'A');
       } else {
         Put(*next);
       }
     }
     if (next && *next == '(') { // NaN(...)
       while (next && *next != ')') {
-        next = io.NextInField(remaining);
+        next = io.NextInField(remaining, edit);
       }
     }
     exponent = 0;
   } else if (first == decimal || (first >= '0' && first <= '9') ||
       first == 'E' || first == 'D' || first == 'Q') {
     Put('.'); // input field is normalized to a fraction
     auto start{got};
     bool bzMode{(edit.modes.editingFlags & blankZero) != 0};
-    for (; next; next = io.NextInField(remaining, decimal)) {
+    for (; next; next = io.NextInField(remaining, edit)) {
       char32_t ch{*next};
       if (ch == ' ' || ch == '\t') {
         if (bzMode) {
           ch = '0'; // BZ mode - treat blank as if it were zero
         } else {
           continue;
         }
       }
if (got == start) {
       Put('0'); // emit at least one digit
     }
     if (next &&
         (*next == 'e' || *next == 'E' || *next == 'd' || *next == 'D' ||
             *next == 'q' || *next == 'Q')) {
       // Optional exponent letter.  Blanks are allowed between the
       // optional exponent letter and the exponent value.
       io.SkipSpaces(remaining);
-      next = io.NextInField(remaining);
+      next = io.NextInField(remaining, edit);
     }
     // The default exponent is -kP, but the scale factor doesn't affect
     // an explicit exponent.
     exponent = -edit.modes.scale;
     if (next &&
         (*next == '-' || *next == '+' || (*next >= '0' && *next <= '9') ||
             (bzMode && (*next == ' ' || *next == '\t')))) {
       bool negExpo{*next == '-'};
       if (negExpo || *next == '+') {
-        next = io.NextInField(remaining);
+        next = io.NextInField(remaining, edit);
       }
-      for (exponent = 0; next; next = io.NextInField(remaining)) {
+      for (exponent = 0; next; next = io.NextInField(remaining, edit)) {
         if (*next >= '0' && *next <= '9') {
           exponent = 10 * exponent + *next - '0';
         } else if (bzMode && (*next == ' ' || *next == '\t')) {
           exponent = 10 * exponent;
         } else {
           break;
         }
       }
if (first == 'N' || first == 'I') {
     // TODO: hex FP input
     exponent = 0;
     return 0;
   }
   // Consume the trailing ')' of a list-directed or NAMELIST complex
   // input value.
   if (edit.descriptor == DataEdit::ListDirectedImaginaryPart) {
     if (next && (*next == ' ' || *next == '\t')) {
-      next = io.NextInField(remaining);
+      next = io.NextInField(remaining, edit);
     }
     if (!next) { // NextInField fails on separators like ')'
       next = io.GetCurrentChar();
       if (next && *next == ')') {
         io.HandleRelativePosition(1);
       }
     }
   } else if (remaining) {
     while (next && (*next == ' ' || *next == '\t')) {
-      next = io.NextInField(remaining);
+      next = io.NextInField(remaining, edit);
     }
     if (next) {
       return 0; // error: unused nonblank character in fixed-width field
     }
   }
   return got;
 }
 
default:
     io.GetIoErrorHandler().SignalError(IostatErrorInFormat,
         "Data edit descriptor '%c' may not be used for LOGICAL input",
         edit.descriptor);
     return false;
   }
   std::optional<int> remaining;
   std::optional<char32_t> next{io.PrepareInput(edit, remaining)};
   if (next && *next == '.') { // skip optional period
-    next = io.NextInField(remaining);
+    next = io.NextInField(remaining, edit);
   }
   if (!next) {
     io.GetIoErrorHandler().SignalError("Empty LOGICAL input field");
     return false;
   }
   switch (*next) {
   case 'T':
   case 't':
     x = true;
     break;
   case 'F':
   case 'f':
     x = false;
     break;
   default:
     io.GetIoErrorHandler().SignalError(
         "Bad character '%lc' in LOGICAL input field", *next);
     return false;
   }
   if (remaining) { // ignore the rest of the field
     io.HandleRelativePosition(*remaining);
   } else if (edit.descriptor == DataEdit::ListDirected) {
-    while (io.NextInField(remaining)) { // discard rest of field
+    while (io.NextInField(remaining, edit)) { // discard rest of field
     }
   }
   return true;
 }
 
 // See 13.10.3.1 paragraphs 7-9 in Fortran 2018
 static bool EditDelimitedCharacterInput(
     IoStatementState &io, char *x, std::size_t length, char32_t delimiter) {
if (length > 0) {
       --length;
     }
   }
   std::fill_n(x, length, ' ');
   return result;
 }
 
 static bool EditListDirectedDefaultCharacterInput(
-    IoStatementState &io, char *x, std::size_t length) {
+    IoStatementState &io, char *x, std::size_t length, const DataEdit &edit) {
   auto ch{io.GetCurrentChar()};
   if (ch && (*ch == '\'' || *ch == '"')) {
     io.HandleRelativePosition(1);
     return EditDelimitedCharacterInput(io, x, length, *ch);
   }
   if (IsNamelistName(io) || io.GetConnectionState().IsAtEOF()) {
     return false;
   }
   // Undelimited list-directed character input: stop at a value separator
   // or the end of the current record.
   std::optional<int> remaining{length};
-  for (std::optional<char32_t> next{io.NextInField(remaining)}; next;
-       next = io.NextInField(remaining)) {
+  while (std::optional<char32_t> next{io.NextInField(remaining, edit)}) {
     switch (*next) {
     case ' ':
     case '\t':
     case ',':
     case ';':
     case '/':
       remaining = 0; // value separator: stop
       break;
     default:
       *x++ = *next;
       --length;
     }
   }
   std::fill_n(x, length, ' ');
   return true;
 }
 
 bool EditDefaultCharacterInput(
     IoStatementState &io, const DataEdit &edit, char *x, std::size_t length) {
   switch (edit.descriptor) {
   case DataEdit::ListDirected:
-    return EditListDirectedDefaultCharacterInput(io, x, length);
+    return EditListDirectedDefaultCharacterInput(io, x, length, edit);
   case 'A':
   case 'G':
     break;
   default:
     io.GetIoErrorHandler().SignalError(IostatErrorInFormat,
         "Data edit descriptor '%c' may not be used with a CHARACTER data item",
         edit.descriptor);
     return false;
   }
   if (io.GetConnectionState().IsAtEOF()) {
     return false;
   }
   std::optional<int> remaining{length};
   if (edit.width && *edit.width > 0) {
     remaining = *edit.width;
   }
   // When the field is wider than the variable, we drop the leading
   // characters.  When the variable is wider than the field, there's
   // trailing padding.
   std::int64_t skip{*remaining - static_cast<std::int64_t>(length)};
-  for (std::optional<char32_t> next{io.NextInField(remaining)}; next;
-       next = io.NextInField(remaining)) {
+  while (std::optional<char32_t> next{io.NextInField(remaining, edit)}) {
     if (skip > 0) {
       --skip;
       io.GotChar(-1);
     } else {
       *x++ = *next;
       --length;
     }
   }

flang/runtime/file.h

public:
   const char *path() const { return path_.get(); }
   void set_path(OwningPtr<char> &&, std::size_t bytes);
   std::size_t pathLength() const { return pathLength_; }
   bool mayRead() const { return mayRead_; }
   bool mayWrite() const { return mayWrite_; }
   bool mayPosition() const { return mayPosition_; }
   bool mayAsynchronous() const { return mayAsynchronous_; }
   void set_mayAsynchronous(bool yes) { mayAsynchronous_ = yes; }
-  FileOffset position() const { return position_; }
   bool isTerminal() const { return isTerminal_; }
   std::optional<FileOffset> knownSize() const { return knownSize_; }
 
   bool IsConnected() const { return fd_ >= 0; }
   void Open(OpenStatus, std::optional<Action>, Position, IoErrorHandler &);
   void Predefine(int fd);
   void Close(CloseStatus, IoErrorHandler &);
 

flang/runtime/io-api.cpp

bool IONAME(SetPad)(Cookie cookie, const char *keyword, std::size_t length) {
   connection.modes.pad =
       YesOrNo(keyword, length, "PAD", io.GetIoErrorHandler());
   return true;
 }
 
 bool IONAME(SetPos)(Cookie cookie, std::int64_t pos) {
   IoStatementState &io{*cookie};
   ConnectionState &connection{io.GetConnectionState()};
+  IoErrorHandler &handler{io.GetIoErrorHandler()};
   if (connection.access != Access::Stream) {
-    io.GetIoErrorHandler().SignalError(
-        "POS= may not appear unless ACCESS='STREAM'");
+    handler.SignalError("POS= may not appear unless ACCESS='STREAM'");
     return false;
   }
-  if (pos < 1) {
-    io.GetIoErrorHandler().SignalError(
-        "POS=%zd is invalid", static_cast<std::intmax_t>(pos));
+  if (pos < 1) { // POS=1 is beginning of file (12.6.2.11)
+    handler.SignalError("POS=%zd is invalid", static_cast<std::intmax_t>(pos));
     return false;
   }
   if (auto *unit{io.GetExternalFileUnit()}) {
-    unit->SetPosition(pos);
+    unit->SetPosition(pos - 1, handler);
     return true;
   }
   io.GetIoErrorHandler().Crash("SetPos() on internal unit");
   return false;
 }
 
 bool IONAME(SetRec)(Cookie cookie, std::int64_t rec) {
   IoStatementState &io{*cookie};
   ConnectionState &connection{io.GetConnectionState()};
+  IoErrorHandler &handler{io.GetIoErrorHandler()};
   if (connection.access != Access::Direct) {
-    io.GetIoErrorHandler().SignalError(
-        "REC= may not appear unless ACCESS='DIRECT'");
+    handler.SignalError("REC= may not appear unless ACCESS='DIRECT'");
     return false;
   }
   if (!connection.openRecl) {
-    io.GetIoErrorHandler().SignalError("RECL= was not specified");
+    handler.SignalError("RECL= was not specified");
     return false;
   }
   if (rec < 1) {
-    io.GetIoErrorHandler().SignalError(
-        "REC=%zd is invalid", static_cast<std::intmax_t>(rec));
+    handler.SignalError("REC=%zd is invalid", static_cast<std::intmax_t>(rec));
     return false;
   }
   connection.currentRecordNumber = rec;
   if (auto *unit{io.GetExternalFileUnit()}) {
-    unit->SetPosition((rec - 1) * *connection.openRecl);
+    unit->SetPosition((rec - 1) * *connection.openRecl, handler);
   }
   return true;
 }
 
 bool IONAME(SetRound)(Cookie cookie, const char *keyword, std::size_t length) {
   IoStatementState &io{*cookie};
   ConnectionState &connection{io.GetConnectionState()};
   static const char *keywords[]{"UP", "DOWN", "ZERO", "NEAREST", "COMPATIBLE",

flang/runtime/io-stmt.h

std::optional<char32_t> PrepareInput(
     if (edit.descriptor == DataEdit::ListDirected) {
       GetNextNonBlank();
     } else {
       if (edit.width.value_or(0) > 0) {
         remaining = *edit.width;
       }
       SkipSpaces(remaining);
     }
-    return NextInField(remaining);
+    return NextInField(remaining, edit);
   }
 
   std::optional<char32_t> SkipSpaces(std::optional<int> &remaining) {
     while (!remaining || *remaining > 0) {
       if (auto ch{GetCurrentChar()}) {
         if (*ch != ' ' && *ch != '\t') {
           return ch;
         }
         HandleRelativePosition(1);
         if (remaining) {
           GotChar();
           --*remaining;
         }
       } else {
         break;
       }
     }
     return std::nullopt;
   }
 
+  // Acquires the next input character, respecting any applicable field width
+  // or separator character.
   std::optional<char32_t> NextInField(
-      std::optional<int> &remaining, char32_t decimal = '.') {
-    if (!remaining) { // list-directed or NAMELIST: check for separators
-      if (auto next{GetCurrentChar()}) {
-        if (*next == decimal) { // can be ','
-          HandleRelativePosition(1);
-          return next;
-        }
-        switch (*next) {
-        case ' ':
-        case '\t':
-        case ',':
-        case ';':
-        case '/':
-        case '(':
-        case ')':
-        case '\'':
-        case '"':
-        case '*':
-        case '\n': // for stream access
-          break;
-        default:
-          HandleRelativePosition(1);
-          return next;
-        }
-      }
-    } else if (*remaining > 0) {
-      if (auto next{GetCurrentChar()}) {
-        --*remaining;
-        HandleRelativePosition(1);
-        GotChar();
-        return next;
-      }
-      const ConnectionState &connection{GetConnectionState()};
-      if (!connection.IsAtEOF()) {
-        if (auto length{connection.EffectiveRecordLength()}) {
-          if (connection.positionInRecord >= *length) {
-            IoErrorHandler &handler{GetIoErrorHandler()};
-            if (mutableModes().nonAdvancing) {
-              handler.SignalEor();
-            } else if (connection.openRecl && !connection.modes.pad) {
-              handler.SignalError(IostatRecordReadOverrun);
-            }
-            if (connection.modes.pad) { // PAD='YES'
-              --*remaining;
-              return std::optional<char32_t>{' '};
-            }
-          }
-        }
-      }
-    }
-    return std::nullopt;
-  }
+      std::optional<int> &remaining, const DataEdit &);
 
   // Skips spaces, advances records, and ignores NAMELIST comments
   std::optional<char32_t> GetNextNonBlank() {
     auto ch{GetCurrentChar()};
     bool inNamelist{mutableModes().inNamelist};
     while (!ch || *ch == ' ' || *ch == '\t' || (inNamelist && *ch == '!')) {
       if (ch && (*ch == ' ' || *ch == '\t')) {
         HandleRelativePosition(1);

flang/runtime/io-stmt.cpp

 }
 
 template <Direction DIR>
 ExternalIoStatementState<DIR>::ExternalIoStatementState(
     ExternalFileUnit &unit, const char *sourceFile, int sourceLine)
     : ExternalIoStatementBase{unit, sourceFile, sourceLine}, mutableModes_{
                                                                  unit.modes} {
   if constexpr (DIR == Direction::Output) {
-    // If the last statement was a non advancing IO input statement, the unit
+    // If the last statement was a non-advancing IO input statement, the unit
     // furthestPositionInRecord was not advanced, but the positionInRecord may
     // have been advanced. Advance furthestPositionInRecord here to avoid
     // overwriting the part of the record that has been read with blanks.
     unit.furthestPositionInRecord =
         std::max(unit.furthestPositionInRecord, unit.positionInRecord);
   }
 }
 
bool IoStatementState::EmitField(
   if (length > static_cast<std::size_t>(width)) {
     return EmitRepeated('*', width);
   } else {
     return EmitRepeated(' ', static_cast<int>(width - length)) &&
         Emit(p, length);
   }
 }
 
+std::optional<char32_t> IoStatementState::NextInField(
+    std::optional<int> &remaining, const DataEdit &edit) {
+  if (!remaining) { // Stream, list-directed, or NAMELIST
+    if (auto next{GetCurrentChar()}) {
+      if (edit.IsListDirected()) {
+        // list-directed or NAMELIST: check for separators
+        switch (*next) {
+        case ' ':
+        case '\t':
+        case ';':
+        case '/':
+        case '(':
+        case ')':
+        case '\'':
+        case '"':
+        case '*':
+        case '\n': // for stream access
+          return std::nullopt;
+        case ',':
+          if (edit.modes.editingFlags & decimalComma) {
+            break;
+          } else {
+            return std::nullopt;
+          }
+        default:
+          break;
+        }
+      }
+      HandleRelativePosition(1);
+      GotChar();
+      return next;
+    }
+  } else if (*remaining > 0) {
+    if (auto next{GetCurrentChar()}) {
+      --*remaining;
+      HandleRelativePosition(1);
+      GotChar();
+      return next;
+    }
+    const ConnectionState &connection{GetConnectionState()};
+    if (!connection.IsAtEOF()) {
+      if (auto length{connection.EffectiveRecordLength()}) {
+        if (connection.positionInRecord >= *length) {
+          IoErrorHandler &handler{GetIoErrorHandler()};
+          if (mutableModes().nonAdvancing) {
+            handler.SignalEor();
+          } else if (connection.openRecl && !connection.modes.pad) {
+            handler.SignalError(IostatRecordReadOverrun);
+          }
+          if (connection.modes.pad) { // PAD='YES'
+            --*remaining;
+            return std::optional<char32_t>{' '};
+          }
+        }
+      }
+    }
+  }
+  return std::nullopt;
+}
+
 bool IoStatementState::Inquire(
     InquiryKeywordHash inquiry, char *out, std::size_t chars) {
   return std::visit(
       [&](auto &x) { return x.get().Inquire(inquiry, out, chars); }, u_);
 }
 
 bool IoStatementState::Inquire(InquiryKeywordHash inquiry, bool &out) {
   return std::visit([&](auto &x) { return x.get().Inquire(inquiry, out); }, u_);
case HashInquiryKeyword("NEXTREC"):
     if (unit().access == Access::Direct) {
       result = unit().currentRecordNumber;
     }
     return true;
   case HashInquiryKeyword("NUMBER"):
     result = unit().IsConnected() ? unit().unitNumber() : -1;
     return true;
   case HashInquiryKeyword("POS"):
-    result = unit().position();
+    result = unit().InquirePos();
     return true;
   case HashInquiryKeyword("RECL"):
     if (!unit().IsConnected()) {
       result = -1;
     } else if (unit().access == Access::Stream) {
       result = -2;
     } else if (unit().openRecl) {
       result = *unit().openRecl;

flang/runtime/iostat.cpp

const char *IostatErrorString(int iostat) {
   case IostatRecordReadOverrun:
     return "Excessive input from fixed-size record";
   case IostatInternalWriteOverrun:
     return "Internal write overran available records";
   case IostatErrorInFormat:
     return "Invalid FORMAT";
   case IostatErrorInKeyword:
     return "Bad keyword argument value";
-  case IostatEndfileNonSequential:
-    return "ENDFILE on non-sequential file";
+  case IostatEndfileDirect:
+    return "ENDFILE on direct-access file";
   case IostatEndfileUnwritable:
     return "ENDFILE on read-only file";
   case IostatOpenBadRecl:
     return "OPEN with bad RECL= value";
   case IostatOpenUnknownSize:
     return "OPEN of file of unknown size";
   case IostatOpenBadAppend:
     return "OPEN(POSITION='APPEND') of unpositionable file";

flang/runtime/unit.h

public:
   void FinishReadingRecord(IoErrorHandler &);
   bool AdvanceRecord(IoErrorHandler &);
   void BackspaceRecord(IoErrorHandler &);
   void FlushOutput(IoErrorHandler &);
   void FlushIfTerminal(IoErrorHandler &);
   void Endfile(IoErrorHandler &);
   void Rewind(IoErrorHandler &);
   void EndIoStatement();
-  void SetPosition(std::int64_t pos) {
-    frameOffsetInFile_ = pos;
-    recordOffsetInFrame_ = 0;
-    BeginRecord();
+  void SetPosition(std::int64_t, IoErrorHandler &); // zero-based
+  std::int64_t InquirePos() const {
+    // 12.6.2.11 defines POS=1 as the beginning of file
+    return frameOffsetInFile_ + 1;
   }
 
   ChildIo *GetChildIo() { return child_.get(); }
   ChildIo &PushChildIo(IoStatementState &);
   void PopChildIo(ChildIo &);
 
 private:
   static UnitMap &GetUnitMap();
   const char *FrameNextInput(IoErrorHandler &, std::size_t);
   void BeginSequentialVariableUnformattedInputRecord(IoErrorHandler &);
-  void BeginSequentialVariableFormattedInputRecord(IoErrorHandler &);
+  void BeginVariableFormattedInputRecord(IoErrorHandler &);
   void BackspaceFixedRecord(IoErrorHandler &);
   void BackspaceVariableUnformattedRecord(IoErrorHandler &);
   void BackspaceVariableFormattedRecord(IoErrorHandler &);
-  bool SetSequentialVariableFormattedRecordLength();
+  bool SetVariableFormattedRecordLength();
   void DoImpliedEndfile(IoErrorHandler &);
   void DoEndfile(IoErrorHandler &);
   void CommitWrites();
 
   int unitNumber_{-1};
   Direction direction_{Direction::Output};
-  bool impliedEndfile_{false}; // seq. output has taken place
+  bool impliedEndfile_{false}; // sequential/stream output has taken place
   bool beganReadingRecord_{false};
 
   Lock lock_;
 
   // When an I/O statement is in progress on this unit, holds its state.
   std::variant<std::monostate, OpenStatementState, CloseStatementState,
       ExternalFormattedIoStatementState<Direction::Output>,
       ExternalFormattedIoStatementState<Direction::Input>,

flang/runtime/unit.cpp

if (access == Access::Direct) {
     }
     recordLength = openRecl;
   }
   endfileRecordNumber.reset();
   currentRecordNumber = 1;
   if (totalBytes && access == Access::Direct && openRecl.value_or(0) > 0) {
     endfileRecordNumber = 1 + (*totalBytes / *openRecl);
   }
-  if (position == Position::Append) {
+  if (position == Position::Append && access != Access::Stream) {
     if (!endfileRecordNumber) {
       // Fake it so that we can backspace relative from the end
       endfileRecordNumber = std::numeric_limits<std::int64_t>::max() - 2;
     }
     currentRecordNumber = *endfileRecordNumber;
   }
 }
 
if (got >= need) {
     std::memcpy(data, Frame() + recordOffsetInFrame_ + positionInRecord, bytes);
     if (swapEndianness_) {
       SwapEndianness(data, bytes, elementBytes);
     }
     positionInRecord += bytes;
     furthestPositionInRecord = furthestAfter;
     return true;
   } else {
-    // EOF or error: can be handled & has been signaled
+    handler.SignalEnd();
+    if (access == Access::Sequential) {
       endfileRecordNumber = currentRecordNumber;
+    }
     return false;
   }
 }
 
 std::size_t ExternalFileUnit::GetNextInputBytes(
     const char *&p, IoErrorHandler &handler) {
   RUNTIME_CHECK(handler, direction_ == Direction::Input);
   p = FrameNextInput(handler, 1);

 const char *ExternalFileUnit::FrameNextInput(
     IoErrorHandler &handler, std::size_t bytes) {
   RUNTIME_CHECK(handler, isUnformatted.has_value() && !*isUnformatted);
   if (static_cast<std::int64_t>(positionInRecord + bytes) <=
       recordLength.value_or(positionInRecord + bytes)) {
     auto at{recordOffsetInFrame_ + positionInRecord};
     auto need{static_cast<std::size_t>(at + bytes)};
     auto got{ReadFrame(frameOffsetInFile_, need, handler)};
-    SetSequentialVariableFormattedRecordLength();
+    SetVariableFormattedRecordLength();
     if (got >= need) {
       return Frame() + at;
     }
     handler.SignalEnd();
+    if (access == Access::Sequential) {
       endfileRecordNumber = currentRecordNumber;
     }
+  }
   return nullptr;
 }
 
-bool ExternalFileUnit::SetSequentialVariableFormattedRecordLength() {
-  if (recordLength || access != Access::Sequential) {
+bool ExternalFileUnit::SetVariableFormattedRecordLength() {
+  if (recordLength || access == Access::Direct) {
     return true;
   } else if (FrameLength() > recordOffsetInFrame_) {
     const char *record{Frame() + recordOffsetInFrame_};
     std::size_t bytes{FrameLength() - recordOffsetInFrame_};
     if (const char *nl{
             reinterpret_cast<const char *>(std::memchr(record, '\n', bytes))}) {
       recordLength = nl - record;
       if (*recordLength > 0 && record[*recordLength - 1] == '\r') {
if (access == Access::Direct) {
       auto need{static_cast<std::size_t>(recordOffsetInFrame_ + *openRecl)};
       auto got{ReadFrame(frameOffsetInFile_, need, handler)};
       if (got >= need) {
         recordLength = openRecl;
       } else {
         recordLength.reset();
         handler.SignalEnd();
       }
-    } else if (access == Access::Sequential) {
+    } else {
       recordLength.reset();
       if (IsAtEOF()) {
         handler.SignalEnd();
       } else {
         RUNTIME_CHECK(handler, isUnformatted.has_value());
-        if (isUnformatted.value_or(false)) {
+        if (*isUnformatted) {
+          if (access == Access::Sequential) {
             BeginSequentialVariableUnformattedInputRecord(handler);
-        } else { // formatted
-          BeginSequentialVariableFormattedInputRecord(handler);
+          }
+        } else { // formatted sequential or stream
+          BeginVariableFormattedInputRecord(handler);
         }
       }
     }
   }
   RUNTIME_CHECK(handler,
-      recordLength.has_value() || !IsRecordFile(access) || handler.InError());
+      recordLength.has_value() || !IsRecordFile() || handler.InError());
   return !handler.InError();
 }
 
 void ExternalFileUnit::FinishReadingRecord(IoErrorHandler &handler) {
   RUNTIME_CHECK(handler, direction_ == Direction::Input && beganReadingRecord_);
   beganReadingRecord_ = false;
   if (handler.InError() && handler.GetIoStat() != IostatEor) {
-    // avoid bogus crashes in END/ERR circumstances
-  } else if (access == Access::Sequential) {
+    // Avoid bogus crashes in END/ERR circumstances; but
+    // still increment the current record number so that
+    // an attempted read of an endfile record, followed by
+    // a BACKSPACE, will still be at EOF.
+    ++currentRecordNumber;
+  } else if (IsRecordFile()) {
     RUNTIME_CHECK(handler, recordLength.has_value());
     recordOffsetInFrame_ += *recordLength;
-    if (openRecl && access == Access::Direct) {
-      frameOffsetInFile_ += recordOffsetInFrame_;
-      recordOffsetInFrame_ = 0;
-    } else {
+    if (access != Access::Direct) {
       RUNTIME_CHECK(handler, isUnformatted.has_value());
       recordLength.reset();
       if (isUnformatted.value_or(false)) {
         // Retain footer in frame for more efficient BACKSPACE
         frameOffsetInFile_ += recordOffsetInFrame_;
         recordOffsetInFrame_ = sizeof(std::uint32_t);
       } else { // formatted
         if (FrameLength() > recordOffsetInFrame_ &&
             Frame()[recordOffsetInFrame_] == '\r') {
           ++recordOffsetInFrame_;
         }
         if (FrameLength() >= recordOffsetInFrame_ &&
             Frame()[recordOffsetInFrame_] == '\n') {
           ++recordOffsetInFrame_;
         }
         if (!pinnedFrame || mayPosition()) {
           frameOffsetInFile_ += recordOffsetInFrame_;
           recordOffsetInFrame_ = 0;
         }
       }
     }
-  }
     ++currentRecordNumber;
+  } else { // unformatted stream
+    furthestPositionInRecord =
+        std::max(furthestPositionInRecord, positionInRecord);
+    frameOffsetInFile_ += recordOffsetInFrame_ + furthestPositionInRecord;
+  }
   BeginRecord();
 }
 
 bool ExternalFileUnit::AdvanceRecord(IoErrorHandler &handler) {
   if (direction_ == Direction::Input) {
     FinishReadingRecord(handler);
     return BeginReadingRecord(handler);
   } else { // Direction::Output
     bool ok{true};
     RUNTIME_CHECK(handler, isUnformatted.has_value());
-    if (openRecl && access == Access::Direct) {
-      if (furthestPositionInRecord < *openRecl) {
+    positionInRecord = furthestPositionInRecord;
+    if (access == Access::Direct) {
+      if (furthestPositionInRecord <
+          openRecl.value_or(furthestPositionInRecord)) {
         // Pad remainder of fixed length record
         WriteFrame(
             frameOffsetInFile_, recordOffsetInFrame_ + *openRecl, handler);
         std::memset(Frame() + recordOffsetInFrame_ + furthestPositionInRecord,
             isUnformatted.value_or(false) ? 0 : ' ',
             *openRecl - furthestPositionInRecord);
         furthestPositionInRecord = *openRecl;
       }
-    } else {
-      positionInRecord = furthestPositionInRecord;
-      if (isUnformatted.value_or(false)) {
+    } else if (*isUnformatted) {
+      if (access == Access::Sequential) {
         // Append the length of a sequential unformatted variable-length record
         // as its footer, then overwrite the reserved first four bytes of the
         // record with its length as its header.  These four bytes were skipped
         // over in BeginUnformattedIO<Output>().
         // TODO: Break very large records up into subrecords with negative
         // headers &/or footers
         std::uint32_t length;
         length = furthestPositionInRecord - sizeof length;
         ok = ok &&
             Emit(reinterpret_cast<const char *>(&length), sizeof length,
                 sizeof length, handler);
         positionInRecord = 0;
         ok = ok &&
             Emit(reinterpret_cast<const char *>(&length), sizeof length,
                 sizeof length, handler);
       } else {
-        // Terminate formatted variable length record
-        ok = ok && Emit("\n", 1, 1, handler); // TODO: Windows CR+LF
-      }
-    }
+        // Unformatted stream: nothing to do
+      }
+    } else {
     if (IsAfterEndfile()) {
       return false;
     }
     CommitWrites();
-    impliedEndfile_ = true;
     ++currentRecordNumber;
+    if (access != Access::Direct) {
+      impliedEndfile_ = IsRecordFile();
       if (IsAtEOF()) {
         endfileRecordNumber.reset();
       }
+    }
     return ok;
   }
 }
 
 void ExternalFileUnit::BackspaceRecord(IoErrorHandler &handler) {
-  if (access != Access::Sequential) {
+  if (access == Access::Direct || !IsRecordFile()) {
     handler.SignalError(IostatBackspaceNonSequential,
-        "BACKSPACE(UNIT=%d) on non-sequential file", unitNumber());
+        "BACKSPACE(UNIT=%d) on direct-access file or unformatted stream",
+        unitNumber());
   } else {
     if (IsAfterEndfile()) {
       // BACKSPACE after explicit ENDFILE
       currentRecordNumber = *endfileRecordNumber;
     } else {
       DoImpliedEndfile(handler);
       if (frameOffsetInFile_ + recordOffsetInFrame_ > 0) {
         --currentRecordNumber;

 
 void ExternalFileUnit::FlushIfTerminal(IoErrorHandler &handler) {
   if (isTerminal()) {
     FlushOutput(handler);
   }
 }
 
 void ExternalFileUnit::Endfile(IoErrorHandler &handler) {
-  if (access != Access::Sequential) {
-    handler.SignalError(IostatEndfileNonSequential,
-        "ENDFILE(UNIT=%d) on non-sequential file", unitNumber());
+  if (access == Access::Direct) {
+    handler.SignalError(IostatEndfileDirect,
+        "ENDFILE(UNIT=%d) on direct-access file", unitNumber());
   } else if (!mayWrite()) {
     handler.SignalError(IostatEndfileUnwritable,
         "ENDFILE(UNIT=%d) on read-only file", unitNumber());
   } else if (IsAfterEndfile()) {
     // ENDFILE after ENDFILE
   } else {
     DoEndfile(handler);
+    if (access == Access::Sequential) {
       // Explicit ENDFILE leaves position *after* the endfile record
       RUNTIME_CHECK(handler, endfileRecordNumber.has_value());
       currentRecordNumber = *endfileRecordNumber + 1;
     }
   }
+}
 
 void ExternalFileUnit::Rewind(IoErrorHandler &handler) {
   if (access == Access::Direct) {
     handler.SignalError(IostatRewindNonSequential,
         "REWIND(UNIT=%d) on non-sequential file", unitNumber());
   } else {
-    DoImpliedEndfile(handler);
-    SetPosition(0);
+    SetPosition(0, handler);
     currentRecordNumber = 1;
   }
 }
 
+void ExternalFileUnit::SetPosition(std::int64_t pos, IoErrorHandler &handler) {
+  DoImpliedEndfile(handler);
+  frameOffsetInFile_ = pos;
+  recordOffsetInFrame_ = 0;
+  BeginRecord();
+}
+
 void ExternalFileUnit::EndIoStatement() {
   io_.reset();
   u_.emplace<std::monostate>();
   lock_.Drop();
 }
 
 void ExternalFileUnit::BeginSequentialVariableUnformattedInputRecord(
     IoErrorHandler &handler) {
if (error) {
     handler.SignalError(error, static_cast<std::intmax_t>(currentRecordNumber),
         static_cast<std::intmax_t>(frameOffsetInFile_),
         static_cast<std::intmax_t>(header), static_cast<std::intmax_t>(footer));
     // TODO: error recovery
   }
   positionInRecord = sizeof header;
 }
 
-void ExternalFileUnit::BeginSequentialVariableFormattedInputRecord(
+void ExternalFileUnit::BeginVariableFormattedInputRecord(
     IoErrorHandler &handler) {
   if (this == defaultInput) {
     if (defaultOutput) {
       defaultOutput->FlushOutput(handler);
     }
     if (errorOutput) {
       errorOutput->FlushOutput(handler);
     }
   }
   std::size_t length{0};
   do {
     std::size_t need{length + 1};
     length =
         ReadFrame(frameOffsetInFile_, recordOffsetInFrame_ + need, handler) -
         recordOffsetInFrame_;
     if (length < need) {
       if (length > 0) {
         // final record w/o \n
         recordLength = length;
       } else {
         handler.SignalEnd();
       }
       break;
     }
-  } while (!SetSequentialVariableFormattedRecordLength());
+  } while (!SetVariableFormattedRecordLength());
 }
 
 void ExternalFileUnit::BackspaceFixedRecord(IoErrorHandler &handler) {
   RUNTIME_CHECK(handler, openRecl.has_value());
   if (frameOffsetInFile_ < *openRecl) {
     handler.SignalError(IostatBackspaceAtFirstRecord);
   } else {
     frameOffsetInFile_ -= *openRecl;
if (*recordLength > 0 &&
       Frame()[recordOffsetInFrame_ + *recordLength - 1] == '\r') {
     --*recordLength;
   }
 }
 
 void ExternalFileUnit::DoImpliedEndfile(IoErrorHandler &handler) {
   if (impliedEndfile_) {
     impliedEndfile_ = false;
-    if (access == Access::Sequential && mayPosition()) {
+    if (access != Access::Direct && IsRecordFile() && mayPosition()) {
       DoEndfile(handler);
     }
   }
 }
 
 void ExternalFileUnit::DoEndfile(IoErrorHandler &handler) {
+  if (access == Access::Sequential) {
     endfileRecordNumber = currentRecordNumber;
+  }
+  FlushOutput(handler);
   Truncate(frameOffsetInFile_ + recordOffsetInFrame_, handler);
   BeginRecord();
   impliedEndfile_ = false;
 }
 
 void ExternalFileUnit::CommitWrites() {
   frameOffsetInFile_ +=
       recordOffsetInFrame_ + recordLength.value_or(furthestPositionInRecord);