[PoC][Flang] Process "line" Flang preprocessor directive

[domada]

Summary

Flang "line" preprocessor directive contains information about original source file. Currently this information is missed when user compiles code with -save-temps flag. In consequence we observe:

1. Different location of the syntax error. If -save-temps flag is attached then Flang reports error in the preprocessed file
2. Different debug info. If -save-temps flag is attached then debug info points to preprocessed code (not to the original source file)
3. Mismatch of OpenMP offloading functions names when -save-temps flag is attached because OpenMP offloading function names depend on the name of the source file. For target offloading Flang generates two preprocessed files (for host and the device).

This patch aims to solve these issues by adding support of Flang "line" directive. "line" directive holds information about original source file. If we process "line" directive then we can be sure that the location of the tokens is independent from the status of -save-temps flag.

Detailed discussion about Flang preprocessor and the issues related to token location for Flang and -save-temps file can be found here: https://discourse.llvm.org/t/rfc-flang-improve-debug-information-user-error-messages-and-fix-openmp-function-mismatch-for-save-temps-flag/71417

[kiranchandramohan]

flang/lib/Parser/preprocessor.cpp

421–422

Nit: Use braced initialization in the frontend here and elsewhere where it is applicable.

flang/test/Driver/save-temps-token-localization.f90

1

We need a test to show that this works as well.

[klausler]

flang/lib/Parser/preprocessor.cpp

467

Please check my understanding. Are you in fact trying to implement the '#line' preprocessing directive by actually *reading* the original file and prescanning it again to replace the content of the '*.I' file? And doing so without any of the original search path or predefined macro or line length or source format options?

[domada]

flang/lib/Parser/preprocessor.cpp

467

Hi,
thank you for your feedback. Could you provide some links to the resources/documentation/discussion about Flang preprocessor which will be helpful for implementation of the "line" directive? Could we modify the "line" directive in similar way to the linenum for GCC : https://gcc.gnu.org/onlinedocs/gcc-3.0.2/cpp_9.html ?

I called this patch as PoC just to get feedback if my approach is feasible. My goal is to force Flang to use original location of the tokens for -save-temps flag. Currently Flang use tokens from the preprocessor if the -save-temps flag is set. It causes several issues which does not appear in Clang. Please read the discussion: https://discourse.llvm.org/t/rfc-flang-improve-debug-information-user-error-messages-and-fix-openmp-function-mismatch-for-save-temps-flag/71417 to get more details about the issues connected with -save-temps flag.

[klausler]

flang/lib/Parser/preprocessor.cpp

467

My comment was a question, not "feedback", but I suppose the answer is a "yes". If so, this is not the right approach.

[klausler]

See https://reviews.llvm.org/D153910 for an implementation of #line.

[domada]

Patch not needed. Feature already implemented by https://reviews.llvm.org/D153910

flang/lib/Parser/preprocessor.cpp

 TokenSequence Preprocessor::ReplaceMacros(
     const TokenSequence &tokens, Prescanner &prescanner) {
   if (std::optional<TokenSequence> repl{MacroReplacement(tokens, prescanner)}) {
     return std::move(*repl);
   }
   return tokens;
 }
 
+static std::string getFileName(const std::string &fileNamePath,
+    const TokenSequence &dir, Prescanner &prescanner, size_t tokenPosition) {
+  int offset = 0;
+  int numberOfSkippedCharacters = 0;
+  // skip outer quotation marks
+  if (fileNamePath.length() > 2 && fileNamePath[0] == '\"') {
+    if (fileNamePath[fileNamePath.length() - 1] != '\"') {
+      prescanner.Say(
+          dir.GetTokenProvenanceRange(tokenPosition), "missing '\"'"_err_en_US);
+      return std::string();
+    } else {
+      offset = 1;
+      numberOfSkippedCharacters = 2;
+    }
+  }
+
+  return fileNamePath.substr(
+      offset, fileNamePath.length() - numberOfSkippedCharacters);
+}
+
 void Preprocessor::Directive(const TokenSequence &dir, Prescanner &prescanner) {
   std::size_t tokens{dir.SizeInTokens()};
   std::size_t j{dir.SkipBlanks(0)};
   if (j == tokens) {
     return;
   }
   if (dir.TokenAt(j).ToString() != "#") {
     prescanner.Say(dir.GetTokenProvenanceRange(j), "missing '#'"_err_en_US);
void Preprocessor::Directive(const TokenSequence &dir, Prescanner &prescanner) {
   std::size_t dirOffset{j};
   std::string dirName{ToLowerCaseLetters(dir.TokenAt(dirOffset).ToString())};
   j = dir.SkipBlanks(j + 1);
   CharBlock nameToken;
   if (j < tokens && IsLegalIdentifierStart(dir.TokenAt(j)[0])) {
     nameToken = dir.TokenAt(j);
   }
   if (dirName == "line") {
-    // #line is ignored
+    std::optional<std::string> prependPath;
+    if (const auto *currentFile{allSources_.GetSourceFile(j)}) {
+      prependPath = DirectoryName(currentFile->path());
+    }
+    std::string fileNamePath{dir.TokenAt(j).ToString()};
+    std::string fileName = getFileName(fileNamePath, dir, prescanner, j);
+    if (fileName.empty())
+      return;
+    std::string buf;
+    llvm::raw_string_ostream error{buf};
+    // Read original file to get propper token localization
+    const SourceFile *originalFile{
+        allSources_.Open(fileName, error, std::move(prependPath))};
+    if (!originalFile) {
+      prescanner.Say(dir.GetTokenProvenanceRange(j), "line error %s"_err_en_US,
+          error.str());
+    } else if (originalFile->bytes() > 0) {
+      ProvenanceRange fileRange{
+          allSources_.AddIncludedFile(*originalFile, dir.GetProvenanceRange())};
+      Prescanner{prescanner}
+          .set_encoding(originalFile->encoding())
+          .Prescan(fileRange);
+    }
+    // Skip tokens from preprocessed file which are located in the original file
+    while (!prescanner.IsAtEnd()) {
+      if (!prescanner.IsNextLinePreprocessorDirective()) {
+        prescanner.NextLine();
+        continue;
+      }
+    }
   } else if (dirName == "define") {
     if (nameToken.empty()) {
       prescanner.Say(dir.GetTokenProvenanceRange(j < tokens ? j : tokens - 1),
           "#define: missing or invalid name"_err_en_US);
       return;
     }
     nameToken = SaveTokenAsName(nameToken);
     definitions_.erase(nameToken);

flang/test/Driver/save-temps-token-localization.f90

+! Ensure -save-temps flag does not have influence on error diagnostic
+
+!--------------------------
+! FLANG DRIVER (flang)
+!--------------------------
+! RUN: not %flang -save-temps %s  2>&1 | FileCheck %s --check-prefix=WITH_SAVE_TEMPS
+
+
+! WITH_SAVE_TEMPS: save-temps-token-localization.f90:14:3: error: No explicit type declared for 'ij' 
+
+subroutine test
+  implicit none
+  integer :: i
+  ij=10
+  print *,i
+end subroutine