[preprocessor] Add an opportunity to retain excluded conditional blocks

[emankov]

Summary

It is handy for clang tooling, for instance, in source to source transformation.

[vpykhtin]

I'm not an expert in this part of the compiler, but looks good.

[erichkeane]

I don't have a good idea of the intent of this patch. It seems to me that this could do some pretty nasty damage to C code, right?

Can you give better examples/tests of where this is important or useful?

[emankov]

In D66597#1643060, @erichkeane wrote:

I don't have a good idea of the intent of this patch. It seems to me that this could do some pretty nasty damage to C code, right?

Right, it could as well as SingleFileParseMode flag.

Can you give better examples/tests of where this is important or useful?

Sure, as I mentioned in the description, source to source transformation; in my case CUDA -> HIP transformation.

CUDA:

#ifdef SOME_MACRO
  axpy_kernel <<<1, 3>>> (a, y, x);
#else
  axpy_kernel <<<1, 4>>> (a, x, y);
#endif

HIP:

#ifdef SOME_MACRO
  hipLaunchKernelGGL(axpy_kernel, dim3(1), dim3(3), 0, 0, a, y, x);
#else
  hipLaunchKernelGGL(axpy_kernel, dim3(1), dim3(4), 0, 0, a, x, y);
#endif

clang/include/clang-c/Index.h

enum CXTranslationUnit_Flags {
   /**
    * Used to indicate that non-errors from included files should be ignored.
    *
    * If set, clang_getDiagnosticSetFromTU() will not report e.g. warnings from
    * included files anymore. This speeds up clang_getDiagnosticSetFromTU() for
    * the case where these warnings are not of interest, as for an IDE for
    * example, which typically shows only the diagnostics in the main file.
    */
-  CXTranslationUnit_IgnoreNonErrorsFromIncludedFiles = 0x4000
+  CXTranslationUnit_IgnoreNonErrorsFromIncludedFiles = 0x4000,
+
+  /**
+   * Tells the preprocessor not to skip excluded conditional blocks.
+   */
+  CXTranslationUnit_RetainExcludedConditionalBlocks = 0x8000,
 };
 
 /**
  * Returns the set of flags that is suitable for parsing a translation
  * unit that is being edited.
  *
  * The set of flags returned provide options for \c clang_parseTranslationUnit()
  * to indicate that the translation unit is likely to be reparsed many times,

clang/include/clang/Frontend/ASTUnit.h

static ASTUnit *LoadFromCommandLine(
       TranslationUnitKind TUKind = TU_Complete,
       bool CacheCodeCompletionResults = false,
       bool IncludeBriefCommentsInCodeCompletion = false,
       bool AllowPCHWithCompilerErrors = false,
       SkipFunctionBodiesScope SkipFunctionBodies =
           SkipFunctionBodiesScope::None,
       bool SingleFileParse = false, bool UserFilesAreVolatile = false,
       bool ForSerialization = false,
+      bool RetainExcludedConditionalBlocks = false,
       llvm::Optional<StringRef> ModuleFormat = llvm::None,
       std::unique_ptr<ASTUnit> *ErrAST = nullptr,
       IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS = nullptr);
 
   /// Reparse the source files using the same command-line options that
   /// were originally used to produce this translation unit.
   ///
   /// \param VFS - A llvm::vfs::FileSystem to be used for all file accesses.

clang/include/clang/Lex/PreprocessorOptions.h

public:
   /// Whether the compiler instance should retain (i.e., not free)
   /// the buffers associated with remapped files.
   ///
   /// This flag defaults to false; it can be set true only through direct
   /// manipulation of the compiler invocation object, in cases where the
   /// compiler invocation and its buffers will be reused.
   bool RetainRemappedFileBuffers = false;
 
+  /// When enabled, excluded conditional blocks retain in the main file.
+  bool RetainExcludedConditionalBlocks = false;
+
   /// The Objective-C++ ARC standard library that we should support,
   /// by providing appropriate definitions to retrofit the standard library
   /// with support for lifetime-qualified pointers.
   ObjCXXARCStandardLibraryKind ObjCXXARCStandardLibrary = ARCXX_nolib;
 
   /// Records the set of modules
   class FailedModulesSet {
     llvm::StringSet<> Failed;
void resetNonModularOptions() {
     ChainedIncludes.clear();
     DumpDeserializedPCHDecls = false;
     ImplicitPCHInclude.clear();
     SingleFileParseMode = false;
     LexEditorPlaceholders = true;
     RetainRemappedFileBuffers = true;
     PrecompiledPreambleBytes.first = 0;
     PrecompiledPreambleBytes.second = false;
+    RetainExcludedConditionalBlocks = false;
   }
 };
 
 } // namespace clang
 
 #endif // LLVM_CLANG_LEX_PREPROCESSOROPTIONS_H_

clang/lib/Frontend/ASTUnit.cpp

ASTUnit *ASTUnit::LoadFromCommandLine(
     std::shared_ptr<PCHContainerOperations> PCHContainerOps,
     IntrusiveRefCntPtr<DiagnosticsEngine> Diags, StringRef ResourceFilesPath,
     bool OnlyLocalDecls, CaptureDiagsKind CaptureDiagnostics,
     ArrayRef<RemappedFile> RemappedFiles, bool RemappedFilesKeepOriginalName,
     unsigned PrecompilePreambleAfterNParses, TranslationUnitKind TUKind,
     bool CacheCodeCompletionResults, bool IncludeBriefCommentsInCodeCompletion,
     bool AllowPCHWithCompilerErrors, SkipFunctionBodiesScope SkipFunctionBodies,
     bool SingleFileParse, bool UserFilesAreVolatile, bool ForSerialization,
+    bool RetainExcludedConditionalBlocks,
     llvm::Optional<StringRef> ModuleFormat, std::unique_ptr<ASTUnit> *ErrAST,
     IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS) {
   assert(Diags.get() && "no DiagnosticsEngine was provided");
 
   SmallVector<StoredDiagnostic, 4> StoredDiagnostics;
 
   std::shared_ptr<CompilerInvocation> CI;
 
ASTUnit *ASTUnit::LoadFromCommandLine(
   for (const auto &RemappedFile : RemappedFiles) {
     CI->getPreprocessorOpts().addRemappedFile(RemappedFile.first,
                                               RemappedFile.second);
   }
   PreprocessorOptions &PPOpts = CI->getPreprocessorOpts();
   PPOpts.RemappedFilesKeepOriginalName = RemappedFilesKeepOriginalName;
   PPOpts.AllowPCHWithCompilerErrors = AllowPCHWithCompilerErrors;
   PPOpts.SingleFileParseMode = SingleFileParse;
+  PPOpts.RetainExcludedConditionalBlocks = RetainExcludedConditionalBlocks;
 
   // Override the resources path.
   CI->getHeaderSearchOpts().ResourceDir = ResourceFilesPath;
 
   CI->getFrontendOpts().SkipFunctionBodies =
       SkipFunctionBodies == SkipFunctionBodiesScope::PreambleAndMainFile;
 
   if (ModuleFormat)

clang/lib/Lex/PPDirectives.cpp

void Preprocessor::HandleIfdefDirective(Token &Result,
 
   if (Callbacks) {
     if (isIfndef)
       Callbacks->Ifndef(DirectiveTok.getLocation(), MacroNameTok, MD);
     else
       Callbacks->Ifdef(DirectiveTok.getLocation(), MacroNameTok, MD);
   }
 
+  bool RetainExcludedCB = PPOpts->RetainExcludedConditionalBlocks &&
+    getSourceManager().isInMainFile(DirectiveTok.getLocation());
+
   // Should we include the stuff contained by this directive?
   if (PPOpts->SingleFileParseMode && !MI) {
     // In 'single-file-parse mode' undefined identifiers trigger parsing of all
     // the directive blocks.
     CurPPLexer->pushConditionalLevel(DirectiveTok.getLocation(),
                                      /*wasskip*/false, /*foundnonskip*/false,
                                      /*foundelse*/false);
-  } else if (!MI == isIfndef) {
+  } else if (!MI == isIfndef || RetainExcludedCB) {
     // Yes, remember that we are inside a conditional, then lex the next token.
     CurPPLexer->pushConditionalLevel(DirectiveTok.getLocation(),
                                      /*wasskip*/false, /*foundnonskip*/true,
                                      /*foundelse*/false);
   } else {
     // No, skip the contents of this block.
     SkipExcludedConditionalBlock(HashToken.getLocation(),
                                  DirectiveTok.getLocation(),
else
       CurPPLexer->MIOpt.EnterTopLevelConditional();
   }
 
   if (Callbacks)
     Callbacks->If(
         IfToken.getLocation(), DER.ExprRange,
         (ConditionalTrue ? PPCallbacks::CVK_True : PPCallbacks::CVK_False));
 
+  bool RetainExcludedCB = PPOpts->RetainExcludedConditionalBlocks &&
+    getSourceManager().isInMainFile(IfToken.getLocation());
+
   // Should we include the stuff contained by this directive?
   if (PPOpts->SingleFileParseMode && DER.IncludedUndefinedIds) {
     // In 'single-file-parse mode' undefined identifiers trigger parsing of all
     // the directive blocks.
     CurPPLexer->pushConditionalLevel(IfToken.getLocation(), /*wasskip*/false,
                                      /*foundnonskip*/false, /*foundelse*/false);
-  } else if (ConditionalTrue) {
+  } else if (ConditionalTrue || RetainExcludedCB) {
     // Yes, remember that we are inside a conditional, then lex the next token.
     CurPPLexer->pushConditionalLevel(IfToken.getLocation(), /*wasskip*/false,
                                    /*foundnonskip*/true, /*foundelse*/false);
   } else {
     // No, skip the contents of this block.
     SkipExcludedConditionalBlock(HashToken.getLocation(), IfToken.getLocation(),
                                  /*Foundnonskip*/ false,
                                  /*FoundElse*/ false);
if (CurPPLexer->getConditionalStackDepth() == 0)
     CurPPLexer->MIOpt.EnterTopLevelConditional();
 
   // If this is a #else with a #else before it, report the error.
   if (CI.FoundElse) Diag(Result, diag::pp_err_else_after_else);
 
   if (Callbacks)
     Callbacks->Else(Result.getLocation(), CI.IfLoc);
 
-  if (PPOpts->SingleFileParseMode && !CI.FoundNonSkip) {
+  bool RetainExcludedCB = PPOpts->RetainExcludedConditionalBlocks &&
+    getSourceManager().isInMainFile(Result.getLocation());
+
+  if ((PPOpts->SingleFileParseMode && !CI.FoundNonSkip) || RetainExcludedCB) {
     // In 'single-file-parse mode' undefined identifiers trigger parsing of all
     // the directive blocks.
     CurPPLexer->pushConditionalLevel(CI.IfLoc, /*wasskip*/false,
                                      /*foundnonskip*/false, /*foundelse*/true);
     return;
   }
 
   // Finally, skip the rest of the contents of this block.
void Preprocessor::HandleElifDirective(Token &ElifToken,
 
   // If this is a #elif with a #else before it, report the error.
   if (CI.FoundElse) Diag(ElifToken, diag::pp_err_elif_after_else);
 
   if (Callbacks)
     Callbacks->Elif(ElifToken.getLocation(), ConditionRange,
                     PPCallbacks::CVK_NotEvaluated, CI.IfLoc);
 
-  if (PPOpts->SingleFileParseMode && !CI.FoundNonSkip) {
+  bool RetainExcludedCB = PPOpts->RetainExcludedConditionalBlocks &&
+    getSourceManager().isInMainFile(ElifToken.getLocation());
+
+  if ((PPOpts->SingleFileParseMode && !CI.FoundNonSkip) || RetainExcludedCB) {
     // In 'single-file-parse mode' undefined identifiers trigger parsing of all
     // the directive blocks.
     CurPPLexer->pushConditionalLevel(ElifToken.getLocation(), /*wasskip*/false,
                                      /*foundnonskip*/false, /*foundelse*/false);
     return;
   }
 
   // Finally, skip the rest of the contents of this block.
   SkipExcludedConditionalBlock(
       HashToken.getLocation(), CI.IfLoc, /*Foundnonskip*/ true,
       /*FoundElse*/ CI.FoundElse, ElifToken.getLocation());
 }

clang/test/Index/retain-excluded-conditional-blocks.m

+// RUN: c-index-test -retain-excluded-conditional-blocks %s | FileCheck %s
+
+#include <stdint.h>
+
+// CHECK: TypedefDecl=intptr_t
+
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=MyCls
+@interface MyCls
+// CHECK: [[@LINE+1]]:8: ObjCInstanceMethodDecl=some_meth
+-(void)some_meth;
+@end
+
+#if 1
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test1
+@interface Test1 @end
+#else
+// CHECK: [[@LINE+1]]:12:
+@interface Test2 @end
+#endif
+
+#if 0
+// CHECK: [[@LINE+1]]:12:
+@interface Test3 @end
+#else
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test4
+@interface Test4 @end
+#endif
+
+#if SOMETHING_NOT_DEFINED
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test5
+@interface Test5 @end
+#else
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test6
+@interface Test6 @end
+#endif
+
+#define SOMETHING_DEFINED 1
+#if SOMETHING_DEFINED
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test7
+@interface Test7 @end
+#else
+// CHECK: [[@LINE+1]]:12:
+@interface Test8 @end
+#endif
+
+#if defined(SOMETHING_NOT_DEFINED)
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test9
+@interface Test9 @end
+#else
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test10
+@interface Test10 @end
+#endif
+
+#if defined(SOMETHING_DEFINED)
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test11
+@interface Test11 @end
+#else
+// CHECK: [[@LINE+1]]:12:
+@interface Test12 @end
+#endif
+
+#if SOMETHING_NOT_DEFINED1
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test13
+@interface Test13 @end
+#elif SOMETHING_NOT_DEFINED2
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test14
+@interface Test14 @end
+#else
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test15
+@interface Test15 @end
+#endif
+
+#ifdef SOMETHING_NOT_DEFINED
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test19
+@interface Test19 @end
+#else
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test20
+@interface Test20 @end
+#endif
+
+#ifdef SOMETHING_DEFINED
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test21
+@interface Test21 @end
+#else
+// CHECK: [[@LINE+1]]:12:
+@interface Test22 @end
+#endif
+
+#ifndef SOMETHING_NOT_DEFINED
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test23
+@interface Test23 @end
+#else
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test24
+@interface Test24 @end
+#endif
+
+#ifndef SOMETHING_DEFINED
+// CHECK: [[@LINE+1]]:12:
+@interface Test25 @end
+#else
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test26
+@interface Test26 @end
+#endif
+
+#if 1 < SOMETHING_NOT_DEFINED
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test27
+@interface Test27 @end
+#else
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test28
+@interface Test28 @end
+#endif
+
+#if SOMETHING_NOT_DEFINED
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test29
+@interface Test29 @end
+#endif
+
+#ifdef SOMETHING_NOT_DEFINED
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test30
+@interface Test30 @end
+#endif
+
+#ifdef SOMETHING_DEFINED
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test31
+@interface Test31 @end
+#elif !defined(SOMETHING_NOT_DEFINED)
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test32
+@interface Test32 @end
+#else
+// CHECK: [[@LINE+1]]:12: ObjCInterfaceDecl=Test33
+@interface Test33 @end
+#endif

clang/tools/c-index-test/c-index-test.c

static int perform_single_file_parse(const char *filename) {
   }
 
   result = perform_test_load(Idx, TU, /*filter=*/"all", /*prefix=*/NULL, FilteredPrintingVisitor, /*PostVisit=*/NULL,
                              /*CommentSchemaFile=*/NULL);
   clang_disposeIndex(Idx);
   return result;
 }
 
+static int perform_file_retain_excluded_cb(const char *filename) {
+  CXIndex Idx;
+  CXTranslationUnit TU;
+  enum CXErrorCode Err;
+  int result;
+
+  Idx = clang_createIndex(/* excludeDeclsFromPCH */1,
+                          /* displayDiagnostics=*/1);
+
+  Err = clang_parseTranslationUnit2(Idx, filename,
+                                    /*command_line_args=*/NULL,
+                                    /*num_command_line_args=*/0,
+                                    /*unsaved_files=*/NULL,
+                                    /*num_unsaved_files=*/0,
+                                    CXTranslationUnit_RetainExcludedConditionalBlocks, &TU);
+  if (Err != CXError_Success) {
+    fprintf(stderr, "Unable to load translation unit!\n");
+    describeLibclangFailure(Err);
+    clang_disposeIndex(Idx);
+    return 1;
+  }
+
+  result = perform_test_load(Idx, TU, /*filter=*/"all", /*prefix=*/NULL, FilteredPrintingVisitor, /*PostVisit=*/NULL,
+                             /*CommentSchemaFile=*/NULL);
+  clang_disposeIndex(Idx);
+  return result;
+}
+
 /******************************************************************************/
 /* Logic for testing clang_getCursor().                                       */
 /******************************************************************************/
 
 static void print_cursor_file_scan(CXTranslationUnit TU, CXCursor cursor,
                                    unsigned start_line, unsigned start_col,
                                    unsigned end_line, unsigned end_col,
                                    const char *prefix) {
if (strstr(argv[1], "-memory-usage"))
       postVisit = PrintMemoryUsage;
     
     if (I)
       return perform_test_load_source(argc - 3, argv + 3, argv[2], I,
                                       postVisit);
   }
   else if (argc >= 3 && strcmp(argv[1], "-single-file-parse") == 0)
     return perform_single_file_parse(argv[2]);
+  else if (argc >= 3 && strcmp(argv[1], "-retain-excluded-conditional-blocks") == 0)
+    return perform_file_retain_excluded_cb(argv[2]);
   else if (argc >= 4 && strcmp(argv[1], "-test-file-scan") == 0)
     return perform_file_scan(argv[2], argv[3],
                              argc >= 5 ? argv[4] : 0);
   else if (argc > 2 && strstr(argv[1], "-test-annotate-tokens=") == argv[1])
     return perform_token_annotation(argc, argv);
   else if (argc > 2 && strcmp(argv[1], "-test-inclusion-stack-source") == 0)
     return perform_test_load_source(argc - 2, argv + 2, "all", NULL,
                                     PrintInclusionStack);

clang/tools/libclang/CIndex.cpp

TranslationUnitKind TUKind
     = (options & (CXTranslationUnit_Incomplete |
                   CXTranslationUnit_SingleFileParse))? TU_Prefix : TU_Complete;
   bool CacheCodeCompletionResults
     = options & CXTranslationUnit_CacheCompletionResults;
   bool IncludeBriefCommentsInCodeCompletion
     = options & CXTranslationUnit_IncludeBriefCommentsInCodeCompletion;
   bool SingleFileParse = options & CXTranslationUnit_SingleFileParse;
   bool ForSerialization = options & CXTranslationUnit_ForSerialization;
+  bool RetainExcludedCB = options &
+    CXTranslationUnit_RetainExcludedConditionalBlocks;
   SkipFunctionBodiesScope SkipFunctionBodies = SkipFunctionBodiesScope::None;
   if (options & CXTranslationUnit_SkipFunctionBodies) {
     SkipFunctionBodies =
         (options & CXTranslationUnit_LimitSkipFunctionBodiesToPreamble)
             ? SkipFunctionBodiesScope::Preamble
             : SkipFunctionBodiesScope::PreambleAndMainFile;
   }
 
clang_parseTranslationUnit_Impl(CXIndex CIdx, const char *source_filename,
   std::unique_ptr<ASTUnit> Unit(ASTUnit::LoadFromCommandLine(
       Args->data(), Args->data() + Args->size(),
       CXXIdx->getPCHContainerOperations(), Diags,
       CXXIdx->getClangResourcesPath(), CXXIdx->getOnlyLocalDecls(),
       CaptureDiagnostics, *RemappedFiles.get(),
       /*RemappedFilesKeepOriginalName=*/true, PrecompilePreambleAfterNParses,
       TUKind, CacheCodeCompletionResults, IncludeBriefCommentsInCodeCompletion,
       /*AllowPCHWithCompilerErrors=*/true, SkipFunctionBodies, SingleFileParse,
-      /*UserFilesAreVolatile=*/true, ForSerialization,
+      /*UserFilesAreVolatile=*/true, ForSerialization, RetainExcludedCB,
       CXXIdx->getPCHContainerOperations()->getRawReader().getFormat(),
       &ErrUnit));
 
   // Early failures in LoadFromCommandLine may return with ErrUnit unset.
   if (!Unit && !ErrUnit)
     return CXError_ASTReadError;
 
   if (NumErrors != Diags->getClient()->getNumErrors()) {