[Clang][Flang][AMDGPU] Add support for AMDGPU to Flang driver

[domada]

Summary

Scope of changes:

1. Extract common code between Clang and Flang for parsing AMDGPU features
2. Add function which adds implicit target features for AMDGPU as Clang does
3. Add AMDGPU target as one of valid targets for Flang

[domada]

Fixed formatting

[domada]

Patch rebased

[domada]

D146612 presents the lowering from MLIR attributes to LLVM IR.

[awarzynski]

Really nice to see some shared code being elevated out of Clang into LLVM, thanks!

I've only reviewed on the Flang driver changes. I will let the OpenMP experts to review the remaining bits. All in all looks good, I've only made some small suggestions.

Thanks for working on this!

flang/lib/Frontend/FrontendActions.cpp

93

This method could be simplified by following https://llvm.org/docs/CodingStandards.html#use-early-exits-and-continue-to-simplify-code. For example:

std::string CodeGenAction::getAllTargetFeatures()  {
  if (!triple.isAMDGPU()) {
    allFeaturesStr = llvm::join(targetOpts.featuresAsWritten.begin(),
                                targetOpts.featuresAsWritten.end(), ",");
    return allFeaturesStr;
  }

  // The logic for AMDGPU
  // Perhaps add a dedicated hook: getExplicitAndImplicitAMDGPUTargetFeatures()
}

Btw, this method does different things depending on the triple. Perhaps rename it as something more generic, e.g. getTargetFeatures? I think that the current name, getAllTargetFeatures, is a bit misleading (i.e. what does "all" mean?).

Also:

• make it static
• document

[domada]

Applied remarks.
Moved OpenMP changes to https://reviews.llvm.org/D146612 .

[domada]

flang/lib/Frontend/FrontendActions.cpp

93

Hi,
thanks for the review. I applied your remarks. I also moved OpenMP related changes to the child review.

[awarzynski]

A few more comments, but mostly nits. Btw, is this patch sufficient to generate code for AMDGPU? Or, put differently, what's the level of support atm?

clang/lib/Driver/ToolChains/Flang.cpp

107

Should there be a test for this triple as well?

flang/lib/Frontend/FrontendActions.cpp

134–137

[nit] I would probably flip this as:

if (triple.isAMDGPU()) {
  // Clang does not append all target features to the clang -cc1 invocation.
  // Some AMDGPU features are passed implicitly by the Clang frontend.
  // That's why we need to extract implicit AMDGPU target features and add
  // them to the target features specified by the user
  return getExplicitAndImplicitAMDGPUTargetFeatures(ci, targetOpts, triple);
  }

I know that I suggested the opposite in my previous comment, but you have simplified this since :) In any case, feel free to ignore.

139–142

[nit] IMHO this is documenting an implementation detail that would be more relevant inside getExplicitAndImplicitAMDGPUTargetFeatures.

More importantly, you are suggesting that the driver is doing whatever it is doing "because that's what Clang does". Consistency with Clang is important (you could call it out in the commit message) :) However, it would be even nicer to understand the actual rationale behind these implicit features. Any idea? Perhaps there are some clues in git history?

Also, perhaps a TODO to share this code with Clang? (to make it even clearer that the frontend driver aims for full consistency with Clang here).

[domada]

flang/lib/Frontend/FrontendActions.cpp

139–142

I think that the main difference between Clang and Flang is the lack of TargetInfo class.

TargetInfo classes are Clang specific and they are responsible for parsing/adding default target features. Every target performs initialization in different way (compare for example AArch64 vs AMDGPU target initialization.

I don't want to make TargetInfo class Clang indendent (see discussion: https://discourse.llvm.org/t/rfc-targetinfo-library/64342 ). I also don't want to reimplement the whole TargetInfo class in Flang, because Flang already uses LLVM TargetMachine class (see: https://llvm.org/doxygen/classllvm_1_1TargetMachine.html and https://github.com/llvm/llvm-project/blob/main/flang/lib/Frontend/FrontendActions.cpp#L614 ) which can play similar role as Clang TargetInfo IMO.

That's why I decided to implement getExplicitAndImplicitAMDGPUTargetFeatures function which performs initialization of default AMDGPU target features.

[awarzynski]

flang/lib/Frontend/FrontendActions.cpp

139–142

Thanks for this comprehensive overview! It would be helpful if this rationale was included in the summary (in the spirit of documenting things for our future selves).

So, there isn't anything special about AMDGPU here, is there? We will have to implement similar hooks for other targets at some point too, right? Or perhaps there's some reason to do this for AMDGPU sooner rather than later?

I'm not against this change, just want to better understand the wider context.

[domada]

Rebase & applied review remarks

[domada]

flang/lib/Frontend/FrontendActions.cpp

139–142

Hi,
I modified the comment above to be more informative.

IMO, we need to add similar hooks for other targets. For example:

clang --target=aarch64 t.c -S -emit-llvm -v 
// I see in the logs explicit target features:
// +neon,  +v8a ,  -fmv
// However, generated t.ll contains 4 target features:
"target-features"="+fp-armv8,+neon,+v8a,-fmv"
// It looks like target feature +fp-armv8 is implicit

[yaxunl]

LGTM about the AMDGPU TargetInfo change.

[tschuett]

Do you want to move the AMDGPU changes into AMDGPU.cpp next to AMD.cpp? From the conversation, there seems to be more target specific behaviours.

[yaxunl]

In D145579#4224157, @tschuett wrote:

Do you want to move the AMDGPU changes into AMDGPU.cpp next to AMD.cpp? From the conversation, there seems to be more target specific behaviours.

I prefer to defer further refactoring to the future, so that downstream branches have time to digest this change first.

[domada]

Patch rebased

[awarzynski]

Thanks for the updates, mostly looks good. Just a couple of extra questions about the test coverage.

flang/lib/Frontend/FrontendActions.cpp

106–110

108

Are you able to add a test that will trigger this?

139–142

Thanks for checking!

flang/test/Driver/target-cpu-features.f90

57

Hm, there aren't any "implicit" target features here.

[domada]

In D145579#4224157, @tschuett wrote:

Do you want to move the AMDGPU changes into AMDGPU.cpp next to AMD.cpp? From the conversation, there seems to be more target specific behaviours.

@tschuett No. I don't plan to further refactor AMDGPU::TargeTInfo Clang class. My current goal is to add function attributes to the LLVM IR for Fortran OpenMP code and I don't need to make more changes in Clang to finish my goal.

[tschuett]

I wanted to ask whether you want to put an AMDGPU.cpp and AMD.cpp file in the flang/lib/Frontend directory.

[domada]

Patch rebased and added new test for checking incorrect wavefront sizes AMDGPU target features.

[domada]

flang/lib/Frontend/FrontendActions.cpp

108

Yes, please check the newest patch.

flang/test/Driver/target-cpu-features.f90

57

You will see them in MLIR code:
https://reviews.llvm.org/D146612#change-ciOHRHlq0yvL (file: mlir/test/Target/LLVMIR/openmp-llvm.mlir )
Clang also does not list these features in command line for AMDGPU. Slightly different situation is for example for ARM target. Clang list 3 options as -cc1 options and it attaches 4 target options to the generated LLVM IR:

clang --target=aarch64 t.c -S -emit-llvm -v 
// I see in the command line 3 explicit target features:
// +neon,  +v8a ,  -fmv
// However, generated t.ll contains 4 target features:
"target-features"="+fp-armv8,+neon,+v8a,-fmv"
// It looks like target feature +fp-armv8 is implicit

[domada]

In D145579#4226542, @tschuett wrote:

I wanted to ask whether you want to put an AMDGPU.cpp and AMD.cpp file in the flang/lib/Frontend directory.

@tschuett No, I don't plan to modify further flang/lib/Frontend.

[awarzynski]

Thanks for implementing this, LGTM!

clang/lib/Basic/Targets/AMDGPU.cpp

 
 ArrayRef<const char *> AMDGPUTargetInfo::getGCCRegNames() const {
   return llvm::ArrayRef(GCCRegNames);
 }
 
 bool AMDGPUTargetInfo::initFeatureMap(
     llvm::StringMap<bool> &Features, DiagnosticsEngine &Diags, StringRef CPU,
     const std::vector<std::string> &FeatureVec) const {
-  const bool IsNullCPU = CPU.empty();
-  bool IsWave32Capable = false;
 
   using namespace llvm::AMDGPU;
-
-  // XXX - What does the member GPU mean if device name string passed here?
-  if (isAMDGCN(getTriple())) {
-    switch (llvm::AMDGPU::parseArchAMDGCN(CPU)) {
-    case GK_GFX1103:
-    case GK_GFX1102:
-    case GK_GFX1101:
-    case GK_GFX1100:
-      IsWave32Capable = true;
-      Features["ci-insts"] = true;
-      Features["dot5-insts"] = true;
-      Features["dot7-insts"] = true;
-      Features["dot8-insts"] = true;
-      Features["dot9-insts"] = true;
-      Features["dot10-insts"] = true;
-      Features["dl-insts"] = true;
-      Features["16-bit-insts"] = true;
-      Features["dpp"] = true;
-      Features["gfx8-insts"] = true;
-      Features["gfx9-insts"] = true;
-      Features["gfx10-insts"] = true;
-      Features["gfx10-3-insts"] = true;
-      Features["gfx11-insts"] = true;
-      break;
-    case GK_GFX1036:
-    case GK_GFX1035:
-    case GK_GFX1034:
-    case GK_GFX1033:
-    case GK_GFX1032:
-    case GK_GFX1031:
-    case GK_GFX1030:
-      IsWave32Capable = true;
-      Features["ci-insts"] = true;
-      Features["dot1-insts"] = true;
-      Features["dot2-insts"] = true;
-      Features["dot5-insts"] = true;
-      Features["dot6-insts"] = true;
-      Features["dot7-insts"] = true;
-      Features["dot10-insts"] = true;
-      Features["dl-insts"] = true;
-      Features["16-bit-insts"] = true;
-      Features["dpp"] = true;
-      Features["gfx8-insts"] = true;
-      Features["gfx9-insts"] = true;
-      Features["gfx10-insts"] = true;
-      Features["gfx10-3-insts"] = true;
-      Features["s-memrealtime"] = true;
-      Features["s-memtime-inst"] = true;
-      break;
-    case GK_GFX1012:
-    case GK_GFX1011:
-      Features["dot1-insts"] = true;
-      Features["dot2-insts"] = true;
-      Features["dot5-insts"] = true;
-      Features["dot6-insts"] = true;
-      Features["dot7-insts"] = true;
-      Features["dot10-insts"] = true;
-      [[fallthrough]];
-    case GK_GFX1013:
-    case GK_GFX1010:
-      IsWave32Capable = true;
-      Features["dl-insts"] = true;
-      Features["ci-insts"] = true;
-      Features["16-bit-insts"] = true;
-      Features["dpp"] = true;
-      Features["gfx8-insts"] = true;
-      Features["gfx9-insts"] = true;
-      Features["gfx10-insts"] = true;
-      Features["s-memrealtime"] = true;
-      Features["s-memtime-inst"] = true;
-      break;
-    case GK_GFX940:
-      Features["gfx940-insts"] = true;
-      Features["fp8-insts"] = true;
-      [[fallthrough]];
-    case GK_GFX90A:
-      Features["gfx90a-insts"] = true;
-      [[fallthrough]];
-    case GK_GFX908:
-      Features["dot3-insts"] = true;
-      Features["dot4-insts"] = true;
-      Features["dot5-insts"] = true;
-      Features["dot6-insts"] = true;
-      Features["mai-insts"] = true;
-      [[fallthrough]];
-    case GK_GFX906:
-      Features["dl-insts"] = true;
-      Features["dot1-insts"] = true;
-      Features["dot2-insts"] = true;
-      Features["dot7-insts"] = true;
-      Features["dot10-insts"] = true;
-      [[fallthrough]];
-    case GK_GFX90C:
-    case GK_GFX909:
-    case GK_GFX904:
-    case GK_GFX902:
-    case GK_GFX900:
-      Features["gfx9-insts"] = true;
-      [[fallthrough]];
-    case GK_GFX810:
-    case GK_GFX805:
-    case GK_GFX803:
-    case GK_GFX802:
-    case GK_GFX801:
-      Features["gfx8-insts"] = true;
-      Features["16-bit-insts"] = true;
-      Features["dpp"] = true;
-      Features["s-memrealtime"] = true;
-      [[fallthrough]];
-    case GK_GFX705:
-    case GK_GFX704:
-    case GK_GFX703:
-    case GK_GFX702:
-    case GK_GFX701:
-    case GK_GFX700:
-      Features["ci-insts"] = true;
-      [[fallthrough]];
-    case GK_GFX602:
-    case GK_GFX601:
-    case GK_GFX600:
-      Features["s-memtime-inst"] = true;
-      break;
-    case GK_NONE:
-      break;
-    default:
-      llvm_unreachable("Unhandled GPU!");
-    }
-  } else {
-    if (CPU.empty())
-      CPU = "r600";
-
-    switch (llvm::AMDGPU::parseArchR600(CPU)) {
-    case GK_CAYMAN:
-    case GK_CYPRESS:
-    case GK_RV770:
-    case GK_RV670:
-      // TODO: Add fp64 when implemented.
-      break;
-    case GK_TURKS:
-    case GK_CAICOS:
-    case GK_BARTS:
-    case GK_SUMO:
-    case GK_REDWOOD:
-    case GK_JUNIPER:
-    case GK_CEDAR:
-    case GK_RV730:
-    case GK_RV710:
-    case GK_RS880:
-    case GK_R630:
-    case GK_R600:
-      break;
-    default:
-      llvm_unreachable("Unhandled GPU!");
-    }
-  }
-
+  fillAMDGPUFeatureMap(CPU, getTriple(), Features);
   if (!TargetInfo::initFeatureMap(Features, Diags, CPU, FeatureVec))
     return false;
 
-  // FIXME: Not diagnosing wavefrontsize32 on wave64 only targets.
-  const bool HaveWave32 =
-      (IsWave32Capable || IsNullCPU) && Features.count("wavefrontsize32");
-  const bool HaveWave64 = Features.count("wavefrontsize64");
-
   // TODO: Should move this logic into TargetParser
-  if (HaveWave32 && HaveWave64) {
-    Diags.Report(diag::err_invalid_feature_combination)
-        << "'wavefrontsize32' and 'wavefrontsize64' are mutually exclusive";
+  std::string ErrorMsg;
+  if (!insertWaveSizeFeature(CPU, getTriple(), Features, ErrorMsg)) {
+    Diags.Report(diag::err_invalid_feature_combination) << ErrorMsg;
     return false;
   }
 
-  // Don't assume any wavesize with an unknown subtarget.
-  if (!IsNullCPU) {
-    // Default to wave32 if available, or wave64 if not
-    if (!HaveWave32 && !HaveWave64) {
-      StringRef DefaultWaveSizeFeature =
-          IsWave32Capable ? "wavefrontsize32" : "wavefrontsize64";
-      Features.insert(std::make_pair(DefaultWaveSizeFeature, true));
-    }
-  }
-
   return true;
 }
 
 void AMDGPUTargetInfo::fillValidCPUList(
     SmallVectorImpl<StringRef> &Values) const {
   if (isAMDGCN(getTriple()))
     llvm::AMDGPU::fillValidArchListAMDGCN(Values);
   else

clang/lib/Driver/ToolChains/CommonArgs.cpp

return llvm::StringSwitch<std::string>(GPUName)
         .Cases("rv610", "rv620", "rs780", "rs880")
         .Case("rv740", "rv770")
         .Case("palm", "cedar")
         .Cases("sumo", "sumo2", "sumo")
         .Case("hemlock", "cypress")
         .Case("aruba", "cayman")
         .Default(GPUName.str());
   }
+  if (Arg *A = Args.getLastArg(options::OPT_march_EQ)) {
+    return getProcessorFromTargetID(T, A->getValue()).str();
+  }
   return "";
 }
 
 static std::string getLanaiTargetCPU(const ArgList &Args) {
   if (Arg *A = Args.getLastArg(options::OPT_mcpu_EQ)) {
     return A->getValue();
   }
   return "";

clang/lib/Driver/ToolChains/Flang.cpp

if (!CPU.empty()) {
     CmdArgs.push_back("-target-cpu");
     CmdArgs.push_back(Args.MakeArgString(CPU));
   }
 
   // Add the target features.
   switch (TC.getArch()) {
   default:
     break;
+  case llvm::Triple::r600:
+  case llvm::Triple::amdgcn:
   case llvm::Triple::aarch64:
   case llvm::Triple::riscv64:
   case llvm::Triple::x86_64:
     getTargetFeatures(D, Triple, Args, CmdArgs, /*ForAs*/ false);
     break;
   }
 
   // TODO: Add target specific flags, ABI, mtune option etc.

flang/include/flang/Frontend/FrontendActions.h

class CodeGenAction : public FrontendAction {
   void executeAction() override;
   /// Runs prescan, parsing, sema and lowers to MLIR.
   bool beginSourceFileAction() override;
   /// Sets up LLVM's TargetMachine.
   bool setUpTargetMachine();
   /// Runs the optimization (aka middle-end) pipeline on the LLVM module
   /// associated with this action.
   void runOptimizationPipeline(llvm::raw_pwrite_stream &os);
+  /// Produces the string which represents all target features
+  std::string getAllTargetFeatures();
 
 protected:
   CodeGenAction(BackendActionTy act) : action{act} {};
   /// @name MLIR
   /// {
   std::unique_ptr<mlir::ModuleOp> mlirModule;
   std::unique_ptr<mlir::MLIRContext> mlirCtx;
   /// }

flang/lib/Frontend/FrontendActions.cpp

 #include "llvm/MC/TargetRegistry.h"
 #include "llvm/Object/OffloadBinary.h"
 #include "llvm/Passes/PassBuilder.h"
 #include "llvm/Passes/PassPlugin.h"
 #include "llvm/Passes/StandardInstrumentations.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/SourceMgr.h"
 #include "llvm/Target/TargetMachine.h"
+#include "llvm/TargetParser/TargetParser.h"
 #include "llvm/Transforms/Utils/ModuleUtils.h"
 #include <memory>
 
 using namespace Fortran::frontend;
 
 // Declare plugin extension function declarations.
 #define HANDLE_EXTENSION(Ext)                                                  \
   llvm::PassPluginLibraryInfo get##Ext##PluginInfo();
bool PrescanAndSemaDebugAction::beginSourceFileAction() {
   // semantic checks are made to succeed unconditionally to prevent this action
   // from exiting early (i.e. in the presence of semantic errors). We should
   // never do this in actions intended for end-users or otherwise regular
   // compiler workflows!
   return runPrescan() && runParse() && (runSemanticChecks() || true) &&
          (generateRtTypeTables() || true);
 }
 
+std::string CodeGenAction::getAllTargetFeatures() {
+  std::string allFeaturesStr;
+  CompilerInstance &ci = this->getInstance();
+  const TargetOptions &targetOpts = ci.getInvocation().getTargetOpts();
+  const llvm::Triple triple(targetOpts.triple);
+
+  // Clang does not append all target features to the clang -cc1 invocation.
+  // Some AMDGPU features are passed implicitly by the Clang frontend.
+  // That's why we need to extract implicit AMDGPU target features and add
+  // them to the target features specified by the user
+  if (triple.isAMDGPU()) {
+    llvm::StringRef cpu = targetOpts.cpu;
+    llvm::StringMap<bool> implicitFeaturesMap;
+    std::string errorMsg;
+    // Get the set of implicit target features
+    llvm::AMDGPU::fillAMDGPUFeatureMap(cpu, triple, implicitFeaturesMap);
+    if (!llvm::AMDGPU::insertWaveSizeFeature(cpu, triple, implicitFeaturesMap,
+                                             errorMsg)) {
+      llvm::SMDiagnostic err;
+      CompilerInstance &ci = this->getInstance();
+      err.print(errorMsg.data(), llvm::errs());
+      unsigned diagID = ci.getDiagnostics().getCustomDiagID(
+          clang::DiagnosticsEngine::Error, "Unsupported feature ID");
+      ci.getDiagnostics().Report(diagID);
+      return allFeaturesStr;
+    }
+
+    // Add target features specified by the user
+    for (auto &userFeature : targetOpts.featuresAsWritten) {
+      std::string userKeyString = userFeature.substr(1);
+      implicitFeaturesMap[userKeyString] = (userFeature[0] == '+');
+    }
+    llvm::SmallVector<std::string> featuresVec;
+    for (auto &implicitFeatureItem : implicitFeaturesMap) {
+      featuresVec.push_back(
+          (llvm::Twine(implicitFeatureItem.second ? "+" : "-") +
+           implicitFeatureItem.first().str())
+              .str());
+    }
+
+    allFeaturesStr = llvm::join(featuresVec, ",");
+  } else {
+    allFeaturesStr = llvm::join(targetOpts.featuresAsWritten.begin(),
+                                targetOpts.featuresAsWritten.end(), ",");
+  }
+  return allFeaturesStr;
+}
+
 static void setMLIRDataLayout(mlir::ModuleOp &mlirModule,
                               const llvm::DataLayout &dl) {
   mlir::MLIRContext *context = mlirModule.getContext();
   mlirModule->setAttr(
       mlir::LLVM::LLVMDialect::getDataLayoutAttrName(),
       mlir::StringAttr::get(context, dl.getStringRepresentation()));
   mlir::DataLayoutSpecInterface dlSpec = mlir::translateDataLayout(dl, context);
   mlirModule->setAttr(mlir::DLTIDialect::kDataLayoutAttrName, dlSpec);
 }
 
lower::LoweringBridge lb = Fortran::lower::LoweringBridge::create(
       ci.getInvocation().getSemanticsContext().targetCharacteristics(),
       ci.getParsing().allCooked(), ci.getInvocation().getTargetOpts().triple,
       kindMap, ci.getInvocation().getLoweringOpts(),
       ci.getInvocation().getFrontendOpts().envDefaults);
 
   // Fetch module from lb, so we can set
   mlirModule = std::make_unique<mlir::ModuleOp>(lb.getModule());
 
+  if (!setUpTargetMachine())
+    return false;
+
   if (ci.getInvocation().getFrontendOpts().features.IsEnabled(
           Fortran::common::LanguageFeature::OpenMP)) {
     mlir::omp::OpenMPDialect::setIsDevice(
         *mlirModule, ci.getInvocation().getLangOpts().OpenMPIsDevice);
+    mlir::omp::OpenMPDialect::setTargetCpu(*mlirModule, tm->getTargetCPU());
+    mlir::omp::OpenMPDialect::setTargetCpuFeatures(
+        *mlirModule, tm->getTargetFeatureString());
   }
 
-  if (!setUpTargetMachine())
-    return false;
   const llvm::DataLayout &dl = tm->createDataLayout();
   setMLIRDataLayout(*mlirModule, dl);
 
   // Create a parse tree and lower it to FIR
   Fortran::parser::Program &parseTree{*ci.getParsing().parseTree()};
   lb.lower(parseTree, ci.getInvocation().getSemanticsContext());
 
   // run the default passes.
bool CodeGenAction::setUpTargetMachine() {
   }
 
   // Create `TargetMachine`
   const auto &CGOpts = ci.getInvocation().getCodeGenOpts();
   std::optional<llvm::CodeGenOpt::Level> OptLevelOrNone =
       llvm::CodeGenOpt::getLevel(CGOpts.OptimizationLevel);
   assert(OptLevelOrNone && "Invalid optimization level!");
   llvm::CodeGenOpt::Level OptLevel = *OptLevelOrNone;
-  std::string featuresStr = llvm::join(targetOpts.featuresAsWritten.begin(),
-                                       targetOpts.featuresAsWritten.end(), ",");
+  std::string featuresStr = getAllTargetFeatures();
   tm.reset(theTarget->createTargetMachine(
       theTriple, /*CPU=*/targetOpts.cpu,
       /*Features=*/featuresStr, llvm::TargetOptions(),
       /*Reloc::Model=*/CGOpts.getRelocationModel(),
       /*CodeModel::Model=*/std::nullopt, OptLevel));
   assert(tm && "Failed to create TargetMachine");
   return true;
 }

flang/test/Driver/target-cpu-features.f90

 ! RUN: | FileCheck %s -check-prefix=CHECK-SKYLAKE
 
 ! RUN: %flang --target=x86_64h-linux-gnu -c %s -### 2>&1 \
 ! RUN: | FileCheck %s -check-prefix=CHECK-X86_64H
 
 ! RUN: %flang --target=riscv64-linux-gnu -c %s -### 2>&1 \
 ! RUN: | FileCheck %s -check-prefix=CHECK-RV64
 
+! RUN: %flang --target=amdgcn-amd-amdhsa -mcpu=gfx908 -c %s -### 2>&1 \
+! RUN: | FileCheck %s -check-prefix=CHECK-AMDGPU
+
 
 ! CHECK-A57: "-fc1" "-triple" "aarch64-unknown-linux-gnu"
 ! CHECK-A57-SAME: "-target-cpu" "cortex-a57" "-target-feature" "+v8a" "-target-feature" "+aes" "-target-feature" "+crc" "-target-feature" "+fp-armv8" "-target-feature" "+sha2" "-target-feature" "+neon"
 
 ! CHECK-A76: "-fc1" "-triple" "aarch64-unknown-linux-gnu"
 ! CHECK-A76-SAME: "-target-cpu" "cortex-a76" "-target-feature" "+v8.2a" "-target-feature" "+aes" "-target-feature" "+crc" "-target-feature" "+dotprod" "-target-feature" "+fp-armv8" "-target-feature" "+fullfp16" "-target-feature" "+lse" "-target-feature" "+ras" "-target-feature" "+rcpc" "-target-feature" "+rdm" "-target-feature" "+sha2" "-target-feature" "+neon" "-target-feature" "+ssbs"
 
 ! CHECK-ARMV9: "-fc1" "-triple" "aarch64-unknown-linux-gnu"
 ! CHECK-ARMV9-SAME: "-target-cpu" "generic" "-target-feature" "+neon" "-target-feature" "+v9a" "-target-feature" "+sve" "-target-feature" "+sve2"
 
 ! CHECK-NO-A57: "-fc1" "-triple" "x86_64-unknown-linux-gnu"
 ! CHECK-NO-A57-NOT: cortex-a57
 ! CHECK-NO-A57-SAME: "-target-cpu" "x86-64"
 ! CHECK-NO-A57-NOT: cortex-a57
 
 ! CHECK-SKYLAKE: "-fc1" "-triple" "x86_64-unknown-linux-gnu"
 ! CHECK-SKYLAKE-SAME: "-target-cpu" "skylake"
 
 ! CHECK-X86_64H: "-fc1" "-triple" "x86_64h-unknown-linux-gnu"
 ! CHECK-X86_64H-SAME: "-target-cpu" "x86-64" "-target-feature" "-rdrnd" "-target-feature" "-aes" "-target-feature" "-pclmul" "-target-feature" "-rtm" "-target-feature" "-fsgsbase"
 
 ! CHECK-RV64: "-fc1" "-triple" "riscv64-unknown-linux-gnu"
 ! CHECK-RV64-SAME: "-target-cpu" "generic-rv64" "-target-feature" "+m" "-target-feature" "+a" "-target-feature" "+f" "-target-feature" "+d" "-target-feature" "+c"
+
+! CHECK-AMDGPU: "-fc1" "-triple" "amdgcn-amd-amdhsa"
+! CHECK-AMDGPU-SAME: "-target-cpu" "gfx908"

flang/test/Lower/OpenMP/target_cpu_features.f90

+!REQUIRES: amdgpu-registered-target
+!RUN: %flang_fc1 -emit-fir -triple amdgcn-amd-amdhsa -target-cpu gfx908 -fopenmp %s -o - | FileCheck %s
+
+!===============================================================================
+! Target_Enter Simple
+!===============================================================================
+
+!CHECK: omp.target_cpu = "gfx908",
+!CHECK-SAME: omp.target_cpu_features = "+dot3-insts,+dot4-insts,+s-memtime-inst,
+!CHECK-SAME: +16-bit-insts,+s-memrealtime,+dot6-insts,+dl-insts,+wavefrontsize64,
+!CHECK-SAME: +gfx9-insts,+gfx8-insts,+ci-insts,+dot10-insts,+dot7-insts,
+!CHECK-SAME: +dot1-insts,+dot5-insts,+mai-insts,+dpp,+dot2-insts"
+!CHECK-LABEL: func.func @_QPomp_target_simple() {
+subroutine omp_target_simple
+end subroutine omp_target_simple
+

llvm/include/llvm/TargetParser/TargetParser.h

 //===-- TargetParser - Parser for target features ---------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements a target parser to recognise hardware features such as
 // FPU/CPU/ARCH names as well as specific support such as HDIV, etc.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_TARGETPARSER_TARGETPARSER_H
 #define LLVM_TARGETPARSER_TARGETPARSER_H
 
+#include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringRef.h"
 
 namespace llvm {
 
 template <typename T> class SmallVectorImpl;
 class Triple;
 
 // Target specific information in their own namespaces.

 unsigned getArchAttrAMDGCN(GPUKind AK);
 unsigned getArchAttrR600(GPUKind AK);
 
 void fillValidArchListAMDGCN(SmallVectorImpl<StringRef> &Values);
 void fillValidArchListR600(SmallVectorImpl<StringRef> &Values);
 
 IsaVersion getIsaVersion(StringRef GPU);
 
+/// Fills Features map with default values for given target GPU
+void fillAMDGPUFeatureMap(StringRef GPU, const Triple &T,
+                          StringMap<bool> &Features);
+
+/// Inserts wave size feature for given GPU into features map
+bool insertWaveSizeFeature(StringRef GPU, const Triple &T,
+                           StringMap<bool> &Features, std::string &ErrorMsg);
+
 } // namespace AMDGPU
 } // namespace llvm
 
 #endif

llvm/lib/TargetParser/TargetParser.cpp

 StringRef AMDGPU::getCanonicalArchName(const Triple &T, StringRef Arch) {
   assert(T.isAMDGPU());
   auto ProcKind = T.isAMDGCN() ? parseArchAMDGCN(Arch) : parseArchR600(Arch);
   if (ProcKind == GK_NONE)
     return StringRef();
 
   return T.isAMDGCN() ? getArchNameAMDGCN(ProcKind) : getArchNameR600(ProcKind);
 }
+
+void AMDGPU::fillAMDGPUFeatureMap(StringRef GPU, const Triple &T,
+                                  StringMap<bool> &Features) {
+  // XXX - What does the member GPU mean if device name string passed here?
+  if (T.isAMDGCN()) {
+    switch (parseArchAMDGCN(GPU)) {
+    case GK_GFX1103:
+    case GK_GFX1102:
+    case GK_GFX1101:
+    case GK_GFX1100:
+      Features["ci-insts"] = true;
+      Features["dot5-insts"] = true;
+      Features["dot7-insts"] = true;
+      Features["dot8-insts"] = true;
+      Features["dot9-insts"] = true;
+      Features["dot10-insts"] = true;
+      Features["dl-insts"] = true;
+      Features["16-bit-insts"] = true;
+      Features["dpp"] = true;
+      Features["gfx8-insts"] = true;
+      Features["gfx9-insts"] = true;
+      Features["gfx10-insts"] = true;
+      Features["gfx10-3-insts"] = true;
+      Features["gfx11-insts"] = true;
+      break;
+    case GK_GFX1036:
+    case GK_GFX1035:
+    case GK_GFX1034:
+    case GK_GFX1033:
+    case GK_GFX1032:
+    case GK_GFX1031:
+    case GK_GFX1030:
+      Features["ci-insts"] = true;
+      Features["dot1-insts"] = true;
+      Features["dot2-insts"] = true;
+      Features["dot5-insts"] = true;
+      Features["dot6-insts"] = true;
+      Features["dot7-insts"] = true;
+      Features["dot10-insts"] = true;
+      Features["dl-insts"] = true;
+      Features["16-bit-insts"] = true;
+      Features["dpp"] = true;
+      Features["gfx8-insts"] = true;
+      Features["gfx9-insts"] = true;
+      Features["gfx10-insts"] = true;
+      Features["gfx10-3-insts"] = true;
+      Features["s-memrealtime"] = true;
+      Features["s-memtime-inst"] = true;
+      break;
+    case GK_GFX1012:
+    case GK_GFX1011:
+      Features["dot1-insts"] = true;
+      Features["dot2-insts"] = true;
+      Features["dot5-insts"] = true;
+      Features["dot6-insts"] = true;
+      Features["dot7-insts"] = true;
+      Features["dot10-insts"] = true;
+      [[fallthrough]];
+    case GK_GFX1013:
+    case GK_GFX1010:
+      Features["dl-insts"] = true;
+      Features["ci-insts"] = true;
+      Features["16-bit-insts"] = true;
+      Features["dpp"] = true;
+      Features["gfx8-insts"] = true;
+      Features["gfx9-insts"] = true;
+      Features["gfx10-insts"] = true;
+      Features["s-memrealtime"] = true;
+      Features["s-memtime-inst"] = true;
+      break;
+    case GK_GFX940:
+      Features["gfx940-insts"] = true;
+      Features["fp8-insts"] = true;
+      [[fallthrough]];
+    case GK_GFX90A:
+      Features["gfx90a-insts"] = true;
+      [[fallthrough]];
+    case GK_GFX908:
+      Features["dot3-insts"] = true;
+      Features["dot4-insts"] = true;
+      Features["dot5-insts"] = true;
+      Features["dot6-insts"] = true;
+      Features["mai-insts"] = true;
+      [[fallthrough]];
+    case GK_GFX906:
+      Features["dl-insts"] = true;
+      Features["dot1-insts"] = true;
+      Features["dot2-insts"] = true;
+      Features["dot7-insts"] = true;
+      Features["dot10-insts"] = true;
+      [[fallthrough]];
+    case GK_GFX90C:
+    case GK_GFX909:
+    case GK_GFX904:
+    case GK_GFX902:
+    case GK_GFX900:
+      Features["gfx9-insts"] = true;
+      [[fallthrough]];
+    case GK_GFX810:
+    case GK_GFX805:
+    case GK_GFX803:
+    case GK_GFX802:
+    case GK_GFX801:
+      Features["gfx8-insts"] = true;
+      Features["16-bit-insts"] = true;
+      Features["dpp"] = true;
+      Features["s-memrealtime"] = true;
+      [[fallthrough]];
+    case GK_GFX705:
+    case GK_GFX704:
+    case GK_GFX703:
+    case GK_GFX702:
+    case GK_GFX701:
+    case GK_GFX700:
+      Features["ci-insts"] = true;
+      [[fallthrough]];
+    case GK_GFX602:
+    case GK_GFX601:
+    case GK_GFX600:
+      Features["s-memtime-inst"] = true;
+      break;
+    case GK_NONE:
+      break;
+    default:
+      llvm_unreachable("Unhandled GPU!");
+    }
+  } else {
+    if (GPU.empty())
+      GPU = "r600";
+
+    switch (llvm::AMDGPU::parseArchR600(GPU)) {
+    case GK_CAYMAN:
+    case GK_CYPRESS:
+    case GK_RV770:
+    case GK_RV670:
+      // TODO: Add fp64 when implemented.
+      break;
+    case GK_TURKS:
+    case GK_CAICOS:
+    case GK_BARTS:
+    case GK_SUMO:
+    case GK_REDWOOD:
+    case GK_JUNIPER:
+    case GK_CEDAR:
+    case GK_RV730:
+    case GK_RV710:
+    case GK_RS880:
+    case GK_R630:
+    case GK_R600:
+      break;
+    default:
+      llvm_unreachable("Unhandled GPU!");
+    }
+  }
+}
+
+static bool isWave32Capable(StringRef GPU, const Triple &T) {
+  bool IsWave32Capable = false;
+  // XXX - What does the member GPU mean if device name string passed here?
+  if (T.isAMDGCN()) {
+    switch (parseArchAMDGCN(GPU)) {
+    case GK_GFX1103:
+    case GK_GFX1102:
+    case GK_GFX1101:
+    case GK_GFX1100:
+    case GK_GFX1036:
+    case GK_GFX1035:
+    case GK_GFX1034:
+    case GK_GFX1033:
+    case GK_GFX1032:
+    case GK_GFX1031:
+    case GK_GFX1030:
+    case GK_GFX1012:
+    case GK_GFX1011:
+    case GK_GFX1013:
+    case GK_GFX1010:
+      IsWave32Capable = true;
+      break;
+    default:
+      break;
+    }
+  }
+  return IsWave32Capable;
+}
+
+bool AMDGPU::insertWaveSizeFeature(StringRef GPU, const Triple &T,
+                                   StringMap<bool> &Features,
+                                   std::string &ErrorMsg) {
+  bool IsWave32Capable = isWave32Capable(GPU, T);
+  const bool IsNullGPU = GPU.empty();
+  // FIXME: Not diagnosing wavefrontsize32 on wave64 only targets.
+  const bool HaveWave32 =
+      (IsWave32Capable || IsNullGPU) && Features.count("wavefrontsize32");
+  const bool HaveWave64 = Features.count("wavefrontsize64");
+  if (HaveWave32 && HaveWave64) {
+    ErrorMsg = "'wavefrontsize32' and 'wavefrontsize64' are mutually exclusive";
+    return false;
+  }
+  // Don't assume any wavesize with an unknown subtarget.
+  if (!IsNullGPU) {
+    // Default to wave32 if available, or wave64 if not
+    if (!HaveWave32 && !HaveWave64) {
+      StringRef DefaultWaveSizeFeature =
+          IsWave32Capable ? "wavefrontsize32" : "wavefrontsize64";
+      Features.insert(std::make_pair(DefaultWaveSizeFeature, true));
+    }
+  }
+  return true;
+}

mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td

def OpenMP_Dialect : Dialect {
 
   let extraClassDeclaration = [{
     // Set the omp.is_device attribute on the module with the specified boolean
     static void setIsDevice(Operation* module, bool isDevice);
 
     // Return the value of the omp.is_device attribute stored in the module if it
     // exists, otherwise return false by default
     static bool getIsDevice(Operation* module);
+
+    // Set the omp.target_cpu attribute on the module with the specified string
+    static void setTargetCpu(Operation* module, StringRef cpu);
+
+    // Return the value of the omp.target_cpu attribute stored in the module if it
+    // exists, otherwise return empty by default
+    static std::string getTargetCpu(Operation* module);
+
+    // Set the omp.target_cpu_features attribute on the module with
+    // the specified string
+    static void setTargetCpuFeatures(Operation* module, StringRef cpuFeatures);
+
+    // Return the value of the omp.target_cpu_features attribute stored in
+    // the module if it exists, otherwise return empty by default
+    static std::string getTargetCpuFeatures(Operation* module);
   }];
 }
 
 // OmpCommon requires definition of OpenACC_Dialect.
 include "mlir/Dialect/OpenMP/OmpCommon.td"
 
 class OpenMP_Op<string mnemonic, list<Trait> traits = []> :
       Op<OpenMP_Dialect, mnemonic, traits>;

mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp

 // exists, otherwise return false by default
 bool OpenMPDialect::getIsDevice(Operation* module) {
   if (Attribute isDevice = module->getAttr("omp.is_device"))
     if (isDevice.isa<mlir::BoolAttr>())
       return isDevice.dyn_cast<BoolAttr>().getValue();
   return false;
 }
 
+// Set the omp.target_cpu attribute on the module with the specified string
+void OpenMPDialect::setTargetCpu(Operation *module, llvm::StringRef cpu) {
+  module->setAttr(mlir::StringAttr::get(module->getContext(),
+                                        llvm::Twine{"omp.target_cpu"}),
+                  mlir::StringAttr::get(module->getContext(), cpu));
+}
+
+// Return the value of the omp.target_cpu attribute stored in the module if it
+// exists, otherwise return empty by default
+std::string OpenMPDialect::getTargetCpu(Operation *module) {
+  if (Attribute targetCpu = module->getAttr("omp.target_cpu"))
+    if (targetCpu.isa<mlir::StringAttr>())
+      return targetCpu.dyn_cast<StringAttr>().getValue().str();
+  return llvm::Twine{""}.str();
+}
+
+// Set the omp.target_cpu_features attribute on the module with
+// the specified string
+void OpenMPDialect::setTargetCpuFeatures(Operation *module,
+                                         llvm::StringRef cpuFeatures) {
+  module->setAttr(mlir::StringAttr::get(module->getContext(),
+                                        llvm::Twine{"omp.target_cpu_features"}),
+                  mlir::StringAttr::get(module->getContext(), cpuFeatures));
+}
+
+// Return the value of the omp.target_cpu_features attribute stored in the
+// module if it exists, otherwise return empty by default
+std::string OpenMPDialect::getTargetCpuFeatures(Operation *module) {
+  if (Attribute targetCpu = module->getAttr("omp.target_cpu_features"))
+    if (targetCpu.isa<mlir::StringAttr>())
+      return targetCpu.dyn_cast<StringAttr>().getValue().str();
+  return llvm::Twine{""}.str();
+}
+
 #define GET_ATTRDEF_CLASSES
 #include "mlir/Dialect/OpenMP/OpenMPOpsAttributes.cpp.inc"
 
 #define GET_OP_CLASSES
 #include "mlir/Dialect/OpenMP/OpenMPOps.cpp.inc"