[flang] Add a subset of PowerPC MMA (Matrix Multiply Accelerate) intrinsics

[DanielCChen]

Summary

This patch includes the a subset of MMA intrinsics that are included in the mma intrinsic module:

mma_assemble_acc
mma_assemble_pair
mma_build_acc
mma_disassemble_acc
mma_disassemble_pair

[tschuett]

I bet that the mma ops are only available on newer ppc cores and not on the old Apple ones. I could not find a feature check.

[DanielCChen]

In D155725#4515348, @tschuett wrote:

I bet that the mma ops are only available on newer ppc cores and not on the old Apple ones. I could not find a feature check.

Correct. The MMA intrinsics are for Power 10 only. It is documented in IBM's compiler Language Reference

[tschuett]

I wanted to say that I could try to use them on an Apple and the compiler would crash somewhere, see https://reviews.llvm.org/D152131.

[DanielCChen]

In D155725#4515726, @tschuett wrote:

I wanted to say that I could try to use them on an Apple and the compiler would crash somewhere, see https://reviews.llvm.org/D152131.

Right. There is a pending work item that will add semantic checking of Power10 cpu for these MMA intrinsics. Because there are some vector intrinsics that are for Power 10 only as well. We deferred the work until we have all MMA and vector intrinsics in.

[DanielCChen]

Add the MMA intrinsic module and the testing of LLVM IR.

[DanielCChen]

Attempt to fix the x64 Window build failure that failed due to linker error as:

FIRBuilder.lib(PPCIntrinsicCall.cpp.obj) : error LNK2019: unresolved external symbol "void __cdecl Fortran::common::die(char const *,...)" (?die@common@Fortran@@YAXPEBDZZ) referenced in function "class mlir::FunctionType __cdecl fir::getMmaIrFuncType(class mlir::MLIRContext *,enum fir::MMAOp)" (?getMmaIrFuncType@fir@@YA?AVFunctionType@mlir@@PEAVMLIRContext@3@W4MMAOp@1@@Z)
bin\tco.exe : fatal error LNK1120: 1 unresolved externals

[DanielCChen]

Fixing a build failure using clang as the build compiler.

lib/Optimizer/Builder/PPCIntrinsicCall.cpp:1100:3: error: default label in switch which covers all enumeration values [-Werror,-Wcovered-switch-default]

[kkwli0]

flang/lib/Optimizer/Builder/PPCIntrinsicCall.cpp

349

I don't think this is a relevant change for this patch.

371

Nit:

• "quadCnt: number of arguments that has __vector_quad type"

407

See comment above

704

Unrelated change

812

Unrelated change

846

Unrelated change

858

Unrelated change

1214

Unrelated change

1222

Unrelated change

1225

Unrelated change

1293

use braced initializers

1301

use braced initializers

1363

use braced initializers

flang/lib/Optimizer/Dialect/FIROps.cpp

958–964

Nit: "e.g. fir.vector<4:ui32> => mlir.vector<4xi32>"

flang/module/mma.f90

83

"according to Fortran"

91

"according to Fortran"

[DanielCChen]

flang/lib/Optimizer/Builder/PPCIntrinsicCall.cpp

349

Sure. I will remove the unrelated changes for a separate patch.

[DanielCChen]

Address review comments to remove unrelated changes.

[DanielCChen]

It seems the module and the test are missing from the last update. Put them back in after rebase.

[tschuett]

Could you please get the pwr10 check in before continuing to add more unchecked intrinsics.

[klausler]

In D155725#4532997, @tschuett wrote:

Could you please get the pwr10 check in before continuing to add more unchecked intrinsics.

Why is this important to you? This is a compiler under active development that is not yet suitable for production use.

[DanielCChen]

In D155725#4532997, @tschuett wrote:

Could you please get the pwr10 check in before continuing to add more unchecked intrinsics.

The pwr10 check that covers all PowerPC intrinsics in this patch is in https://reviews.llvm.org/D156243.
I will try to land that one before this one.

[tschuett]

In D155725#4532999, @klausler wrote:

In D155725#4532997, @tschuett wrote:

Could you please get the pwr10 check in before continuing to add more unchecked intrinsics.

Why is this important to you? This is a compiler under active development that is not yet suitable for production use.

The buildbots run the test suite from trunk and no buildbot caught the fact that there are issues trying to run the power10 intrinsics on a power9. There are missing tests and/or checks.

[DanielCChen]

In D155725#4533144, @tschuett wrote:

In D155725#4532999, @klausler wrote:

In D155725#4532997, @tschuett wrote:

Could you please get the pwr10 check in before continuing to add more unchecked intrinsics.

Why is this important to you? This is a compiler under active development that is not yet suitable for production use.

The buildbots run the test suite from trunk and no buildbot caught the fact that there are issues trying to run the power10 intrinsics on a power9. There are missing tests and/or checks.

These Power10 intrinsic test in this patch and the other one (to be landed) has "-mcpu=pwr10" specified. It also only dumps the LLRM IR. In theory, they should not run on Power9. That being said, as I mentioned in the previous note, I put the semantic checking code in a a different patch, which should issue a compile time error message for unsupported cpu option.

[kkwli0]

LG

[DanielCChen]

In D155725#4535401, @kkwli0 wrote:

LG

Thanks. I will wait one more day in case there are more comments.

[vdonaldson]

Builds are failing to compile this code in PPCIntrinsicCall.cpp:

const char *getMmaIrIntrName(MMAOp mmaOp) {
  switch (mmaOp) {
  case MMAOp::AssembleAcc:
    return "llvm.ppc.mma.assemble.acc";
  case MMAOp::AssemblePair:
    return "llvm.ppc.vsx.assemble.pair";
  case MMAOp::DisassembleAcc:
    return "llvm.ppc.mma.disassemble.acc";
  case MMAOp::DisassemblePair:
    return "llvm.ppc.vsx.disassemble.pair";
  }
}

mlir::FunctionType getMmaIrFuncType(mlir::MLIRContext *context, MMAOp mmaOp) {
  switch (mmaOp) {
  case MMAOp::AssembleAcc:
    return genMmaVqFuncType(context, /*Quad*/ 0, /*Pair*/ 0, /*Vector*/ 4);
  case MMAOp::AssemblePair:
    return genMmaVpFuncType(context, /*Quad*/ 0, /*Pair*/ 0, /*Vector*/ 2);
  case MMAOp::DisassembleAcc:
    return genMmaDisassembleFuncType(context, mmaOp);
  case MMAOp::DisassemblePair:
    return genMmaDisassembleFuncType(context, mmaOp);
  }
}

with the message

flang/lib/Optimizer/Builder/PPCIntrinsicCall.cpp:1288:1: error: control reaches end of non-void function [-Werror=return-type]

These two switches probably need an error default.

[DanielCChen]

In D155725#4547578, @vdonaldson wrote:

Builds are failing to compile this code in PPCIntrinsicCall.cpp:

const char *getMmaIrIntrName(MMAOp mmaOp) {
  switch (mmaOp) {
  case MMAOp::AssembleAcc:
    return "llvm.ppc.mma.assemble.acc";
  case MMAOp::AssemblePair:
    return "llvm.ppc.vsx.assemble.pair";
  case MMAOp::DisassembleAcc:
    return "llvm.ppc.mma.disassemble.acc";
  case MMAOp::DisassemblePair:
    return "llvm.ppc.vsx.disassemble.pair";
  }
}

mlir::FunctionType getMmaIrFuncType(mlir::MLIRContext *context, MMAOp mmaOp) {
  switch (mmaOp) {
  case MMAOp::AssembleAcc:
    return genMmaVqFuncType(context, /*Quad*/ 0, /*Pair*/ 0, /*Vector*/ 4);
  case MMAOp::AssemblePair:
    return genMmaVpFuncType(context, /*Quad*/ 0, /*Pair*/ 0, /*Vector*/ 2);
  case MMAOp::DisassembleAcc:
    return genMmaDisassembleFuncType(context, mmaOp);
  case MMAOp::DisassemblePair:
    return genMmaDisassembleFuncType(context, mmaOp);
  }
}

with the message

flang/lib/Optimizer/Builder/PPCIntrinsicCall.cpp:1288:1: error: control reaches end of non-void function [-Werror=return-type]

These two switches probably need an error default.

@vdonaldson Thanks for bringing it up! I am looking into it now.

[DanielCChen]

Thanks to @klausler for fixing it!

flang/include/flang/Optimizer/Builder/PPCIntrinsicCall.h

enum class VecOp {
   Slo,
   Sr,
   Srl,
   Sro,
   Sub,
   Xor
 };
 
+/// Enums used to templatize and share lowering of PowerPC MMA intrinsics.
+enum class MMAOp {
+  AssembleAcc,
+  AssemblePair,
+  DisassembleAcc,
+  DisassemblePair,
+};
+
+enum class MMAHandlerOp {
+  NoOp,
+  SubToFunc,
+  SubToFuncReverseArgOnLE,
+  FirstArgIsResult,
+};
+
 // Wrapper struct to encapsulate information for a vector type. Preserves
 // sign of eleTy if eleTy is signed/unsigned integer. Helps with vector type
 // conversions.
 struct VecTypeInfo {
   mlir::Type eleTy;
   uint64_t len;
 
   mlir::Type toFirVectorType() { return fir::VectorType::get(len, eleTy); }

 struct PPCIntrinsicLibrary : IntrinsicLibrary {
 
   // Constructors.
   explicit PPCIntrinsicLibrary(fir::FirOpBuilder &builder, mlir::Location loc)
       : IntrinsicLibrary(builder, loc) {}
   PPCIntrinsicLibrary() = delete;
   PPCIntrinsicLibrary(const PPCIntrinsicLibrary &) = delete;
 
+  // PPC MMA intrinsic generic handler
+  template <MMAOp IntrId, MMAHandlerOp HandlerOp>
+  void genMmaIntr(llvm::ArrayRef<fir::ExtendedValue>);
+
   // PPC intrinsic handlers.
   template <bool isImm>
   void genMtfsf(llvm::ArrayRef<fir::ExtendedValue>);
 
   fir::ExtendedValue genVecAbs(mlir::Type resultType,
                                llvm::ArrayRef<fir::ExtendedValue> args);
-
   template <VecOp>
   fir::ExtendedValue
   genVecAddAndMulSubXor(mlir::Type resultType,
                         llvm::ArrayRef<fir::ExtendedValue> args);
 
   template <VecOp>
   fir::ExtendedValue genVecCmp(mlir::Type resultType,
                                llvm::ArrayRef<fir::ExtendedValue> args);

flang/lib/Optimizer/Builder/IntrinsicCall.cpp

 //===----------------------------------------------------------------------===//
 
 const IntrinsicArgumentLoweringRules *
 getIntrinsicArgumentLowering(llvm::StringRef specificName) {
   llvm::StringRef name = genericName(specificName);
   if (const IntrinsicHandler *handler = findIntrinsicHandler(name))
     if (!handler->argLoweringRules.hasDefaultRules())
       return &handler->argLoweringRules;
+  if (const IntrinsicHandler *ppcHandler = findPPCIntrinsicHandler(name))
+    if (!ppcHandler->argLoweringRules.hasDefaultRules())
+      return &ppcHandler->argLoweringRules;
   return nullptr;
 }
 
 /// Return how argument \p argName should be lowered given the rules for the
 /// intrinsic function.
 fir::ArgLoweringRule
 lowerIntrinsicArgumentAs(const IntrinsicArgumentLoweringRules &rules,
                          unsigned position) {

flang/lib/Optimizer/Builder/PPCIntrinsicCall.cpp

 //===-- PPCIntrinsicCall.cpp ----------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Helper routines for constructing the FIR dialect of MLIR for PowerPC
 // intrinsics. Extensive use of MLIR interfaces and MLIR's coding style
 // (https://mlir.llvm.org/getting_started/DeveloperGuide/) is used in this
 // module.
 //
 //===----------------------------------------------------------------------===//
 
 #include "flang/Optimizer/Builder/PPCIntrinsicCall.h"
+#include "flang/Evaluate/common.h"
 #include "flang/Optimizer/Builder/FIRBuilder.h"
-#include "flang/Optimizer/Builder/IntrinsicCall.h"
 #include "flang/Optimizer/Builder/MutableBox.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 
 namespace fir {
 
 using PI = PPCIntrinsicLibrary;
 
 // PPC specific intrinsic handlers.
 static constexpr IntrinsicHandler ppcHandlers[]{
+    {"__ppc_mma_assemble_acc",
+     static_cast<IntrinsicLibrary::SubroutineGenerator>(
+         &PI::genMmaIntr<MMAOp::AssembleAcc, MMAHandlerOp::SubToFunc>),
+     {{{"acc", asAddr},
+       {"arg1", asValue},
+       {"arg2", asValue},
+       {"arg3", asValue},
+       {"arg4", asValue}}},
+     /*isElemental=*/true},
+    {"__ppc_mma_assemble_pair",
+     static_cast<IntrinsicLibrary::SubroutineGenerator>(
+         &PI::genMmaIntr<MMAOp::AssemblePair, MMAHandlerOp::SubToFunc>),
+     {{{"pair", asAddr}, {"arg1", asValue}, {"arg2", asValue}}},
+     /*isElemental=*/true},
+    {"__ppc_mma_build_acc",
+     static_cast<IntrinsicLibrary::SubroutineGenerator>(
+         &PI::genMmaIntr<MMAOp::AssembleAcc,
+                         MMAHandlerOp::SubToFuncReverseArgOnLE>),
+     {{{"acc", asAddr},
+       {"arg1", asValue},
+       {"arg2", asValue},
+       {"arg3", asValue},
+       {"arg4", asValue}}},
+     /*isElemental=*/true},
+    {"__ppc_mma_disassemble_acc",
+     static_cast<IntrinsicLibrary::SubroutineGenerator>(
+         &PI::genMmaIntr<MMAOp::DisassembleAcc, MMAHandlerOp::SubToFunc>),
+     {{{"data", asAddr}, {"acc", asValue}}},
+     /*isElemental=*/true},
+    {"__ppc_mma_disassemble_pair",
+     static_cast<IntrinsicLibrary::SubroutineGenerator>(
+         &PI::genMmaIntr<MMAOp::DisassemblePair, MMAHandlerOp::SubToFunc>),
+     {{{"data", asAddr}, {"pair", asValue}}},
+     /*isElemental=*/true},
     {"__ppc_mtfsf",
      static_cast<IntrinsicLibrary::SubroutineGenerator>(&PI::genMtfsf<false>),
      {{{"mask", asValue}, {"r", asValue}}},
      /*isElemental=*/false},
     {"__ppc_mtfsfi",
      static_cast<IntrinsicLibrary::SubroutineGenerator>(&PI::genMtfsf<true>),
      {{{"bf", asValue}, {"i", asValue}}},
      /*isElemental=*/false},
genFuncType<Ty::RealVector<8>, Ty::RealVector<8>, Ty::RealVector<8>,
                  Ty::RealVector<8>>,
      genLibCall},
 };
 
 const IntrinsicHandler *findPPCIntrinsicHandler(llvm::StringRef name) {
   auto compare = [](const IntrinsicHandler &ppcHandler, llvm::StringRef name) {
     return name.compare(ppcHandler.name) > 0;
   };
   auto result = llvm::lower_bound(ppcHandlers, name, compare);
   return result != std::end(ppcHandlers) && result->name == name ? result
                                                                  : nullptr;
 }
 
 using RtMap = Fortran::common::StaticMultimapView<MathOperation>;
 static constexpr RtMap ppcMathOps(ppcMathOperations);
 static_assert(ppcMathOps.Verify() && "map must be sorted");
 
 std::pair<const MathOperation *, const MathOperation *>
 checkPPCMathOperationsRange(llvm::StringRef name) {
   return ppcMathOps.equal_range(name);
 }
 
+static mlir::FunctionType genMmaVpFuncType(mlir::MLIRContext *context,
+                                           int quadCnt, int pairCnt, int vecCnt,
+                                           int intCnt = 0,
+                                           int vecElemBitSize = 8,
+                                           int intBitSize = 32) {
+  // Constructs a function type with the following signature:
+  // Result type: __vector_pair
+  // Arguments:
+  //   quadCnt: number of arguments that has __vector_quad type, followed by
+  //   pairCnt: number of arguments that has __vector_pair type, followed by
+  //   vecCnt: number of arguments that has vector(integer) type, followed by
+  //   intCnt: number of arguments that has integer type
+  //   vecElemBitSize: specifies the size of vector elements in bits
+  //   intBitSize: specifies the size of integer arguments in bits
+  auto vType{mlir::VectorType::get(
+      128 / vecElemBitSize, mlir::IntegerType::get(context, vecElemBitSize))};
+  auto vpType{fir::VectorType::get(256, mlir::IntegerType::get(context, 1))};
+  auto vqType{fir::VectorType::get(512, mlir::IntegerType::get(context, 1))};
+  auto iType{mlir::IntegerType::get(context, intBitSize)};
+  llvm::SmallVector<mlir::Type> argTypes;
+  for (int i = 0; i < quadCnt; ++i) {
+    argTypes.push_back(vqType);
+  }
+  for (int i = 0; i < pairCnt; ++i) {
+    argTypes.push_back(vpType);
+  }
+  for (int i = 0; i < vecCnt; ++i) {
+    argTypes.push_back(vType);
+  }
+  for (int i = 0; i < intCnt; ++i) {
+    argTypes.push_back(iType);
+  }
+
+  return mlir::FunctionType::get(context, argTypes, {vpType});
+}
+
+static mlir::FunctionType genMmaVqFuncType(mlir::MLIRContext *context,
+                                           int quadCnt, int pairCnt, int vecCnt,
+                                           int intCnt = 0,
+                                           int vecElemBitSize = 8,
+                                           int intBitSize = 32) {
+  // Constructs a function type with the following signature:
+  // Result type: __vector_quad
+  // Arguments:
+  //   quadCnt: number of arguments that has __vector_quad type, followed by
+  //   pairCnt: number of arguments that has __vector_pair type, followed by
+  //   vecCnt: number of arguments that has vector(integer) type, followed by
+  //   intCnt: number of arguments that has integer type
+  //   vecElemBitSize: specifies the size of vector elements in bits
+  //   intBitSize: specifies the size of integer arguments in bits
+  auto vType{mlir::VectorType::get(
+      128 / vecElemBitSize, mlir::IntegerType::get(context, vecElemBitSize))};
+  auto vpType{fir::VectorType::get(256, mlir::IntegerType::get(context, 1))};
+  auto vqType{fir::VectorType::get(512, mlir::IntegerType::get(context, 1))};
+  auto iType{mlir::IntegerType::get(context, intBitSize)};
+  llvm::SmallVector<mlir::Type> argTypes;
+  for (int i = 0; i < quadCnt; ++i) {
+    argTypes.push_back(vqType);
+  }
+  for (int i = 0; i < pairCnt; ++i) {
+    argTypes.push_back(vpType);
+  }
+  for (int i = 0; i < vecCnt; ++i) {
+    argTypes.push_back(vType);
+  }
+  for (int i = 0; i < intCnt; ++i) {
+    argTypes.push_back(iType);
+  }
+
+  return mlir::FunctionType::get(context, argTypes, {vqType});
+}
+
+mlir::FunctionType genMmaDisassembleFuncType(mlir::MLIRContext *context,
+                                             MMAOp mmaOp) {
+  auto vType{mlir::VectorType::get(16, mlir::IntegerType::get(context, 8))};
+  llvm::SmallVector<mlir::Type> members;
+
+  if (mmaOp == MMAOp::DisassembleAcc) {
+    auto vqType{fir::VectorType::get(512, mlir::IntegerType::get(context, 1))};
+    members.push_back(vType);
+    members.push_back(vType);
+    members.push_back(vType);
+    members.push_back(vType);
+    auto resType{mlir::LLVM::LLVMStructType::getLiteral(context, members)};
+    return mlir::FunctionType::get(context, {vqType}, {resType});
+  } else if (mmaOp == MMAOp::DisassemblePair) {
+    auto vpType{fir::VectorType::get(256, mlir::IntegerType::get(context, 1))};
+    members.push_back(vType);
+    members.push_back(vType);
+    auto resType{mlir::LLVM::LLVMStructType::getLiteral(context, members)};
+    return mlir::FunctionType::get(context, {vpType}, {resType});
+  } else {
+    llvm_unreachable(
+        "Unsupported intrinsic code for function signature generator");
+  }
+}
+
 //===----------------------------------------------------------------------===//
 // PowerPC specific intrinsic handlers.
 //===----------------------------------------------------------------------===//
 
 // MTFSF, MTFSFI
 template <bool isImm>
 void PPCIntrinsicLibrary::genMtfsf(llvm::ArrayRef<fir::ExtendedValue> args) {
   assert(args.size() == 2);
PPCIntrinsicLibrary::genVecAnyCompare(mlir::Type resultType,
 
   mlir::FunctionType ftype{nullptr};
   llvm::StringRef fname;
   const auto i32Ty{mlir::IntegerType::get(context, 32)};
   llvm::SmallVector<mlir::Value> cmpArgs;
   mlir::Value op{nullptr};
   const auto width{vTypeInfo.eleTy.getIntOrFloatBitWidth()};
 
   if (auto elementTy = mlir::dyn_cast<mlir::IntegerType>(vTypeInfo.eleTy)) {
     std::pair<llvm::StringRef, mlir::FunctionType> bi;
     bi = (elementTy.isUnsignedInteger())
              ? uiBuiltin[std::pair(ParamTypeId::UnsignedVector, width)]
              : uiBuiltin[std::pair(ParamTypeId::IntegerVector, width)];
 
     fname = std::get<0>(bi);
     ftype = std::get<1>(bi);
 
static std::map<std::pair<Cmp, int>,
                                        Ty::RealVector<4>>(context, builder))},
            {std::make_pair(Cmp::gtOrLt, 64),
             std::make_pair("llvm.ppc.vsx.xvcmpgtdp",
                            genFuncType<Ty::UnsignedVector<8>, Ty::RealVector<8>,
                                        Ty::RealVector<8>>(context, builder))}};
 
   const auto width{vTypeInfo.eleTy.getIntOrFloatBitWidth()};
   std::pair<llvm::StringRef, mlir::FunctionType> specFunc;
   if (auto elementTy = mlir::dyn_cast<mlir::IntegerType>(vTypeInfo.eleTy))
     specFunc =
         (elementTy.isUnsignedInteger())
             ? iuBuiltinName[std::make_pair(ParamTypeId::UnsignedVector, width)]
             : iuBuiltinName[std::make_pair(ParamTypeId::IntegerVector, width)];
   else if (vTypeInfo.isFloat())
     specFunc = (vop == VecOp::Cmpge || vop == VecOp::Cmple)
                    ? rGBI[std::make_pair(Cmp::geOrLe, width)]
                    : rGBI[std::make_pair(Cmp::gtOrLt, width)];
PPCIntrinsicLibrary::genVecCmp(mlir::Type resultType,
   std::pair<llvm::StringRef, mlir::FunctionType> funcTyNam{
       getVecCmpFuncTypeAndName(vecTyInfo, vop, builder)};
 
   mlir::func::FuncOp funcOp = builder.addNamedFunction(
       loc, std::get<0>(funcTyNam), std::get<1>(funcTyNam));
 
   mlir::Value res{nullptr};
 
   if (auto eTy = vecTyInfo.eleTy.dyn_cast<mlir::IntegerType>()) {
     constexpr int firstArg{0};
     constexpr int secondArg{1};
     std::map<VecOp, std::array<int, 2>> argOrder{
         {VecOp::Cmpge, {secondArg, firstArg}},
         {VecOp::Cmple, {firstArg, secondArg}},
         {VecOp::Cmpgt, {firstArg, secondArg}},
         {VecOp::Cmplt, {secondArg, firstArg}}};
 
     // Construct the function return type, unsigned vector, for conversion.
     auto itype = mlir::IntegerType::get(context, eTy.getWidth(),
                                         mlir::IntegerType::Unsigned);
     auto returnType = fir::VectorType::get(vecTyInfo.len, itype);
 
     switch (vop) {
     case VecOp::Cmpgt:
     case VecOp::Cmplt: {
       // arg1 > arg2 --> vcmpgt(arg1, arg2)
       // arg1 < arg2 --> vcmpgt(arg2, arg1)
       mlir::Value vargs[]{argBases[argOrder[vop][0]],
                           argBases[argOrder[vop][1]]};
if (vop == VecOp::Sl || vop == VecOp::Sr) {
     auto funcTy{genFuncType<Ty::IntegerVector<4>, Ty::IntegerVector<4>,
                             Ty::IntegerVector<4>>(context, builder)};
     mlir::func::FuncOp funcOp{builder.addNamedFunction(loc, funcName, funcTy)};
     auto callOp{builder.create<fir::CallOp>(loc, funcOp, mlirVecArgs)};
 
     // If the result vector type is different from the original type, need
     // to convert to mlir vector, bitcast and then convert back to fir vector.
     if (callOp.getResult(0).getType() != argTypes[0]) {
       auto res = builder.createConvert(loc, bcVecTy, callOp.getResult(0));
       res = builder.create<mlir::vector::BitCastOp>(loc, mlirTyArgs[0], res);
       shftRes = builder.createConvert(loc, argTypes[0], res);
     } else {
       shftRes = callOp.getResult(0);
     }
   } else if (vop == VecOp::Sld || vop == VecOp::Sldw) {
     assert(args.size() == 3);
     auto constIntOp =
         mlir::dyn_cast<mlir::arith::ConstantOp>(argBases[2].getDefiningOp())
             .getValue()
             .dyn_cast_or_null<mlir::IntegerAttr>();
     assert(constIntOp && "expected integer constant argument");
 
     // Bitcast to vector<16xi8>
     auto vi8Ty{mlir::VectorType::get(16, builder.getIntegerType(8))};
     if (mlirTyArgs[0] != vi8Ty) {
       mlirVecArgs[0] =
           builder.create<mlir::LLVM::BitcastOp>(loc, vi8Ty, mlirVecArgs[0])
               .getResult();
if (vop == VecOp::Sl || vop == VecOp::Sr) {
 
     return builder.createConvert(loc, resultType, shftRes);
   } else
     llvm_unreachable("Invalid vector operation for generator");
 
   return shftRes;
 }
 
+const char *getMmaIrIntrName(MMAOp mmaOp) {
+  switch (mmaOp) {
+  case MMAOp::AssembleAcc:
+    return "llvm.ppc.mma.assemble.acc";
+  case MMAOp::AssemblePair:
+    return "llvm.ppc.vsx.assemble.pair";
+  case MMAOp::DisassembleAcc:
+    return "llvm.ppc.mma.disassemble.acc";
+  case MMAOp::DisassemblePair:
+    return "llvm.ppc.vsx.disassemble.pair";
+  }
+}
+
+mlir::FunctionType getMmaIrFuncType(mlir::MLIRContext *context, MMAOp mmaOp) {
+  switch (mmaOp) {
+  case MMAOp::AssembleAcc:
+    return genMmaVqFuncType(context, /*Quad*/ 0, /*Pair*/ 0, /*Vector*/ 4);
+  case MMAOp::AssemblePair:
+    return genMmaVpFuncType(context, /*Quad*/ 0, /*Pair*/ 0, /*Vector*/ 2);
+  case MMAOp::DisassembleAcc:
+    return genMmaDisassembleFuncType(context, mmaOp);
+  case MMAOp::DisassemblePair:
+    return genMmaDisassembleFuncType(context, mmaOp);
+  }
+}
+
+template <MMAOp IntrId, MMAHandlerOp HandlerOp>
+void PPCIntrinsicLibrary::genMmaIntr(llvm::ArrayRef<fir::ExtendedValue> args) {
+  auto context{builder.getContext()};
+  mlir::FunctionType intrFuncType{getMmaIrFuncType(context, IntrId)};
+  mlir::func::FuncOp funcOp{
+      builder.addNamedFunction(loc, getMmaIrIntrName(IntrId), intrFuncType)};
+  llvm::SmallVector<mlir::Value> intrArgs;
+
+  // Depending on SubToFunc, change the subroutine call to a function call.
+  // First argument represents the result. Rest of the arguments
+  // are shifted one position to form the actual argument list.
+  size_t argStart{0};
+  size_t argStep{1};
+  size_t e{args.size()};
+  if (HandlerOp == MMAHandlerOp::SubToFunc) {
+    // The first argument becomes function result. Start from the second
+    // argument.
+    argStart = 1;
+  } else if (HandlerOp == MMAHandlerOp::SubToFuncReverseArgOnLE) {
+    // Reverse argument order on little-endian target only.
+    // The reversal does not depend on the setting of non-native-order option.
+    if (Fortran::evaluate::isHostLittleEndian) {
+      // Load the arguments in reverse order.
+      argStart = args.size() - 1;
+      // The first argument becomes function result. Stop at the second
+      // argument.
+      e = 0;
+      argStep = -1;
+    } else {
+      // Load the arguments in natural order.
+      // The first argument becomes function result. Start from the second
+      // argument.
+      argStart = 1;
+    }
+  }
+
+  for (size_t i = argStart, j = 0; i != e; i += argStep, ++j) {
+    auto v{fir::getBase(args[i])};
+    if (i == 0 && HandlerOp == MMAHandlerOp::FirstArgIsResult) {
+      // First argument is passed in as an address. We need to load
+      // the content to match the LLVM interface.
+      v = builder.create<fir::LoadOp>(loc, v);
+    }
+    auto vType{v.getType()};
+    mlir::Type targetType{intrFuncType.getInput(j)};
+    if (vType != targetType) {
+      if (targetType.isa<mlir::VectorType>()) {
+        // Perform vector type conversion for arguments passed by value.
+        auto eleTy{vType.dyn_cast<fir::VectorType>().getEleTy()};
+        auto len{vType.dyn_cast<fir::VectorType>().getLen()};
+        mlir::VectorType mlirType = mlir::VectorType::get(len, eleTy);
+        auto v0{builder.createConvert(loc, mlirType, v)};
+        auto v1{builder.create<mlir::vector::BitCastOp>(loc, targetType, v0)};
+        intrArgs.push_back(v1);
+      } else if (targetType.isa<mlir::IntegerType>() &&
+                 vType.isa<mlir::IntegerType>()) {
+        auto v0{builder.createConvert(loc, targetType, v)};
+        intrArgs.push_back(v0);
+      } else {
+        llvm::errs() << "\nUnexpected type conversion requested: "
+                     << " from " << vType << " to " << targetType << "\n";
+        llvm_unreachable("Unsupported type conversion for argument to PowerPC "
+                         "MMA intrinsic");
+      }
+    } else {
+      intrArgs.push_back(v);
+    }
+  }
+  auto callSt{builder.create<fir::CallOp>(loc, funcOp, intrArgs)};
+  if (HandlerOp == MMAHandlerOp::SubToFunc ||
+      HandlerOp == MMAHandlerOp::SubToFuncReverseArgOnLE ||
+      HandlerOp == MMAHandlerOp::FirstArgIsResult) {
+    // Convert pointer type if needed.
+    mlir::Value callResult{callSt.getResult(0)};
+    mlir::Value destPtr{fir::getBase(args[0])};
+    mlir::Type callResultPtrType{builder.getRefType(callResult.getType())};
+    if (destPtr.getType() != callResultPtrType) {
+      destPtr = builder.create<fir::ConvertOp>(loc, callResultPtrType, destPtr);
+    }
+    // Copy the result.
+    builder.create<fir::StoreOp>(loc, callResult, destPtr);
+  }
+}
+
 } // namespace fir

flang/lib/Optimizer/Dialect/FIROps.cpp

 
 bool fir::ConvertOp::isPointerCompatible(mlir::Type ty) {
   return ty.isa<fir::ReferenceType, fir::PointerType, fir::HeapType,
                 fir::LLVMPointerType, mlir::MemRefType, mlir::FunctionType,
                 fir::TypeDescType>();
 }
 
 static std::optional<mlir::Type> getVectorElementType(mlir::Type ty) {
-  if (mlir::isa<fir::VectorType>(ty)) {
-    auto elemTy = mlir::dyn_cast<fir::VectorType>(ty).getEleTy();
+  mlir::Type elemTy;
+  if (mlir::isa<fir::VectorType>(ty))
+    elemTy = mlir::dyn_cast<fir::VectorType>(ty).getEleTy();
+  else if (mlir::isa<mlir::VectorType>(ty))
+    elemTy = mlir::dyn_cast<mlir::VectorType>(ty).getElementType();
+  else
+    return std::nullopt;
 
-    // fir.vector<4:ui32> is converted to mlir.vector<4xi32>
-    if (elemTy.isUnsignedInteger()) {
-      elemTy = mlir::IntegerType::get(
-          ty.getContext(),
-          mlir::dyn_cast<mlir::IntegerType>(elemTy).getWidth());
-    }
-    return elemTy;
-  } else if (mlir::isa<mlir::VectorType>(ty))
-    return mlir::dyn_cast<mlir::VectorType>(ty).getElementType();
-
-  return std::nullopt;
+  // e.g. fir.vector<4:ui32> => mlir.vector<4xi32>
+  // e.g. mlir.vector<4xui32> => mlir.vector<4xi32>
+        ty.getContext(), mlir::dyn_cast<mlir::IntegerType>(elemTy).getWidth());
 }
 
 static std::optional<uint64_t> getVectorLen(mlir::Type ty) {
   if (mlir::isa<fir::VectorType>(ty))
     return mlir::dyn_cast<fir::VectorType>(ty).getLen();
   else if (mlir::isa<mlir::VectorType>(ty)) {
     // fir.vector only supports 1-D vector
     if (!(mlir::dyn_cast<mlir::VectorType>(ty).isScalable()))

flang/lib/Semantics/check-call.cpp

static void CheckExplicitDataArg(const characteristics::DummyDataObject &dummy,
   }
 
   // Derived type actual argument checks
   const Symbol *actualFirstSymbol{evaluate::GetFirstSymbol(actual)};
   bool actualIsAsynchronous{
       actualFirstSymbol && actualFirstSymbol->attrs().test(Attr::ASYNCHRONOUS)};
   bool actualIsVolatile{
       actualFirstSymbol && actualFirstSymbol->attrs().test(Attr::VOLATILE)};
-  if (const auto *derived{evaluate::GetDerivedTypeSpec(actualType.type())}) {
+  const auto *derived{evaluate::GetDerivedTypeSpec(actualType.type())};
+  if (derived && !derived->IsVectorType()) {
     if (dummy.type.type().IsAssumedType()) {
       if (!derived->parameters().empty()) { // 15.5.2.4(2)
         messages.Say(
             "Actual argument associated with TYPE(*) %s may not have a parameterized derived type"_err_en_US,
             dummyName);
       }
       if (const Symbol *
           tbp{FindImmediateComponent(*derived, [](const Symbol &symbol) {

flang/lib/Semantics/semantics.cpp

if (!program_.v.empty()) {
     if (frontModule &&
         (std::get<parser::Statement<parser::ModuleStmt>>(frontModule->value().t)
                     .statement.v.source == "__fortran_builtins" ||
             std::get<parser::Statement<parser::ModuleStmt>>(
                 frontModule->value().t)
                     .statement.v.source == "__ppc_types")) {
       // Don't try to read the builtins module when we're actually building it.
     } else if (frontModule &&
-        std::get<parser::Statement<parser::ModuleStmt>>(frontModule->value().t)
-                .statement.v.source == "__ppc_intrinsics") {
+        (std::get<parser::Statement<parser::ModuleStmt>>(frontModule->value().t)
+                    .statement.v.source == "__ppc_intrinsics" ||
+            std::get<parser::Statement<parser::ModuleStmt>>(
+                frontModule->value().t)
+                    .statement.v.source == "mma")) {
       // The derived type definition for the vectors is needed.
       context_.UsePPCBuiltinTypesModule();
     } else {
       context_.UseFortranBuiltinsModule();
       llvm::Triple targetTriple{llvm::Triple(
           llvm::Triple::normalize(llvm::sys::getDefaultTargetTriple()))};
       // Only use __ppc_intrinsics module when targetting PowerPC arch
       if (context_.targetCharacteristics().isPPC()) {

flang/module/mma.f90

+!===-- module/mma.f90 ------------------------------------------------------===!
+!
+! Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+! See https://llvm.org/LICENSE.txt for license information.
+! SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+!
+!===------------------------------------------------------------------------===!
+
+module mma
+  implicit none
+  private
+
+  abstract interface
+
+!! ========== 3 arguments subroutine interface ===============================!!
+!! subroutine s(__vector_pair, vector(i), vector(i))
+#define ELEM_SUB_VPVIVI(VKIND) \
+  elemental subroutine sub_vpvi##VKIND##vi##VKIND(pair, arg1, arg2); \
+    __vector_pair, intent(out) :: pair ; \
+    vector(integer(VKIND)), intent(in) :: arg1, arg2; \
+  end subroutine ;
+
+!! subroutine s(__vector_pair, vector(u), vector(u))
+#define ELEM_SUB_VPVUVU(VKIND) \
+  elemental subroutine sub_vpvu##VKIND##vu##VKIND(pair, arg1, arg2); \
+    __vector_pair, intent(out) :: pair ; \
+    vector(unsigned(VKIND)), intent(in) :: arg1, arg2; \
+  end subroutine ;
+
+!! subroutine s(__vector_pair, vector(r), vector(r))
+#define ELEM_SUB_VPVRVR(VKIND) \
+  elemental subroutine sub_vpvr##VKIND##vr##VKIND(pair, arg1, arg2); \
+    __vector_pair, intent(out) :: pair ; \
+    vector(real(VKIND)), intent(in) :: arg1, arg2; \
+  end subroutine ;
+
+  ELEM_SUB_VPVIVI(1) ELEM_SUB_VPVIVI(2)
+  ELEM_SUB_VPVIVI(4) ELEM_SUB_VPVIVI(8)
+  ELEM_SUB_VPVUVU(1) ELEM_SUB_VPVUVU(2)
+  ELEM_SUB_VPVUVU(4) ELEM_SUB_VPVUVU(8)
+  ELEM_SUB_VPVRVR(4) ELEM_SUB_VPVRVR(8)
+
+#undef ELEM_SUB_VPVIVI
+#undef ELEM_SUB_VPVUVU
+#undef ELEM_SUB_VPVRVR
+
+!! ========== 5 arguments subroutine interface ===============================!!
+!! subroutine s(__vector_quad, vector(i), vector(i), vector(i), vector(i))
+#define ELEM_SUB_VQVIVIVIVI(VKIND) \
+  elemental subroutine sub_vqvi##VKIND##vi##VKIND##vi##VKIND##vi##VKIND(acc, arg1, arg2, arg3, arg4); \
+    __vector_quad, intent(out) :: acc; \
+    vector(integer(VKIND)), intent(in) :: arg1, arg2, arg3, arg4; \
+  end subroutine ;
+
+!! subroutine s(__vector_quad, vector(u), vector(u), vector(u), vector(u))
+#define ELEM_SUB_VQVUVUVUVU(VKIND) \
+  elemental subroutine sub_vqvu##VKIND##vu##VKIND##vu##VKIND##vu##VKIND(acc, arg1, arg2, arg3, arg4); \
+    __vector_quad, intent(out) :: acc; \
+    vector(unsigned(VKIND)), intent(in) :: arg1, arg2, arg3, arg4; \
+  end subroutine ;
+
+!! subroutine s(__vector_quad, vector(r), vector(r), vector(r), vector(r))
+#define ELEM_SUB_VQVRVRVRVR(VKIND) \
+  elemental subroutine sub_vqvr##VKIND##vr##VKIND##vr##VKIND##vr##VKIND(acc, arg1, arg2, arg3, arg4); \
+    __vector_quad, intent(out) :: acc; \
+    vector(real(VKIND)), intent(in) :: arg1, arg2, arg3, arg4; \
+  end subroutine ;
+
+  ELEM_SUB_VQVIVIVIVI(1) ELEM_SUB_VQVIVIVIVI(2)
+  ELEM_SUB_VQVIVIVIVI(4) ELEM_SUB_VQVIVIVIVI(8)
+  ELEM_SUB_VQVUVUVUVU(1) ELEM_SUB_VQVUVUVUVU(2)
+  ELEM_SUB_VQVUVUVUVU(4) ELEM_SUB_VQVUVUVUVU(8)
+  ELEM_SUB_VQVRVRVRVR(4) ELEM_SUB_VQVRVRVRVR(8)
+
+#undef ELEM_SUB_VQVRVRVRVR
+#undef ELEM_SUB_VQVUVUVUVU
+#undef ELEM_SUB_VQVIVIVIVI
+
+!! ========== non-macro interface =============================================!!
+  elemental subroutine sub_atvp(data, pair)
+    ! Dummy arg 'data' is supposed to be intent(out) of any type,
+    ! but according to Fortran 2018: C709: Type(*) arguments can not have
+    ! intent(out) attribute. Use intent(inout) instead.
+    type(*), intent(inout) :: data
+    __vector_pair, intent(inout) :: pair
+  end subroutine
+
+  elemental subroutine sub_atvq(data, acc)
+    ! Dummy arg 'data' is supposed to be intent(out) of any type,
+    ! but according to Fortran 2018: C709: Type(*) arguments can not have
+    ! intent(out) attribute. Use intent(inout) instead.
+    type(*), intent(inout) :: data
+    __vector_quad, intent(inout) :: acc
+  end subroutine
+
+  end interface
+
+
+#define SUB_VQ_VI_VI_VI_VI(NAME, VKIND) __ppc_##NAME##_vi##VKIND##vi##VKIND##vi##VKIND##vi##VKIND
+#define SUB_VQ_VU_VU_VU_VU(NAME, VKIND) __ppc_##NAME##_vu##VKIND##vu##VKIND##vu##VKIND##vu##VKIND
+#define SUB_VQ_VR_VR_VR_VR(NAME, VKIND) __ppc_##NAME##_vr##VKIND##vr##VKIND##vr##VKIND##vr##VKIND
+
+#define VEC_SUB_VQ_VI_VI_VI_VI(NAME, VKIND) \
+  procedure(sub_vqvi##VKIND##vi##VKIND##vi##VKIND##vi##VKIND) :: SUB_VQ_VI_VI_VI_VI(NAME, VKIND);
+#define VEC_SUB_VQ_VU_VU_VU_VU(NAME, VKIND) \
+  procedure(sub_vqvu##VKIND##vu##VKIND##vu##VKIND##vu##VKIND) :: SUB_VQ_VU_VU_VU_VU(NAME, VKIND);
+#define VEC_SUB_VQ_VR_VR_VR_VR(NAME, VKIND) \
+  procedure(sub_vqvr##VKIND##vr##VKIND##vr##VKIND##vr##VKIND) :: SUB_VQ_VR_VR_VR_VR(NAME, VKIND);
+
+! mma_assemble_acc
+  VEC_SUB_VQ_VI_VI_VI_VI(mma_assemble_acc,1)
+  VEC_SUB_VQ_VI_VI_VI_VI(mma_assemble_acc,2)
+  VEC_SUB_VQ_VI_VI_VI_VI(mma_assemble_acc,4)
+  VEC_SUB_VQ_VI_VI_VI_VI(mma_assemble_acc,8)
+  VEC_SUB_VQ_VU_VU_VU_VU(mma_assemble_acc,1)
+  VEC_SUB_VQ_VU_VU_VU_VU(mma_assemble_acc,2)
+  VEC_SUB_VQ_VU_VU_VU_VU(mma_assemble_acc,4)
+  VEC_SUB_VQ_VU_VU_VU_VU(mma_assemble_acc,8)
+  VEC_SUB_VQ_VR_VR_VR_VR(mma_assemble_acc,4)
+  VEC_SUB_VQ_VR_VR_VR_VR(mma_assemble_acc,8)
+  interface mma_assemble_acc
+    procedure :: SUB_VQ_VI_VI_VI_VI(mma_assemble_acc,1)
+    procedure :: SUB_VQ_VI_VI_VI_VI(mma_assemble_acc,2)
+    procedure :: SUB_VQ_VI_VI_VI_VI(mma_assemble_acc,4)
+    procedure :: SUB_VQ_VI_VI_VI_VI(mma_assemble_acc,8)
+    procedure :: SUB_VQ_VU_VU_VU_VU(mma_assemble_acc,1)
+    procedure :: SUB_VQ_VU_VU_VU_VU(mma_assemble_acc,2)
+    procedure :: SUB_VQ_VU_VU_VU_VU(mma_assemble_acc,4)
+    procedure :: SUB_VQ_VU_VU_VU_VU(mma_assemble_acc,8)
+    procedure :: SUB_VQ_VR_VR_VR_VR(mma_assemble_acc,4)
+    procedure :: SUB_VQ_VR_VR_VR_VR(mma_assemble_acc,8)
+  end interface
+  public mma_assemble_acc
+
+! mma_build_acc
+  VEC_SUB_VQ_VI_VI_VI_VI(mma_build_acc,1)
+  VEC_SUB_VQ_VI_VI_VI_VI(mma_build_acc,2)
+  VEC_SUB_VQ_VI_VI_VI_VI(mma_build_acc,4)
+  VEC_SUB_VQ_VI_VI_VI_VI(mma_build_acc,8)
+  VEC_SUB_VQ_VU_VU_VU_VU(mma_build_acc,1)
+  VEC_SUB_VQ_VU_VU_VU_VU(mma_build_acc,2)
+  VEC_SUB_VQ_VU_VU_VU_VU(mma_build_acc,4)
+  VEC_SUB_VQ_VU_VU_VU_VU(mma_build_acc,8)
+  VEC_SUB_VQ_VR_VR_VR_VR(mma_build_acc,4)
+  VEC_SUB_VQ_VR_VR_VR_VR(mma_build_acc,8)
+  interface mma_build_acc
+    procedure :: SUB_VQ_VI_VI_VI_VI(mma_build_acc,1)
+    procedure :: SUB_VQ_VI_VI_VI_VI(mma_build_acc,2)
+    procedure :: SUB_VQ_VI_VI_VI_VI(mma_build_acc,4)
+    procedure :: SUB_VQ_VI_VI_VI_VI(mma_build_acc,8)
+    procedure :: SUB_VQ_VU_VU_VU_VU(mma_build_acc,1)
+    procedure :: SUB_VQ_VU_VU_VU_VU(mma_build_acc,2)
+    procedure :: SUB_VQ_VU_VU_VU_VU(mma_build_acc,4)
+    procedure :: SUB_VQ_VU_VU_VU_VU(mma_build_acc,8)
+    procedure :: SUB_VQ_VR_VR_VR_VR(mma_build_acc,4)
+    procedure :: SUB_VQ_VR_VR_VR_VR(mma_build_acc,8)
+  end interface
+  public mma_build_acc
+
+#undef VEC_SUB_VQ_VR_VR_VR_VR
+#undef VEC_SUB_VQ_VU_VU_VU_VU
+#undef VEC_SUB_VQ_VI_VI_VI_VI
+#undef SUB_VQ_VR_VR_VR_VR
+#undef SUB_VQ_VU_VU_VU_VU
+#undef SUB_VQ_VI_VI_VI_VI
+
+#define SUB_VP_VI_VI(NAME, VKIND) __ppc_##NAME##_vi##VKIND##vi##VKIND
+#define SUB_VP_VU_VU(NAME, VKIND) __ppc_##NAME##_vu##VKIND##vu##VKIND
+#define SUB_VP_VR_VR(NAME, VKIND) __ppc_##NAME##_vr##VKIND##vr##VKIND
+
+#define VEC_SUB_VP_VI_VI(NAME, VKIND) \
+  procedure(sub_vpvi##VKIND##vi##VKIND) :: SUB_VP_VI_VI(NAME, VKIND);
+#define VEC_SUB_VP_VU_VU(NAME, VKIND) \
+  procedure(sub_vpvu##VKIND##vu##VKIND) :: SUB_VP_VU_VU(NAME, VKIND);
+#define VEC_SUB_VP_VR_VR(NAME, VKIND) \
+  procedure(sub_vpvr##VKIND##vr##VKIND) :: SUB_VP_VR_VR(NAME, VKIND);
+
+! mma_assemble_pair
+  VEC_SUB_VP_VI_VI(mma_assemble_pair,1) VEC_SUB_VP_VI_VI(mma_assemble_pair,2)
+  VEC_SUB_VP_VI_VI(mma_assemble_pair,4) VEC_SUB_VP_VI_VI(mma_assemble_pair,8)
+  VEC_SUB_VP_VU_VU(mma_assemble_pair,1) VEC_SUB_VP_VU_VU(mma_assemble_pair,2)
+  VEC_SUB_VP_VU_VU(mma_assemble_pair,4) VEC_SUB_VP_VU_VU(mma_assemble_pair,8)
+  VEC_SUB_VP_VR_VR(mma_assemble_pair,4) VEC_SUB_VP_VR_VR(mma_assemble_pair,8)
+  interface mma_assemble_pair
+    procedure :: SUB_VP_VI_VI(mma_assemble_pair,1)
+    procedure :: SUB_VP_VI_VI(mma_assemble_pair,2)
+    procedure :: SUB_VP_VI_VI(mma_assemble_pair,4)
+    procedure :: SUB_VP_VI_VI(mma_assemble_pair,8)
+    procedure :: SUB_VP_VU_VU(mma_assemble_pair,1)
+    procedure :: SUB_VP_VU_VU(mma_assemble_pair,2)
+    procedure :: SUB_VP_VU_VU(mma_assemble_pair,4)
+    procedure :: SUB_VP_VU_VU(mma_assemble_pair,8)
+    procedure :: SUB_VP_VR_VR(mma_assemble_pair,4)
+    procedure :: SUB_VP_VR_VR(mma_assemble_pair,8)
+  end interface
+  public mma_assemble_pair
+
+#undef VEC_SUB_VP_VR_VR
+#undef VEC_SUB_VP_VU_VU
+#undef VEC_SUB_VP_VI_VI
+#undef SUB_VP_VR_VR
+#undef SUB_VP_VU_VU
+#undef SUB_VP_VI_VI
+
+! mma_disassemble_acc
+  procedure(sub_atvq) :: __ppc_mma_disassemble_acc
+  interface mma_disassemble_acc
+    procedure :: __ppc_mma_disassemble_acc
+  end interface
+  public mma_disassemble_acc
+
+! mma_disassemble_pair
+  procedure(sub_atvp) :: __ppc_mma_disassemble_pair
+  interface mma_disassemble_pair
+    procedure :: __ppc_mma_disassemble_pair
+  end interface
+  public mma_disassemble_pair
+
+end module
+

flang/test/Lower/PowerPC/ppc-mma-assemble-disassemble.f90

+! RUN: %flang --target=powerpc64le-unknown-linux-gnu -mcpu=pwr10 -emit-llvm -S %s -o - | FileCheck --check-prefixes="CHECK" %s
+! REQUIRES: target=powerpc{{.*}}
+
+! mma_assemble_acc
+
+      subroutine test_assemble_acc_i1()
+      use, intrinsic :: mma
+      implicit none
+      vector(integer(1)) vi10, vi11, vi12, vi13
+      __vector_quad :: cq
+      call mma_assemble_acc(cq, vi10, vi11, vi12, vi13)
+      end subroutine test_assemble_acc_i1
+
+! CHECK-LABEL: @test_assemble_acc_i1
+! CHECK:  %1 = alloca <512 x i1>, i64 1, align 64
+! CHECK:  %2 = alloca <16 x i8>, i64 1, align 16
+! CHECK:  %3 = alloca <16 x i8>, i64 1, align 16
+! CHECK:  %4 = alloca <16 x i8>, i64 1, align 16
+! CHECK:  %5 = alloca <16 x i8>, i64 1, align 16
+! CHECK:  %6 = load <16 x i8>, ptr %2, align 16
+! CHECK:  %7 = load <16 x i8>, ptr %3, align 16
+! CHECK:  %8 = load <16 x i8>, ptr %4, align 16
+! CHECK:  %9 = load <16 x i8>, ptr %5, align 16
+! CHECK:  %10 = call <512 x i1> @llvm.ppc.mma.assemble.acc(<16 x i8> %6, <16 x i8> %7, <16 x i8> %8, <16 x i8> %9)
+! CHECK:  store <512 x i1> %10, ptr %1, align 64
+
+      subroutine test_assemble_acc_i2()
+      use, intrinsic :: mma
+      implicit none
+      vector(integer(2)) vi10, vi11, vi12, vi13
+      __vector_quad :: cq
+      call mma_assemble_acc(cq, vi10, vi11, vi12, vi13)
+      end subroutine test_assemble_acc_i2
+
+! CHECK-LABEL: @test_assemble_acc_i2
+! CHECK:  %1 = alloca <512 x i1>, i64 1, align 64
+! CHECK:  %2 = alloca <8 x i16>, i64 1, align 16
+! CHECK:  %3 = alloca <8 x i16>, i64 1, align 16
+! CHECK:  %4 = alloca <8 x i16>, i64 1, align 16
+! CHECK:  %5 = alloca <8 x i16>, i64 1, align 16
+! CHECK:  %6 = load <8 x i16>, ptr %2, align 16
+! CHECK:  %7 = load <8 x i16>, ptr %3, align 16
+! CHECK:  %8 = load <8 x i16>, ptr %4, align 16
+! CHECK:  %9 = load <8 x i16>, ptr %5, align 16
+! CHECK:  %10 = bitcast <8 x i16> %6 to <16 x i8>
+! CHECK:  %11 = bitcast <8 x i16> %7 to <16 x i8>
+! CHECK:  %12 = bitcast <8 x i16> %8 to <16 x i8>
+! CHECK:  %13 = bitcast <8 x i16> %9 to <16 x i8>
+! CHECK:  %14 = call <512 x i1> @llvm.ppc.mma.assemble.acc(<16 x i8> %10, <16 x i8> %11, <16 x i8> %12, <16 x i8> %13)
+! CHECK:  store <512 x i1> %14, ptr %1, align 64
+
+
+      subroutine test_assemble_acc_i4()
+      use, intrinsic :: mma
+      implicit none
+      vector(integer(4)) vi10, vi11, vi12, vi13
+      __vector_quad :: cq
+      call mma_assemble_acc(cq, vi10, vi11, vi12, vi13)
+      end subroutine test_assemble_acc_i4
+
+! CHECK-LABEL: @test_assemble_acc_i4
+! CHECK:  %1 = alloca <512 x i1>, i64 1, align 64
+! CHECK:  %2 = alloca <4 x i32>, i64 1, align 16
+! CHECK:  %3 = alloca <4 x i32>, i64 1, align 16
+! CHECK:  %4 = alloca <4 x i32>, i64 1, align 16
+! CHECK:  %5 = alloca <4 x i32>, i64 1, align 16
+! CHECK:  %6 = load <4 x i32>, ptr %2, align 16
+! CHECK:  %7 = load <4 x i32>, ptr %3, align 16
+! CHECK:  %8 = load <4 x i32>, ptr %4, align 16
+! CHECK:  %9 = load <4 x i32>, ptr %5, align 16
+! CHECK:  %10 = bitcast <4 x i32> %6 to <16 x i8>
+! CHECK:  %11 = bitcast <4 x i32> %7 to <16 x i8>
+! CHECK:  %12 = bitcast <4 x i32> %8 to <16 x i8>
+! CHECK:  %13 = bitcast <4 x i32> %9 to <16 x i8>
+! CHECK:  %14 = call <512 x i1> @llvm.ppc.mma.assemble.acc(<16 x i8> %10, <16 x i8> %11, <16 x i8> %12, <16 x i8> %13)
+! CHECK:  store <512 x i1> %14, ptr %1, align 64
+
+      subroutine test_assemble_acc_i8()
+      use, intrinsic :: mma
+      implicit none
+      vector(integer(8)) vi10, vi11, vi12, vi13
+      __vector_quad :: cq
+      call mma_assemble_acc(cq, vi10, vi11, vi12, vi13)
+      end subroutine test_assemble_acc_i8
+
+! CHECK-LABEL: @test_assemble_acc_i8
+! CHECK:  %1 = alloca <512 x i1>, i64 1, align 64
+! CHECK:  %2 = alloca <2 x i64>, i64 1, align 16
+! CHECK:  %3 = alloca <2 x i64>, i64 1, align 16
+! CHECK:  %4 = alloca <2 x i64>, i64 1, align 16
+! CHECK:  %5 = alloca <2 x i64>, i64 1, align 16
+! CHECK:  %6 = load <2 x i64>, ptr %2, align 16
+! CHECK:  %7 = load <2 x i64>, ptr %3, align 16
+! CHECK:  %8 = load <2 x i64>, ptr %4, align 16
+! CHECK:  %9 = load <2 x i64>, ptr %5, align 16
+! CHECK:  %10 = bitcast <2 x i64> %6 to <16 x i8>
+! CHECK:  %11 = bitcast <2 x i64> %7 to <16 x i8>
+! CHECK:  %12 = bitcast <2 x i64> %8 to <16 x i8>
+! CHECK:  %13 = bitcast <2 x i64> %9 to <16 x i8>
+! CHECK:  %14 = call <512 x i1> @llvm.ppc.mma.assemble.acc(<16 x i8> %10, <16 x i8> %11, <16 x i8> %12, <16 x i8> %13)
+! CHECK:  store <512 x i1> %14, ptr %1, align 64
+
+
+      subroutine test_assemble_acc_u1()
+      use, intrinsic :: mma
+      implicit none
+      vector(unsigned(1)) vi10, vi11, vi12, vi13
+      __vector_quad :: cq
+      call mma_assemble_acc(cq, vi10, vi11, vi12, vi13)
+      end subroutine test_assemble_acc_u1
+
+! CHECK-LABEL: @test_assemble_acc_u1
+! CHECK:  %1 = alloca <512 x i1>, i64 1, align 64
+! CHECK:  %2 = alloca <16 x i8>, i64 1, align 16
+! CHECK:  %3 = alloca <16 x i8>, i64 1, align 16
+! CHECK:  %4 = alloca <16 x i8>, i64 1, align 16
+! CHECK:  %5 = alloca <16 x i8>, i64 1, align 16
+! CHECK:  %6 = load <16 x i8>, ptr %2, align 16
+! CHECK:  %7 = load <16 x i8>, ptr %3, align 16
+! CHECK:  %8 = load <16 x i8>, ptr %4, align 16
+! CHECK:  %9 = load <16 x i8>, ptr %5, align 16
+! CHECK:  %10 = call <512 x i1> @llvm.ppc.mma.assemble.acc(<16 x i8> %6, <16 x i8> %7, <16 x i8> %8, <16 x i8> %9)
+! CHECK:  store <512 x i1> %10, ptr %1, align 64
+
+      subroutine test_assemble_acc_u2()
+      use, intrinsic :: mma
+      implicit none
+      vector(unsigned(2)) vi10, vi11, vi12, vi13
+      __vector_quad :: cq
+      call mma_assemble_acc(cq, vi10, vi11, vi12, vi13)
+      end subroutine test_assemble_acc_u2
+
+! CHECK-LABEL: @test_assemble_acc_u2
+! CHECK:  %1 = alloca <512 x i1>, i64 1, align 64
+! CHECK:  %2 = alloca <8 x i16>, i64 1, align 16
+! CHECK:  %3 = alloca <8 x i16>, i64 1, align 16
+! CHECK:  %4 = alloca <8 x i16>, i64 1, align 16
+! CHECK:  %5 = alloca <8 x i16>, i64 1, align 16
+! CHECK:  %6 = load <8 x i16>, ptr %2, align 16
+! CHECK:  %7 = load <8 x i16>, ptr %3, align 16
+! CHECK:  %8 = load <8 x i16>, ptr %4, align 16
+! CHECK:  %9 = load <8 x i16>, ptr %5, align 16
+! CHECK:  %10 = bitcast <8 x i16> %6 to <16 x i8>
+! CHECK:  %11 = bitcast <8 x i16> %7 to <16 x i8>
+! CHECK:  %12 = bitcast <8 x i16> %8 to <16 x i8>
+! CHECK:  %13 = bitcast <8 x i16> %9 to <16 x i8>
+! CHECK:  %14 = call <512 x i1> @llvm.ppc.mma.assemble.acc(<16 x i8> %10, <16 x i8> %11, <16 x i8> %12, <16 x i8> %13)
+! CHECK:  store <512 x i1> %14, ptr %1, align 64
+
+      subroutine test_assemble_acc_u4()
+      use, intrinsic :: mma
+      implicit none
+      vector(unsigned(4)) vi10, vi11, vi12, vi13
+      __vector_quad :: cq
+      call mma_assemble_acc(cq, vi10, vi11, vi12, vi13)
+      end subroutine test_assemble_acc_u4
+
+! CHECK-LABEL: @test_assemble_acc_u4
+! CHECK:  %1 = alloca <512 x i1>, i64 1, align 64
+! CHECK:  %2 = alloca <4 x i32>, i64 1, align 16
+! CHECK:  %3 = alloca <4 x i32>, i64 1, align 16
+! CHECK:  %4 = alloca <4 x i32>, i64 1, align 16
+! CHECK:  %5 = alloca <4 x i32>, i64 1, align 16
+! CHECK:  %6 = load <4 x i32>, ptr %2, align 16
+! CHECK:  %7 = load <4 x i32>, ptr %3, align 16
+! CHECK:  %8 = load <4 x i32>, ptr %4, align 16
+! CHECK:  %9 = load <4 x i32>, ptr %5, align 16
+! CHECK:  %10 = bitcast <4 x i32> %6 to <16 x i8>
+! CHECK:  %11 = bitcast <4 x i32> %7 to <16 x i8>
+! CHECK:  %12 = bitcast <4 x i32> %8 to <16 x i8>
+! CHECK:  %13 = bitcast <4 x i32> %9 to <16 x i8>
+! CHECK:  %14 = call <512 x i1> @llvm.ppc.mma.assemble.acc(<16 x i8> %10, <16 x i8> %11, <16 x i8> %12, <16 x i8> %13)
+! CHECK:  store <512 x i1> %14, ptr %1, align 64
+
+      subroutine test_assemble_acc_u8()
+      use, intrinsic :: mma
+      implicit none
+      vector(unsigned(8)) vi10, vi11, vi12, vi13
+      __vector_quad :: cq
+      call mma_assemble_acc(cq, vi10, vi11, vi12, vi13)
+      end subroutine test_assemble_acc_u8
+
+! CHECK-LABEL: @test_assemble_acc_u8
+! CHECK:  %1 = alloca <512 x i1>, i64 1, align 64
+! CHECK:  %2 = alloca <2 x i64>, i64 1, align 16
+! CHECK:  %3 = alloca <2 x i64>, i64 1, align 16
+! CHECK:  %4 = alloca <2 x i64>, i64 1, align 16
+! CHECK:  %5 = alloca <2 x i64>, i64 1, align 16
+! CHECK:  %6 = load <2 x i64>, ptr %2, align 16
+! CHECK:  %7 = load <2 x i64>, ptr %3, align 16
+! CHECK:  %8 = load <2 x i64>, ptr %4, align 16
+! CHECK:  %9 = load <2 x i64>, ptr %5, align 16
+! CHECK:  %10 = bitcast <2 x i64> %6 to <16 x i8>
+! CHECK:  %11 = bitcast <2 x i64> %7 to <16 x i8>
+! CHECK:  %12 = bitcast <2 x i64> %8 to <16 x i8>
+! CHECK:  %13 = bitcast <2 x i64> %9 to <16 x i8>
+! CHECK:  %14 = call <512 x i1> @llvm.ppc.mma.assemble.acc(<16 x i8> %10, <16 x i8> %11, <16 x i8> %12, <16 x i8> %13)
+! CHECK:  store <512 x i1> %14, ptr %1, align 64
+
+      subroutine test_assemble_acc_r4()
+      use, intrinsic :: mma
+      implicit none
+      vector(real(4)) vi10, vi11, vi12, vi13
+      __vector_quad :: cq
+      call mma_assemble_acc(cq, vi10, vi11, vi12, vi13)
+      end subroutine test_assemble_acc_r4
+
+! CHECK-LABEL: @test_assemble_acc_r4
+! CHECK:  %1 = alloca <512 x i1>, i64 1, align 64
+! CHECK:  %2 = alloca <4 x float>, i64 1, align 16
+! CHECK:  %3 = alloca <4 x float>, i64 1, align 16
+! CHECK:  %4 = alloca <4 x float>, i64 1, align 16
+! CHECK:  %5 = alloca <4 x float>, i64 1, align 16
+! CHECK:  %6 = load <4 x float>, ptr %2, align 16
+! CHECK:  %7 = load <4 x float>, ptr %3, align 16
+! CHECK:  %8 = load <4 x float>, ptr %4, align 16
+! CHECK:  %9 = load <4 x float>, ptr %5, align 16
+! CHECK:  %10 = bitcast <4 x float> %6 to <16 x i8>
+! CHECK:  %11 = bitcast <4 x float> %7 to <16 x i8>
+! CHECK:  %12 = bitcast <4 x float> %8 to <16 x i8>
+! CHECK:  %13 = bitcast <4 x float> %9 to <16 x i8>
+! CHECK:  %14 = call <512 x i1> @llvm.ppc.mma.assemble.acc(<16 x i8> %10, <16 x i8> %11, <16 x i8> %12, <16 x i8> %13)
+! CHECK:  store <512 x i1> %14, ptr %1, align 64
+
+      subroutine test_assemble_acc_r8()
+      use, intrinsic :: mma
+      implicit none
+      vector(real(8)) vi10, vi11, vi12, vi13
+      __vector_quad :: cq
+      call mma_assemble_acc(cq, vi10, vi11, vi12, vi13)
+      end subroutine test_assemble_acc_r8
+
+!CHECK-LABEL: @test_assemble_acc_r8
+!CHECK:   %1 = alloca <512 x i1>, i64 1, align 64
+!CHECK:   %2 = alloca <2 x double>, i64 1, align 16
+!CHECK:   %3 = alloca <2 x double>, i64 1, align 16
+!CHECK:   %4 = alloca <2 x double>, i64 1, align 16
+!CHECK:   %5 = alloca <2 x double>, i64 1, align 16
+!CHECK:   %6 = load <2 x double>, ptr %2, align 16
+!CHECK:   %7 = load <2 x double>, ptr %3, align 16
+!CHECK:   %8 = load <2 x double>, ptr %4, align 16
+!CHECK:   %9 = load <2 x double>, ptr %5, align 16
+!CHECK:   %10 = bitcast <2 x double> %6 to <16 x i8>
+!CHECK:   %11 = bitcast <2 x double> %7 to <16 x i8>
+!CHECK:   %12 = bitcast <2 x double> %8 to <16 x i8>
+!CHECK:   %13 = bitcast <2 x double> %9 to <16 x i8>
+!CHECK:   %14 = call <512 x i1> @llvm.ppc.mma.assemble.acc(<16 x i8> %10, <16 x i8> %11, <16 x i8> %12, <16 x i8> %13)
+!CHECK:   store <512 x i1> %14, ptr %1, align 64
+
+! mma_assemble_pair
+
+      subroutine test_mma_assemble_pair_i1()
+      use, intrinsic :: mma
+      implicit none
+      vector(integer(1)) vi10, vi11
+      __vector_pair :: vp
+      call mma_assemble_pair(vp, vi10, vi11)
+      end subroutine test_mma_assemble_pair_i1
+
+!CHECK: @test_mma_assemble_pair_i1_
+!CHECK:  %1 = alloca <16 x i8>, i64 1, align 16
+!CHECK:  %2 = alloca <16 x i8>, i64 1, align 16
+!CHECK:  %3 = alloca <256 x i1>, i64 1, align 32
+!CHECK:  %4 = load <16 x i8>, ptr %1, align 16
+!CHECK:  %5 = load <16 x i8>, ptr %2, align 16
+!CHECK:  %6 = call <256 x i1> @llvm.ppc.vsx.assemble.pair(<16 x i8> %4, <16 x i8> %5)
+!CHECK:  store <256 x i1> %6, ptr %3, align 32
+
+      subroutine test_mma_assemble_pair_i2()
+      use, intrinsic :: mma
+      implicit none
+      vector(integer(2)) vi10, vi11
+      __vector_pair :: vp
+      call mma_assemble_pair(vp, vi10, vi11)
+      end subroutine test_mma_assemble_pair_i2
+
+!CHECK: @test_mma_assemble_pair_i2_
+!CHECK:  %1 = alloca <8 x i16>, i64 1, align 16
+!CHECK:  %2 = alloca <8 x i16>, i64 1, align 16
+!CHECK:  %3 = alloca <256 x i1>, i64 1, align 32
+!CHECK:  %4 = load <8 x i16>, ptr %1, align 16
+!CHECK:  %5 = load <8 x i16>, ptr %2, align 16
+!CHECK:  %6 = bitcast <8 x i16> %4 to <16 x i8>
+!CHECK:  %7 = bitcast <8 x i16> %5 to <16 x i8>
+!CHECK:  %8 = call <256 x i1> @llvm.ppc.vsx.assemble.pair(<16 x i8> %6, <16 x i8> %7)
+!CHECK:  store <256 x i1> %8, ptr %3, align 32
+
+      subroutine test_mma_assemble_pair_i4()
+      use, intrinsic :: mma
+      implicit none
+      vector(integer(4)) vi10, vi11
+      __vector_pair :: vp
+      call mma_assemble_pair(vp, vi10, vi11)
+      end subroutine test_mma_assemble_pair_i4
+
+!CHECK: @test_mma_assemble_pair_i4_
+!CHECK:  %1 = alloca <4 x i32>, i64 1, align 16
+!CHECK:  %2 = alloca <4 x i32>, i64 1, align 16
+!CHECK:  %3 = alloca <256 x i1>, i64 1, align 32
+!CHECK:  %4 = load <4 x i32>, ptr %1, align 16
+!CHECK:  %5 = load <4 x i32>, ptr %2, align 16
+!CHECK:  %6 = bitcast <4 x i32> %4 to <16 x i8>
+!CHECK:  %7 = bitcast <4 x i32> %5 to <16 x i8>
+!CHECK:  %8 = call <256 x i1> @llvm.ppc.vsx.assemble.pair(<16 x i8> %6, <16 x i8> %7)
+!CHECK:  store <256 x i1> %8, ptr %3, align 32
+
+      subroutine test_mma_assemble_pair_i8()
+      use, intrinsic :: mma
+      implicit none
+      vector(integer(8)) vi10, vi11
+      __vector_pair :: vp
+      call mma_assemble_pair(vp, vi10, vi11)
+      end subroutine test_mma_assemble_pair_i8
+
+!CHECK: @test_mma_assemble_pair_i8_
+!CHECK:  %1 = alloca <2 x i64>, i64 1, align 16
+!CHECK:  %2 = alloca <2 x i64>, i64 1, align 16
+!CHECK:  %3 = alloca <256 x i1>, i64 1, align 32
+!CHECK:  %4 = load <2 x i64>, ptr %1, align 16
+!CHECK:  %5 = load <2 x i64>, ptr %2, align 16
+!CHECK:  %6 = bitcast <2 x i64> %4 to <16 x i8>
+!CHECK:  %7 = bitcast <2 x i64> %5 to <16 x i8>
+!CHECK:  %8 = call <256 x i1> @llvm.ppc.vsx.assemble.pair(<16 x i8> %6, <16 x i8> %7)
+!CHECK:  store <256 x i1> %8, ptr %3, align 32
+
+      subroutine test_mma_assemble_pair_u1()
+      use, intrinsic :: mma
+      implicit none
+      vector(unsigned(1)) vi10, vi11
+      __vector_pair :: vp
+      call mma_assemble_pair(vp, vi10, vi11)
+      end subroutine test_mma_assemble_pair_u1
+
+!CHECK: @test_mma_assemble_pair_u1_
+!CHECK:  %1 = alloca <16 x i8>, i64 1, align 16
+!CHECK:  %2 = alloca <16 x i8>, i64 1, align 16
+!CHECK:  %3 = alloca <256 x i1>, i64 1, align 32
+!CHECK:  %4 = load <16 x i8>, ptr %1, align 16
+!CHECK:  %5 = load <16 x i8>, ptr %2, align 16
+!CHECK:  %6 = call <256 x i1> @llvm.ppc.vsx.assemble.pair(<16 x i8> %4, <16 x i8> %5)
+!CHECK:  store <256 x i1> %6, ptr %3, align 32
+
+      subroutine test_mma_assemble_pair_u2()
+      use, intrinsic :: mma
+      implicit none
+      vector(unsigned(2)) vi10, vi11
+      __vector_pair :: vp
+      call mma_assemble_pair(vp, vi10, vi11)
+      end subroutine test_mma_assemble_pair_u2
+
+!CHECK: @test_mma_assemble_pair_u2_
+!CHECK:  %1 = alloca <8 x i16>, i64 1, align 16
+!CHECK:  %2 = alloca <8 x i16>, i64 1, align 16
+!CHECK:  %3 = alloca <256 x i1>, i64 1, align 32
+!CHECK:  %4 = load <8 x i16>, ptr %1, align 16
+!CHECK:  %5 = load <8 x i16>, ptr %2, align 16
+!CHECK:  %6 = bitcast <8 x i16> %4 to <16 x i8>
+!CHECK:  %7 = bitcast <8 x i16> %5 to <16 x i8>
+!CHECK:  %8 = call <256 x i1> @llvm.ppc.vsx.assemble.pair(<16 x i8> %6, <16 x i8> %7)
+!CHECK:  store <256 x i1> %8, ptr %3, align 32
+
+      subroutine test_mma_assemble_pair_u4()
+      use, intrinsic :: mma
+      implicit none
+      vector(unsigned(4)) vi10, vi11
+      __vector_pair :: vp
+      call mma_assemble_pair(vp, vi10, vi11)
+      end subroutine test_mma_assemble_pair_u4
+
+!CHECK: @test_mma_assemble_pair_u4_
+!CHECK:  %1 = alloca <4 x i32>, i64 1, align 16
+!CHECK:  %2 = alloca <4 x i32>, i64 1, align 16
+!CHECK:  %3 = alloca <256 x i1>, i64 1, align 32
+!CHECK:  %4 = load <4 x i32>, ptr %1, align 16
+!CHECK:  %5 = load <4 x i32>, ptr %2, align 16
+!CHECK:  %6 = bitcast <4 x i32> %4 to <16 x i8>
+!CHECK:  %7 = bitcast <4 x i32> %5 to <16 x i8>
+!CHECK:  %8 = call <256 x i1> @llvm.ppc.vsx.assemble.pair(<16 x i8> %6, <16 x i8> %7)
+!CHECK:  store <256 x i1> %8, ptr %3, align 32
+
+      subroutine test_mma_assemble_pair_u8()
+      use, intrinsic :: mma
+      implicit none
+      vector(unsigned(8)) vi10, vi11
+      __vector_pair :: vp
+      call mma_assemble_pair(vp, vi10, vi11)
+      end subroutine test_mma_assemble_pair_u8
+
+!CHECK: @test_mma_assemble_pair_u8_
+!CHECK:  %1 = alloca <2 x i64>, i64 1, align 16
+!CHECK:  %2 = alloca <2 x i64>, i64 1, align 16
+!CHECK:  %3 = alloca <256 x i1>, i64 1, align 32
+!CHECK:  %4 = load <2 x i64>, ptr %1, align 16
+!CHECK:  %5 = load <2 x i64>, ptr %2, align 16
+!CHECK:  %6 = bitcast <2 x i64> %4 to <16 x i8>
+!CHECK:  %7 = bitcast <2 x i64> %5 to <16 x i8>
+!CHECK:  %8 = call <256 x i1> @llvm.ppc.vsx.assemble.pair(<16 x i8> %6, <16 x i8> %7)
+!CHECK:  store <256 x i1> %8, ptr %3, align 32
+
+      subroutine test_mma_assemble_pair_r4()
+      use, intrinsic :: mma
+      implicit none
+      vector(real(4)) vi10, vi11
+      __vector_pair :: vp
+      call mma_assemble_pair(vp, vi10, vi11)
+      end subroutine test_mma_assemble_pair_r4
+
+!CHECK: @test_mma_assemble_pair_r4_
+!CHECK:  %1 = alloca <4 x float>, i64 1, align 16
+!CHECK:  %2 = alloca <4 x float>, i64 1, align 16
+!CHECK:  %3 = alloca <256 x i1>, i64 1, align 32
+!CHECK:  %4 = load <4 x float>, ptr %1, align 16
+!CHECK:  %5 = load <4 x float>, ptr %2, align 16
+!CHECK:  %6 = bitcast <4 x float> %4 to <16 x i8>
+!CHECK:  %7 = bitcast <4 x float> %5 to <16 x i8>
+!CHECK:  %8 = call <256 x i1> @llvm.ppc.vsx.assemble.pair(<16 x i8> %6, <16 x i8> %7)
+!CHECK:  store <256 x i1> %8, ptr %3, align 32
+
+      subroutine test_mma_assemble_pair_r8()
+      use, intrinsic :: mma
+      implicit none
+      vector(real(8)) vi10, vi11
+      __vector_pair :: vp
+      call mma_assemble_pair(vp, vi10, vi11)
+      end subroutine test_mma_assemble_pair_r8
+
+!CHECK: @test_mma_assemble_pair_r8_
+!CHECK:  %1 = alloca <2 x double>, i64 1, align 16
+!CHECK:  %2 = alloca <2 x double>, i64 1, align 16
+!CHECK:  %3 = alloca <256 x i1>, i64 1, align 32
+!CHECK:  %4 = load <2 x double>, ptr %1, align 16
+!CHECK:  %5 = load <2 x double>, ptr %2, align 16
+!CHECK:  %6 = bitcast <2 x double> %4 to <16 x i8>
+!CHECK:  %7 = bitcast <2 x double> %5 to <16 x i8>
+!CHECK:  %8 = call <256 x i1> @llvm.ppc.vsx.assemble.pair(<16 x i8> %6, <16 x i8> %7)
+!CHECK:  store <256 x i1> %8, ptr %3, align 32
+
+! mma_disassemble_acc
+
+      subroutine test_mma_build_acc_i1()
+      use, intrinsic :: mma
+      implicit none
+      vector(integer(1)) vi10, vi11, vi12, vi13
+      __vector_quad :: cq
+      call mma_build_acc(cq, vi10, vi11, vi12, vi13)
+      end subroutine test_mma_build_acc_i1
+
+!CHECK-LABEL: @test_mma_build_acc_i1
+!CHECK:  %1 = alloca <512 x i1>, i64 1, align 64
+!CHECK:  %2 = alloca <16 x i8>, i64 1, align 16
+!CHECK:  %3 = alloca <16 x i8>, i64 1, align 16
+!CHECK:  %4 = alloca <16 x i8>, i64 1, align 16
+!CHECK:  %5 = alloca <16 x i8>, i64 1, align 16
+!CHECK:  %6 = load <16 x i8>, ptr %2, align 16
+!CHECK:  %7 = load <16 x i8>, ptr %3, align 16
+!CHECK:  %8 = load <16 x i8>, ptr %4, align 16
+!CHECK:  %9 = load <16 x i8>, ptr %5, align 16
+!CHECK:  %10 = call <512 x i1> @llvm.ppc.mma.assemble.acc(<16 x i8> %9, <16 x i8> %8, <16 x i8> %7, <16 x i8> %6)
+!CHECK:  store <512 x i1> %10, ptr %1, align 64
+
+      subroutine test_mma_build_acc_i2()
+      use, intrinsic :: mma
+      implicit none
+      vector(integer(2)) vi10, vi11, vi12, vi13
+      __vector_quad :: cq
+      call mma_build_acc(cq, vi10, vi11, vi12, vi13)
+      end subroutine test_mma_build_acc_i2
+
+!CHECK-LABEL: @test_mma_build_acc_i2
+!CHECK:  %1 = alloca <512 x i1>, i64 1, align 64
+!CHECK:  %2 = alloca <8 x i16>, i64 1, align 16
+!CHECK:  %3 = alloca <8 x i16>, i64 1, align 16
+!CHECK:  %4 = alloca <8 x i16>, i64 1, align 16
+!CHECK:  %5 = alloca <8 x i16>, i64 1, align 16
+!CHECK:  %6 = load <8 x i16>, ptr %2, align 16
+!CHECK:  %7 = load <8 x i16>, ptr %3, align 16
+!CHECK:  %8 = load <8 x i16>, ptr %4, align 16
+!CHECK:  %9 = load <8 x i16>, ptr %5, align 16
+!CHECK:  %10 = bitcast <8 x i16> %9 to <16 x i8>
+!CHECK:  %11 = bitcast <8 x i16> %8 to <16 x i8>
+!CHECK:  %12 = bitcast <8 x i16> %7 to <16 x i8>
+!CHECK:  %13 = bitcast <8 x i16> %6 to <16 x i8>
+!CHECK:  %14 = call <512 x i1> @llvm.ppc.mma.assemble.acc(<16 x i8> %10, <16 x i8> %11, <16 x i8> %12, <16 x i8> %13)
+!CHECK:  store <512 x i1> %14, ptr %1, align 64
+
+      subroutine test_mma_build_acc_i4()
+      use, intrinsic :: mma
+      implicit none
+      vector(integer(4)) vi10, vi11, vi12, vi13
+      __vector_quad :: cq
+      call mma_build_acc(cq, vi10, vi11, vi12, vi13)
+      end subroutine test_mma_build_acc_i4
+
+!CHECK-LABEL: @test_mma_build_acc_i4
+!CHECK:  %1 = alloca <512 x i1>, i64 1, align 64
+!CHECK:  %2 = alloca <4 x i32>, i64 1, align 16
+!CHECK:  %3 = alloca <4 x i32>, i64 1, align 16
+!CHECK:  %4 = alloca <4 x i32>, i64 1, align 16
+!CHECK:  %5 = alloca <4 x i32>, i64 1, align 16
+!CHECK:  %6 = load <4 x i32>, ptr %2, align 16
+!CHECK:  %7 = load <4 x i32>, ptr %3, align 16
+!CHECK:  %8 = load <4 x i32>, ptr %4, align 16
+!CHECK:  %9 = load <4 x i32>, ptr %5, align 16
+!CHECK:  %10 = bitcast <4 x i32> %9 to <16 x i8>
+!CHECK:  %11 = bitcast <4 x i32> %8 to <16 x i8>
+!CHECK:  %12 = bitcast <4 x i32> %7 to <16 x i8>
+!CHECK:  %13 = bitcast <4 x i32> %6 to <16 x i8>
+!CHECK:  %14 = call <512 x i1> @llvm.ppc.mma.assemble.acc(<16 x i8> %10, <16 x i8> %11, <16 x i8> %12, <16 x i8> %13)
+!CHECK:  store <512 x i1> %14, ptr %1, align 64
+
+      subroutine test_mma_build_acc_i8()
+      use, intrinsic :: mma
+      implicit none
+      vector(integer(8)) vi10, vi11, vi12, vi13
+      __vector_quad :: cq
+      call mma_build_acc(cq, vi10, vi11, vi12, vi13)
+      end subroutine test_mma_build_acc_i8
+
+!CHECK-LABEL: @test_mma_build_acc_i8
+!CHECK:  %1 = alloca <512 x i1>, i64 1, align 64
+!CHECK:  %2 = alloca <2 x i64>, i64 1, align 16
+!CHECK:  %3 = alloca <2 x i64>, i64 1, align 16
+!CHECK:  %4 = alloca <2 x i64>, i64 1, align 16
+!CHECK:  %5 = alloca <2 x i64>, i64 1, align 16
+!CHECK:  %6 = load <2 x i64>, ptr %2, align 16
+!CHECK:  %7 = load <2 x i64>, ptr %3, align 16
+!CHECK:  %8 = load <2 x i64>, ptr %4, align 16
+!CHECK:  %9 = load <2 x i64>, ptr %5, align 16
+!CHECK:  %10 = bitcast <2 x i64> %9 to <16 x i8>
+!CHECK:  %11 = bitcast <2 x i64> %8 to <16 x i8>
+!CHECK:  %12 = bitcast <2 x i64> %7 to <16 x i8>
+!CHECK:  %13 = bitcast <2 x i64> %6 to <16 x i8>
+!CHECK:  %14 = call <512 x i1> @llvm.ppc.mma.assemble.acc(<16 x i8> %10, <16 x i8> %11, <16 x i8> %12, <16 x i8> %13)
+!CHECK:  store <512 x i1> %14, ptr %1, align 64
+
+      subroutine test_mma_build_acc_u1()
+      use, intrinsic :: mma
+      implicit none
+      vector(unsigned(1)) vi10, vi11, vi12, vi13
+      __vector_quad :: cq
+      call mma_build_acc(cq, vi10, vi11, vi12, vi13)
+      end subroutine test_mma_build_acc_u1
+
+!CHECK-LABEL: @test_mma_build_acc_u1
+!CHECK:  %1 = alloca <512 x i1>, i64 1, align 64
+!CHECK:  %2 = alloca <16 x i8>, i64 1, align 16
+!CHECK:  %3 = alloca <16 x i8>, i64 1, align 16
+!CHECK:  %4 = alloca <16 x i8>, i64 1, align 16
+!CHECK:  %5 = alloca <16 x i8>, i64 1, align 16
+!CHECK:  %6 = load <16 x i8>, ptr %2, align 16
+!CHECK:  %7 = load <16 x i8>, ptr %3, align 16
+!CHECK:  %8 = load <16 x i8>, ptr %4, align 16
+!CHECK:  %9 = load <16 x i8>, ptr %5, align 16
+!CHECK:  %10 = call <512 x i1> @llvm.ppc.mma.assemble.acc(<16 x i8> %9, <16 x i8> %8, <16 x i8> %7, <16 x i8> %6)
+!CHECK:  store <512 x i1> %10, ptr %1, align 64
+
+      subroutine test_mma_build_acc_u2()
+      use, intrinsic :: mma
+      implicit none
+      vector(unsigned(2)) vi10, vi11, vi12, vi13
+      __vector_quad :: cq
+      call mma_build_acc(cq, vi10, vi11, vi12, vi13)
+      end subroutine test_mma_build_acc_u2
+
+!CHECK-LABEL: @test_mma_build_acc_u2
+!CHECK:  %1 = alloca <512 x i1>, i64 1, align 64
+!CHECK:  %2 = alloca <8 x i16>, i64 1, align 16
+!CHECK:  %3 = alloca <8 x i16>, i64 1, align 16
+!CHECK:  %4 = alloca <8 x i16>, i64 1, align 16
+!CHECK:  %5 = alloca <8 x i16>, i64 1, align 16
+!CHECK:  %6 = load <8 x i16>, ptr %2, align 16
+!CHECK:  %7 = load <8 x i16>, ptr %3, align 16
+!CHECK:  %8 = load <8 x i16>, ptr %4, align 16
+!CHECK:  %9 = load <8 x i16>, ptr %5, align 16
+!CHECK:  %10 = bitcast <8 x i16> %9 to <16 x i8>
+!CHECK:  %11 = bitcast <8 x i16> %8 to <16 x i8>
+!CHECK:  %12 = bitcast <8 x i16> %7 to <16 x i8>
+!CHECK:  %13 = bitcast <8 x i16> %6 to <16 x i8>
+!CHECK:  %14 = call <512 x i1> @llvm.ppc.mma.assemble.acc(<16 x i8> %10, <16 x i8> %11, <16 x i8> %12, <16 x i8> %13)
+!CHECK:  store <512 x i1> %14, ptr %1, align 64
+
+      subroutine test_mma_build_acc_u4()
+      use, intrinsic :: mma
+      implicit none
+      vector(unsigned(4)) vi10, vi11, vi12, vi13
+      __vector_quad :: cq
+      call mma_build_acc(cq, vi10, vi11, vi12, vi13)
+      end subroutine test_mma_build_acc_u4
+
+!CHECK-LABEL: @test_mma_build_acc_u4
+!CHECK:  %1 = alloca <512 x i1>, i64 1, align 64
+!CHECK:  %2 = alloca <4 x i32>, i64 1, align 16
+!CHECK:  %3 = alloca <4 x i32>, i64 1, align 16
+!CHECK:  %4 = alloca <4 x i32>, i64 1, align 16
+!CHECK:  %5 = alloca <4 x i32>, i64 1, align 16
+!CHECK:  %6 = load <4 x i32>, ptr %2, align 16
+!CHECK:  %7 = load <4 x i32>, ptr %3, align 16
+!CHECK:  %8 = load <4 x i32>, ptr %4, align 16
+!CHECK:  %9 = load <4 x i32>, ptr %5, align 16
+!CHECK:  %10 = bitcast <4 x i32> %9 to <16 x i8>
+!CHECK:  %11 = bitcast <4 x i32> %8 to <16 x i8>
+!CHECK:  %12 = bitcast <4 x i32> %7 to <16 x i8>
+!CHECK:  %13 = bitcast <4 x i32> %6 to <16 x i8>
+!CHECK:  %14 = call <512 x i1> @llvm.ppc.mma.assemble.acc(<16 x i8> %10, <16 x i8> %11, <16 x i8> %12, <16 x i8> %13)
+!CHECK:  store <512 x i1> %14, ptr %1, align 64
+
+      subroutine test_mma_build_acc_u8()
+      use, intrinsic :: mma
+      implicit none
+      vector(unsigned(8)) vi10, vi11, vi12, vi13
+      __vector_quad :: cq
+      call mma_build_acc(cq, vi10, vi11, vi12, vi13)
+      end subroutine test_mma_build_acc_u8
+
+!CHECK-LABEL: @test_mma_build_acc_u8
+!CHECK:  %1 = alloca <512 x i1>, i64 1, align 64
+!CHECK:  %2 = alloca <2 x i64>, i64 1, align 16
+!CHECK:  %3 = alloca <2 x i64>, i64 1, align 16
+!CHECK:  %4 = alloca <2 x i64>, i64 1, align 16
+!CHECK:  %5 = alloca <2 x i64>, i64 1, align 16
+!CHECK:  %6 = load <2 x i64>, ptr %2, align 16
+!CHECK:  %7 = load <2 x i64>, ptr %3, align 16
+!CHECK:  %8 = load <2 x i64>, ptr %4, align 16
+!CHECK:  %9 = load <2 x i64>, ptr %5, align 16
+!CHECK:  %10 = bitcast <2 x i64> %9 to <16 x i8>
+!CHECK:  %11 = bitcast <2 x i64> %8 to <16 x i8>
+!CHECK:  %12 = bitcast <2 x i64> %7 to <16 x i8>
+!CHECK:  %13 = bitcast <2 x i64> %6 to <16 x i8>
+!CHECK:  %14 = call <512 x i1> @llvm.ppc.mma.assemble.acc(<16 x i8> %10, <16 x i8> %11, <16 x i8> %12, <16 x i8> %13)
+!CHECK:  store <512 x i1> %14, ptr %1, align 64
+
+
+      subroutine test_mma_build_acc_r4()
+      use, intrinsic :: mma
+      implicit none
+      vector(real(4)) vi10, vi11, vi12, vi13
+      __vector_quad :: cq
+      call mma_build_acc(cq, vi10, vi11, vi12, vi13)
+      end subroutine test_mma_build_acc_r4
+
+!CHECK-LABEL: @test_mma_build_acc_r4
+!CHECK:  %1 = alloca <512 x i1>, i64 1, align 64
+!CHECK:  %2 = alloca <4 x float>, i64 1, align 16
+!CHECK:  %3 = alloca <4 x float>, i64 1, align 16
+!CHECK:  %4 = alloca <4 x float>, i64 1, align 16
+!CHECK:  %5 = alloca <4 x float>, i64 1, align 16
+!CHECK:  %6 = load <4 x float>, ptr %2, align 16
+!CHECK:  %7 = load <4 x float>, ptr %3, align 16
+!CHECK:  %8 = load <4 x float>, ptr %4, align 16
+!CHECK:  %9 = load <4 x float>, ptr %5, align 16
+!CHECK:  %10 = bitcast <4 x float> %9 to <16 x i8>
+!CHECK:  %11 = bitcast <4 x float> %8 to <16 x i8>
+!CHECK:  %12 = bitcast <4 x float> %7 to <16 x i8>
+!CHECK:  %13 = bitcast <4 x float> %6 to <16 x i8>
+!CHECK:  %14 = call <512 x i1> @llvm.ppc.mma.assemble.acc(<16 x i8> %10, <16 x i8> %11, <16 x i8> %12, <16 x i8> %13)
+!CHECK:  store <512 x i1> %14, ptr %1, align 64
+
+
+      subroutine test_mma_build_acc_r8()
+      use, intrinsic :: mma
+      implicit none
+      vector(real(8)) vi10, vi11, vi12, vi13
+      __vector_quad :: cq
+      call mma_build_acc(cq, vi10, vi11, vi12, vi13)
+      end subroutine test_mma_build_acc_r8
+
+!CHECK-LABEL: @test_mma_build_acc_r8
+!CHECK:  %1 = alloca <512 x i1>, i64 1, align 64
+!CHECK:  %2 = alloca <2 x double>, i64 1, align 16
+!CHECK:  %3 = alloca <2 x double>, i64 1, align 16
+!CHECK:  %4 = alloca <2 x double>, i64 1, align 16
+!CHECK:  %5 = alloca <2 x double>, i64 1, align 16
+!CHECK:  %6 = load <2 x double>, ptr %2, align 16
+!CHECK:  %7 = load <2 x double>, ptr %3, align 16
+!CHECK:  %8 = load <2 x double>, ptr %4, align 16
+!CHECK:  %9 = load <2 x double>, ptr %5, align 16
+!CHECK:  %10 = bitcast <2 x double> %9 to <16 x i8>
+!CHECK:  %11 = bitcast <2 x double> %8 to <16 x i8>
+!CHECK:  %12 = bitcast <2 x double> %7 to <16 x i8>
+!CHECK:  %13 = bitcast <2 x double> %6 to <16 x i8>
+!CHECK:  %14 = call <512 x i1> @llvm.ppc.mma.assemble.acc(<16 x i8> %10, <16 x i8> %11, <16 x i8> %12, <16 x i8> %13)
+!CHECK:  store <512 x i1> %14, ptr %1, align 64
+
+! mma_disassemble_acc
+
+      subroutine test_disassemble_acc()
+      use, intrinsic :: mma
+      implicit none
+      __vector_quad :: vq
+      real :: data
+      call mma_disassemble_acc(data, vq)
+      end subroutine
+
+!CHECK-LABEL: @test_disassemble_acc_
+!CHECK:  %1 = alloca float, i64 1, align 4
+!CHECK:  %2 = alloca <512 x i1>, i64 1, align 64
+!CHECK:  %3 = load <512 x i1>, ptr %2, align 64
+!CHECK:  %4 = call { <16 x i8>, <16 x i8>, <16 x i8>, <16 x i8> } @llvm.ppc.mma.disassemble.acc(<512 x i1> %3)
+!CHECK:  store { <16 x i8>, <16 x i8>, <16 x i8>, <16 x i8> } %4, ptr %1, align 16
+
+! mma_disassemble_pair
+
+      subroutine test_disassemble_pair()
+      use, intrinsic :: mma
+      implicit none
+      __vector_pair :: vp
+      real :: data
+      call mma_disassemble_pair(data, vp)
+      end subroutine
+
+!CHECK-LABEL: @test_disassemble_pair_
+!CHECK:  %1 = alloca float, i64 1, align 4
+!CHECK:  %2 = alloca <256 x i1>, i64 1, align 32
+!CHECK:  %3 = load <256 x i1>, ptr %2, align 32
+!CHECK:  %4 = call { <16 x i8>, <16 x i8> } @llvm.ppc.vsx.disassemble.pair(<256 x i1> %3)
+!CHECK:  store { <16 x i8>, <16 x i8> } %4, ptr %1, align 16

flang/tools/f18/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   FrontendOpenACC
   FrontendOpenMP
   Support
   )
 
 set(MODULES
   "__fortran_builtins"
   "__fortran_ieee_exceptions"
   "__fortran_type_info"
   "__ppc_types"
   "__ppc_intrinsics"
+  "mma"
   "__cuda_builtins"
   "ieee_arithmetic"
   "ieee_exceptions"
   "ieee_features"
   "iso_c_binding"
   "iso_fortran_env"
   "omp_lib"
   "__fortran_builtins"
   "__fortran_type_info"
 )
 
 # Create module files directly from the top-level module source directory.
 # If CMAKE_CROSSCOMPILING, then the newly built flang-new executable was
 # cross compiled, and thus can't be executed on the build system and thus
 # can't be used for generating module files.
 if (NOT CMAKE_CROSSCOMPILING)
   foreach(filename ${MODULES})
     set(base ${FLANG_INTRINSIC_MODULES_DIR}/${filename})
     if(${filename} STREQUAL "__fortran_builtins")
       set(depends "")
     elseif(${filename} STREQUAL "__ppc_types")
       set(depends "")
-    elseif(${filename} STREQUAL "__ppc_intrinsics")
+    elseif(${filename} STREQUAL "__ppc_intrinsics" OR
+           ${filename} STREQUAL "mma")
       set(depends ${FLANG_INTRINSIC_MODULES_DIR}/__ppc_types.mod)
     else()
       set(depends ${FLANG_INTRINSIC_MODULES_DIR}/__fortran_builtins.mod)
       if(NOT ${filename} STREQUAL "__fortran_type_info")
         set(depends ${FLANG_INTRINSIC_MODULES_DIR}/__fortran_type_info.mod)
       endif()
       if(${filename} STREQUAL "ieee_arithmetic" OR
          ${filename} STREQUAL "ieee_exceptions")
         set(depends ${FLANG_INTRINSIC_MODULES_DIR}/__fortran_ieee_exceptions.mod)
       endif()
     endif()
 
     # The module contains PPC vector types that needs the PPC target.
     set(opts "")
-    if(${filename} STREQUAL "__ppc_intrinsics")
+    if(${filename} STREQUAL "__ppc_intrinsics" OR
+       ${filename} STREQUAL "mma")
       set(opts "--target=ppc64le")
     endif()
 
     add_custom_command(OUTPUT ${base}.mod
       COMMAND ${CMAKE_COMMAND} -E make_directory ${FLANG_INTRINSIC_MODULES_DIR}
       COMMAND flang-new -cpp -fsyntax-only ${opts} -module-dir ${FLANG_INTRINSIC_MODULES_DIR}
         ${FLANG_SOURCE_DIR}/module/${filename}.f90
       DEPENDS flang-new ${FLANG_SOURCE_DIR}/module/${filename}.f90 ${depends}