[NVPTX] Add lowering of i128 params.

[denzp]

Summary

The patch adds support of i128 params lowering. The changes are quite trivial to support i128 as a "special case" of integer type. With this patch, we lower i128 params the same way as aggregates of size 16 bytes: .param .b8 _ [16].

Currently, NVPTX can't deal with the 128 bit integers:

• in some cases because of failed assertions like ValVTs.size() == OutVals.size() && "Bad return value decomposition"
• in other cases emitting PTX with .i128 or .u128 types (which are not valid [1])

[1] http://docs.nvidia.com/cuda/parallel-thread-execution/index.html#fundamental-types

[jlebar]

I'd like tra to look at this -- he's much more of an expert in selection DAG than I am -- but he's on vacation for another week. Are you OK waiting?

[denzp]

In D34555#789076, @jlebar wrote:

I'd like tra to look at this -- he's much more of an expert in selection DAG than I am -- but he's on vacation for another week. Are you OK waiting?

Of course! Code review should never be hurried :)

[tra]

In general the patch looks OK.
One thing that you need to make sure is that alignment of i128 on nvptx side always matches that on the host side. AFAICT, i128 alignment on x64 is 16 (see D28990) , but this patch shows that it's 8 on nvptx side. That would result in different in-memory representation of aggregates between host and device.

lib/Target/NVPTX/NVPTXAsmPrinter.cpp

403 ↗

(On Diff #103725)

Nit. I'd group together (Ty->isIntegerTy() && !Ty->isIntegerTy(128))

1428–1434 ↗

(On Diff #103725)

Please add a test case for this.

lib/Target/NVPTX/NVPTXISelLowering.cpp

1279 ↗

(On Diff #103725)

Same nit as above.

[denzp]

Fixed i128 params alignment
Added regression test for global variables
Couple minor code readability improvements

[denzp]

Thanks for the review, @tra! I fixed alignment and some lines according to your comments.

[tra]

Nice. With alignment=16 we also benefit from 128-bit loads/stores.

[denzp]

Could you please land it for me?

[mkuper]

I've reverted this in r307334 because it breaks a lot of clang tests with:

error: 'error' diagnostics seen but not expected:

(frontend): backend data layout 'e-i64:64-i128:128-v16:16-v32:32-n16:32:64' does not match expected target description 'e-i64:64-v16:16-v32:32-n16:32:64'

[denzp]

Is it safe just to change data layout at clang side? Won't it break backward compatibility?

clang-nvptx-i128-alignment.diff1 KBDownload

[tra]

In D34555#802008, @denzp wrote:

Is it safe just to change data layout at clang side? Won't it break backward compatibility?

Clang and LLVM sides of this change should land together as they have to be in sync.

Other than that, AFAICT it should be safe as it affects only i128 which is not currently supported, so there should be no backward compatibility issues. That's in theory. You may want to ask this question on cfe-dev@.

[denzp]

In D34555#802254, @tra wrote:

In D34555#802008, @denzp wrote:

Is it safe just to change data layout at clang side? Won't it break backward compatibility?

Clang and LLVM sides of this change should land together as they have to be in sync.

Other than that, AFAICT it should be safe as it affects only i128 which is not currently supported, so there should be no backward compatibility issues. That's in theory. You may want to ask this question on cfe-dev@.

Looks like we can simply update Clang's data layout and it should not break anything: cfe-dev thread.

Do I need to create a "differential" for the clang patch? I attached the diff to my previous comment here, and changes there are quite trivial.

cfe/trunk/lib/Basic/Targets.cpp

NVPTXTargetInfo(const llvm::Triple &Triple, const TargetOptions &Opts,
     UseAddrSpaceMapMangling = true;
 
     // Define available target features
     // These must be defined in sorted order!
     NoAsmVariants = true;
     GPU = CudaArch::SM_20;
 
     if (TargetPointerWidth == 32)
-      resetDataLayout("e-p:32:32-i64:64-v16:16-v32:32-n16:32:64");
+      resetDataLayout("e-p:32:32-i64:64-i128:128-v16:16-v32:32-n16:32:64");
     else
-      resetDataLayout("e-i64:64-v16:16-v32:32-n16:32:64");
+      resetDataLayout("e-i64:64-i128:128-v16:16-v32:32-n16:32:64");
 
     // If possible, get a TargetInfo for our host triple, so we can match its
     // types.
     llvm::Triple HostTriple(Opts.HostTriple);
     if (!HostTriple.isNVPTX())
       HostTarget.reset(AllocateTarget(llvm::Triple(Opts.HostTriple), Opts));
 
     // If no host target, make some guesses about the data layout and return.

cfe/trunk/test/CodeGen/target-data.c

 // PPC32-DARWIN: target datalayout = "E-m:o-p:32:32-f64:32:64-n32"
 
 // RUN: %clang_cc1 -triple powerpc64-darwin -o - -emit-llvm %s | \
 // RUN: FileCheck %s -check-prefix=PPC64-DARWIN
 // PPC64-DARWIN: target datalayout = "E-m:o-i64:64-n32:64"
 
 // RUN: %clang_cc1 -triple nvptx-unknown -o - -emit-llvm %s | \
 // RUN: FileCheck %s -check-prefix=NVPTX
-// NVPTX: target datalayout = "e-p:32:32-i64:64-v16:16-v32:32-n16:32:64"
+// NVPTX: target datalayout = "e-p:32:32-i64:64-i128:128-v16:16-v32:32-n16:32:64"
 
 // RUN: %clang_cc1 -triple nvptx64-unknown -o - -emit-llvm %s | \
 // RUN: FileCheck %s -check-prefix=NVPTX64
-// NVPTX64: target datalayout = "e-i64:64-v16:16-v32:32-n16:32:64"
+// NVPTX64: target datalayout = "e-i64:64-i128:128-v16:16-v32:32-n16:32:64"
 
 // RUN: %clang_cc1 -triple r600-unknown -o - -emit-llvm %s | \
 // RUN: FileCheck %s -check-prefix=R600
 // R600: target datalayout = "e-p:32:32-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64"
 
 // RUN: %clang_cc1 -triple r600-unknown -target-cpu cayman -o - -emit-llvm %s \
 // RUN: | FileCheck %s -check-prefix=R600D
 // R600D: target datalayout = "e-p:32:32-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64"

llvm/trunk/lib/Target/NVPTX/NVPTXAsmPrinter.cpp

void NVPTXAsmPrinter::printReturnValStr(const Function *F, raw_ostream &O) {
   bool isABI = (nvptxSubtarget->getSmVersion() >= 20);
 
   if (Ty->getTypeID() == Type::VoidTyID)
     return;
 
   O << " (";
 
   if (isABI) {
-    if (Ty->isFloatingPointTy() || Ty->isIntegerTy()) {
+    if (Ty->isFloatingPointTy() || (Ty->isIntegerTy() && !Ty->isIntegerTy(128))) {
       unsigned size = 0;
       if (auto *ITy = dyn_cast<IntegerType>(Ty)) {
         size = ITy->getBitWidth();
       } else {
         assert(Ty->isFloatingPointTy() && "Floating point type expected here");
         size = Ty->getPrimitiveSizeInBits();
       }
       // PTX ABI requires all scalar return values to be at least 32
       // bits in size.  fp16 normally uses .b16 as its storage type in
       // PTX, so its size must be adjusted here, too.
       if (size < 32)
         size = 32;
 
       O << ".param .b" << size << " func_retval0";
     } else if (isa<PointerType>(Ty)) {
       O << ".param .b" << TLI->getPointerTy(DL).getSizeInBits()
         << " func_retval0";
-    } else if (Ty->isAggregateType() || Ty->isVectorTy()) {
+    } else if (Ty->isAggregateType() || Ty->isVectorTy() || Ty->isIntegerTy(128)) {
       unsigned totalsz = DL.getTypeAllocSize(Ty);
       unsigned retAlignment = 0;
       if (!getAlign(*F, 0, retAlignment))
         retAlignment = DL.getABITypeAlignment(Ty);
       O << ".param .align " << retAlignment << " .b8 func_retval0[" << totalsz
         << "]";
     } else
       llvm_unreachable("Unknown return type");
void NVPTXAsmPrinter::emitPTXGlobalVariable(const GlobalVariable *GVar,
 
   O << ".";
   emitPTXAddressSpace(GVar->getType()->getAddressSpace(), O);
   if (GVar->getAlignment() == 0)
     O << " .align " << (int)DL.getPrefTypeAlignment(ETy);
   else
     O << " .align " << GVar->getAlignment();
 
+  // Special case for i128
+  if (ETy->isIntegerTy(128)) {
+    O << " .b8 ";
+    getSymbol(GVar)->print(O, MAI);
+    O << "[16]";
+    return;
+  }
+
   if (ETy->isFloatingPointTy() || ETy->isIntegerTy() || ETy->isPointerTy()) {
     O << " .";
     O << getPTXFundamentalTypeStr(ETy);
     O << " ";
     getSymbol(GVar)->print(O, MAI);
     return;
   }
 
if (isKernelFunction(*F)) {
           CurrentFnSym->print(O, MAI);
           O << "_param_" << paramIndex;
         }
         continue;
       }
     }
 
     if (!PAL.hasParamAttribute(paramIndex, Attribute::ByVal)) {
-      if (Ty->isAggregateType() || Ty->isVectorTy()) {
+      if (Ty->isAggregateType() || Ty->isVectorTy() || Ty->isIntegerTy(128)) {
         // Just print .param .align <a> .b8 .param[size];
         // <a> = PAL.getparamalignment
         // size = typeallocsize of element type
         unsigned align = PAL.getParamAlignment(paramIndex);
         if (align == 0)
           align = DL.getABITypeAlignment(Ty);
 
         unsigned sz = DL.getTypeAllocSize(Ty);

llvm/trunk/lib/Target/NVPTX/NVPTXISelLowering.cpp

 /// LowerCall, and LowerReturn.
 static void ComputePTXValueVTs(const TargetLowering &TLI, const DataLayout &DL,
                                Type *Ty, SmallVectorImpl<EVT> &ValueVTs,
                                SmallVectorImpl<uint64_t> *Offsets = nullptr,
                                uint64_t StartingOffset = 0) {
   SmallVector<EVT, 16> TempVTs;
   SmallVector<uint64_t, 16> TempOffsets;
 
+  // Special case for i128 - decompose to (i64, i64)
+  if (Ty->isIntegerTy(128)) {
+    ValueVTs.push_back(EVT(MVT::i64));
+    ValueVTs.push_back(EVT(MVT::i64));
+
+    if (Offsets) {
+      Offsets->push_back(StartingOffset + 0);
+      Offsets->push_back(StartingOffset + 8);
+    }
+
+    return;
+  }
+
   ComputeValueVTs(TLI, DL, Ty, TempVTs, &TempOffsets, StartingOffset);
   for (unsigned i = 0, e = TempVTs.size(); i != e; ++i) {
     EVT VT = TempVTs[i];
     uint64_t Off = TempOffsets[i];
     // Split vectors into individual elements, except for v2f16, which
     // we will pass as a single scalar.
     if (VT.isVector()) {
       unsigned NumElts = VT.getVectorNumElements();
std::string NVPTXTargetLowering::getPrototype(
 
   std::stringstream O;
   O << "prototype_" << uniqueCallSite << " : .callprototype ";
 
   if (retTy->getTypeID() == Type::VoidTyID) {
     O << "()";
   } else {
     O << "(";
-    if (retTy->isFloatingPointTy() || retTy->isIntegerTy()) {
+    if (retTy->isFloatingPointTy() || (retTy->isIntegerTy() && !retTy->isIntegerTy(128))) {
       unsigned size = 0;
       if (auto *ITy = dyn_cast<IntegerType>(retTy)) {
         size = ITy->getBitWidth();
       } else {
         assert(retTy->isFloatingPointTy() &&
                "Floating point type expected here");
         size = retTy->getPrimitiveSizeInBits();
       }
       // PTX ABI requires all scalar return values to be at least 32
       // bits in size.  fp16 normally uses .b16 as its storage type in
       // PTX, so its size must be adjusted here, too.
       if (size < 32)
         size = 32;
 
       O << ".param .b" << size << " _";
     } else if (isa<PointerType>(retTy)) {
       O << ".param .b" << PtrVT.getSizeInBits() << " _";
-    } else if (retTy->isAggregateType() || retTy->isVectorTy()) {
+    } else if (retTy->isAggregateType() || retTy->isVectorTy() || retTy->isIntegerTy(128)) {
       auto &DL = CS->getCalledFunction()->getParent()->getDataLayout();
       O << ".param .align " << retAlignment << " .b8 _["
         << DL.getTypeAllocSize(retTy) << "]";
     } else {
       llvm_unreachable("Unknown return type");
     }
     O << ") ";
   }
   O << "_ (";
 
   bool first = true;
 
   unsigned OIdx = 0;
   for (unsigned i = 0, e = Args.size(); i != e; ++i, ++OIdx) {
     Type *Ty = Args[i].Ty;
     if (!first) {
       O << ", ";
     }
     first = false;
 
     if (!Outs[OIdx].Flags.isByVal()) {
-      if (Ty->isAggregateType() || Ty->isVectorTy()) {
+      if (Ty->isAggregateType() || Ty->isVectorTy() || Ty->isIntegerTy(128)) {
         unsigned align = 0;
         const CallInst *CallI = cast<CallInst>(CS->getInstruction());
         // +1 because index 0 is reserved for return type alignment
         if (!getAlign(*CallI, i + 1, align))
           align = DL.getABITypeAlignment(Ty);
         unsigned sz = DL.getTypeAllocSize(Ty);
         O << ".param .align " << align << " .b8 ";
         O << "_";
if (!Outs[OIdx].Flags.isByVal()) {
       SmallVector<EVT, 16> VTs;
       SmallVector<uint64_t, 16> Offsets;
       ComputePTXValueVTs(*this, DL, Ty, VTs, &Offsets);
       unsigned ArgAlign =
           getArgumentAlignment(Callee, CS, Ty, paramCount + 1, DL);
       unsigned AllocSize = DL.getTypeAllocSize(Ty);
       SDVTList DeclareParamVTs = DAG.getVTList(MVT::Other, MVT::Glue);
       bool NeedAlign; // Does argument declaration specify alignment?
-      if (Ty->isAggregateType() || Ty->isVectorTy()) {
+      if (Ty->isAggregateType() || Ty->isVectorTy() || Ty->isIntegerTy(128)) {
         // declare .param .align <align> .b8 .param<n>[<size>];
         SDValue DeclareParamOps[] = {
             Chain, DAG.getConstant(ArgAlign, dl, MVT::i32),
             DAG.getConstant(paramCount, dl, MVT::i32),
             DAG.getConstant(AllocSize, dl, MVT::i32), InFlag};
         Chain = DAG.getNode(NVPTXISD::DeclareParam, dl, DeclareParamVTs,
                             DeclareParamOps);
         NeedAlign = true;
if (Ins.size() > 0) {
     // Declare
     //  .param .align 16 .b8 retval0[<size-in-bytes>], or
     //  .param .b<size-in-bits> retval0
     unsigned resultsz = DL.getTypeAllocSizeInBits(RetTy);
     // Emit ".param .b<size-in-bits> retval0" instead of byte arrays only for
     // these three types to match the logic in
     // NVPTXAsmPrinter::printReturnValStr and NVPTXTargetLowering::getPrototype.
     // Plus, this behavior is consistent with nvcc's.
-    if (RetTy->isFloatingPointTy() || RetTy->isIntegerTy() ||
-        RetTy->isPointerTy()) {
+    if (RetTy->isFloatingPointTy() || RetTy->isPointerTy() ||
+        (RetTy->isIntegerTy() && !RetTy->isIntegerTy(128))) {
       // Scalar needs to be at least 32bit wide
       if (resultsz < 32)
         resultsz = 32;
       SDVTList DeclareRetVTs = DAG.getVTList(MVT::Other, MVT::Glue);
       SDValue DeclareRetOps[] = { Chain, DAG.getConstant(1, dl, MVT::i32),
                                   DAG.getConstant(resultsz, dl, MVT::i32),
                                   DAG.getConstant(0, dl, MVT::i32), InFlag };
       Chain = DAG.getNode(NVPTXISD::DeclareRet, dl, DeclareRetVTs,
if (isImageOrSamplerVal(
       assert(isKernelFunction(*F) &&
              "Only kernels can have image/sampler params");
       InVals.push_back(DAG.getConstant(i + 1, dl, MVT::i32));
       continue;
     }
 
     if (theArgs[i]->use_empty()) {
       // argument is dead
-      if (Ty->isAggregateType()) {
+      if (Ty->isAggregateType() || Ty->isIntegerTy(128)) {
         SmallVector<EVT, 16> vtparts;
 
         ComputePTXValueVTs(*this, DAG.getDataLayout(), Ty, vtparts);
         assert(vtparts.size() > 0 && "empty aggregate type not expected");
         for (unsigned parti = 0, parte = vtparts.size(); parti != parte;
              ++parti) {
           InVals.push_back(DAG.getNode(ISD::UNDEF, dl, Ins[InsIdx].VT));
           ++InsIdx;

llvm/trunk/lib/Target/NVPTX/NVPTXTargetMachine.cpp

 }
 
 static std::string computeDataLayout(bool is64Bit) {
   std::string Ret = "e";
 
   if (!is64Bit)
     Ret += "-p:32:32";
 
-  Ret += "-i64:64-v16:16-v32:32-n16:32:64";
+  Ret += "-i64:64-i128:128-v16:16-v32:32-n16:32:64";
 
   return Ret;
 }
 
 NVPTXTargetMachine::NVPTXTargetMachine(const Target &T, const Triple &TT,
                                        StringRef CPU, StringRef FS,
                                        const TargetOptions &Options,
                                        Optional<Reloc::Model> RM,

llvm/trunk/test/CodeGen/NVPTX/i128-global.ll

+; RUN: llc < %s -O0 -march=nvptx64 -mcpu=sm_20 | FileCheck %s
+
+; CHECK: .visible .global .align 16 .b8 G1[16] = {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
+@G1 = global i128 1
+
+; CHECK: .visible .global .align 16 .b8 G2[16];
+@G2 = global i128 0

llvm/trunk/test/CodeGen/NVPTX/i128-param.ll

+; RUN: llc < %s -O0 -march=nvptx -mcpu=sm_20 | FileCheck %s
+
+; CHECK-LABEL: .visible .func callee(
+; CHECK-NEXT: .param .align 16 .b8 callee_param_0[16],
+; CHECK-NEXT: .param .align 16 .b8 callee_param_1[16],
+define void @callee(i128, i128, i128*) {
+  ; CHECK-DAG: ld.param.v2.u64 {%[[REG0:rd[0-9]+]], %[[REG1:rd[0-9]+]]}, [callee_param_0];
+  ; CHECK-DAG: ld.param.v2.u64 {%[[REG2:rd[0-9]+]], %[[REG3:rd[0-9]+]]}, [callee_param_1];
+
+  ; CHECK:      mul.lo.s64 %[[REG4:rd[0-9]+]], %[[REG0]], %[[REG3]];
+  ; CHECK-NEXT: mul.hi.u64 %[[REG5:rd[0-9]+]], %[[REG0]], %[[REG2]];
+  ; CHECK-NEXT: add.s64    %[[REG6:rd[0-9]+]], %[[REG5]], %[[REG4]];
+  ; CHECK-NEXT: mul.lo.s64 %[[REG7:rd[0-9]+]], %[[REG1]], %[[REG2]];
+  ; CHECK-NEXT: add.s64    %[[REG8:rd[0-9]+]], %[[REG6]], %[[REG7]];
+  ; CHECK-NEXT: mul.lo.s64 %[[REG9:rd[0-9]+]], %[[REG0]], %[[REG2]];
+  %a = mul i128 %0, %1
+
+  store i128 %a, i128* %2
+  ret void
+}
+
+; CHECK-LABEL: .visible .entry caller_kernel(
+; CHECK-NEXT: .param .align 16 .b8 caller_kernel_param_0[16],
+; CHECK-NEXT: .param .align 16 .b8 caller_kernel_param_1[16],
+define ptx_kernel void @caller_kernel(i128, i128, i128*) {
+start:
+  ; CHECK-DAG: ld.param.v2.u64 {%[[REG0:rd[0-9]+]], %[[REG1:rd[0-9]+]]}, [caller_kernel_param_0];
+  ; CHECK-DAG: ld.param.v2.u64 {%[[REG2:rd[0-9]+]], %[[REG3:rd[0-9]+]]}, [caller_kernel_param_1];
+
+  ; CHECK:      { // callseq [[CALLSEQ_ID:[0-9]]], 0
+  ; CHECK:      .param .align 16 .b8 param0[16];
+  ; CHECK-NEXT: st.param.v2.b64   [param0+0], {%[[REG0]], %[[REG1]]}
+  ; CHECK:      .param .align 16 .b8 param1[16];
+  ; CHECK-NEXT: st.param.v2.b64   [param1+0], {%[[REG2]], %[[REG3]]}
+  ; CHECK:      } // callseq [[CALLSEQ_ID]]
+  call void @callee(i128 %0, i128 %1, i128* %2)
+
+  ret void
+}
+
+; CHECK-LABEL: .visible .func caller_func(
+; CHECK-NEXT: .param .align 16 .b8 caller_func_param_0[16],
+; CHECK-NEXT: .param .align 16 .b8 caller_func_param_1[16],
+define void @caller_func(i128, i128, i128*) {
+start:
+  ; CHECK-DAG: ld.param.v2.u64 {%[[REG0:rd[0-9]+]], %[[REG1:rd[0-9]+]]}, [caller_func_param_0]
+  ; CHECK-DAG: ld.param.v2.u64 {%[[REG2:rd[0-9]+]], %[[REG3:rd[0-9]+]]}, [caller_func_param_1]
+
+  ; CHECK: { // callseq [[CALLSEQ_ID:[0-9]]], 0
+  ; CHECK: .param .align 16 .b8 param0[16];
+  ; CHECK: st.param.v2.b64  [param0+0], {%[[REG0]], %[[REG1]]}
+  ; CHECK: .param .align 16 .b8 param1[16];
+  ; CHECK: st.param.v2.b64  [param1+0], {%[[REG2]], %[[REG3]]}
+  ; CHECK: } // callseq [[CALLSEQ_ID]]
+  call void @callee(i128 %0, i128 %1, i128* %2)
+
+  ret void
+}

llvm/trunk/test/CodeGen/NVPTX/i128-retval.ll

+; RUN: llc < %s -O0 -march=nvptx64 -mcpu=sm_20 | FileCheck %s
+
+; CHECK-LABEL: .visible .func (.param .align 16 .b8 func_retval0[16]) callee(
+define i128 @callee(i128) {
+  ; CHECK: ld.param.v2.u64 {%[[REG0:rd[0-9]+]], %[[REG1:rd[0-9]+]]}, [callee_param_0];
+  ; CHECK: st.param.v2.b64 [func_retval0+0], {%[[REG0]], %[[REG1]]}
+  ret i128 %0
+}
+
+; CHECK-LABEL: .visible .func caller(
+define void @caller(i128, i128*) {
+start:
+  ; CHECK-DAG: ld.param.v2.u64 {%[[REG0:rd[0-9]+]], %[[REG1:rd[0-9]+]]}, [caller_param_0];
+  ; CHECK-DAG: ld.param.u64 %[[OUT:rd[0-9]+]],  [caller_param_1];
+
+  ; CHECK: { // callseq 0, 0
+  ; CHECK: .param .align 16 .b8 retval0[16];
+  ; CHECK: call.uni (retval0),
+  ; CHECK: ld.param.v2.b64 {%[[REG2:rd[0-9]+]], %[[REG3:rd[0-9]+]]}, [retval0+0];
+  ; CHECK: } // callseq 0
+  %a = call i128 @callee(i128 %0)
+
+  ; CHECK-DAG: st.u64 [%[[OUT]]], %[[REG2]];
+  ; CHECK-DAG: st.u64 [%[[OUT]]+8], %[[REG3]];
+  store i128 %a, i128* %1
+
+  ret void
+}