[SystemZ] Support i128 inline asm operands

[jonpa]

Summary

As it seems difficult to mix in only some cases of legal i128 (for instance adding i128 as a type of GR128 immediately causes things to not even build), I did some experiments on how to handle this without doing that. This was (as expected) not as simple as one might have hoped - here is what I did so far to make the test case compile:

• Use a new SystemZISD node "SUBREG128" which is used when splitting/joining i128 inline asm values. Implement splitValueIntoRegisterParts() and joinRegisterPartsIntoValue() in SystemZTargetLowering using this new node to keep track of a GR64 reg and a hi64/low64 subreg index. This unfortunately also gets called for other things than inline-assembly, but it seems to work at least during experiments to check for an existing CallingConvention, but I am not really sure that is correct...

• Use a new psuedo instruction (also called SUBREG128) that lowers the SUBREG128 ISD node to an MI that uses a custom inserter hook. Here the INLINEASM instruction is found and replaced by a new one with GR128 operands (INLINEASM is a generic opcode without custom insertion).

• Return GR64 for i128 in SystemZTargetLowering::getRegForInlineAsmConstraint(). Currently GR128 is returned, but that is not really working either as two GR128 vregs were created. The i128 case is identified by an inline asm operand of GR64 that uses two (GR64) registers. This seems to be the only case where a register constraint could produce this, I think.

• Build a new INLINEASM with altered Flag operands and GR128 (tied) operands. This is a little tricky given the tied operands, updating of Flag operands, and making sure that the SUBREG128 instructions are there as expected...

I tried leaving the INLINEASM as it was all the way until the AsmPrinter, but that didn't seem workable with the tied operands as they were then two pairs of tied GR64 registers that didn't belong in a GR128 pair during regalloc...

The approach is sort of simple but yet quite intricate to implement. There are probably things that could be simplified by some change in common-code...

[jonpa]

The idea per the initial patch to follow the general pattern of splitting the illegal i128 during isel and then do a fixup later was proven very cumbersome. I instead tried keeping GR128 as the regclass for the inline-asm operands which was actually simpler, even still a "ugly hack" since it had 2xGR128 for each i128 operands. Even better is to fix that problem so that SelectionDAGBuilder only builds one register operand to begin with, which is what the patch now does.

These are the needed common-code changes:

• New method: TargetLowering::getNumRegistersForInlineAsmOp(). Allow target to override default number of parts for inline-asm operands.
• splitValueIntoRegisterParts() / joinRegisterPartsIntoValue(): Add a new argument to pass the Value* of the CallInst in order to identify the case of an InlineAsm instruction.
• SelectionDAGBuilder.cpp:GetRegistersForValue(): Don't do the RegClass/type fixup for MVT::Untyped.
• SelectionDAGBuilder::visitInlineAsm(): Get the RegClass from the constraint code for MVT::Untyped.

[jonpa]

Patch updated (NFC).

• No need to pass the original Value to splitValueIntoRegisterParts()/joinRegisterPartsIntoValue() to detect the inline-asm case. However, this is still reached for normal calls where the i128 argument has been expanded to 2 x i64 parts, so need to check NumParts argument and do nothing in those cases (for example test/CodeGen/SystemZ/args-09.ll).

• I tried to instead of adding a new hook "getNumRegistersForInlineAsmOp()" (per previous version of patch), pass RC to getNumRegisters() as an optional argument. This can instead be used by target to do an override. I am not quite sure this is preferred: in contrast to calling a separate new hook that puts this in a clear context, this is made general and it requires a (tiny) bit more work in SelectionDAGBuilder of passing the RegisterClass along. Perhaps an alternative would be to return this from getRegForInlineAsmConstraint() and that way have the assigned register, the register class and numparts declared there in one place..?

• I tried to reuse the register class for a tied operand in visitInlineAsm(), but that turned out to fail in cases where the def is a specific physreg, in which case there is no regclass specified...

[uweigand]

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

8626

Instead of recomputing the RC here, can't we get it from the flag word of the matching operand, e.g. using hasRegClassConstraint(OpFlag, RCID)? This would match the way we get the number of registers from the flag word instead of recomputing it as well.

[jonpa]

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

8626

No - the matching operand may be a specific physreg in which case there is no register class specified (e.g. SystemZ/asm-17.ll).

[uweigand]

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

8626

Hmm, maybe I'm confused but I thought in those cases, we'd still get the RC for that physreg installed into the flags. Isnt't that exactly what the code in AddInlineAsmOperands does:

const TargetRegisterClass *RC = MRI.getRegClass(Regs.front());
Flag = InlineAsm::getFlagWordForRegClass(Flag, RC->getID());

(Regs.front() should have been set up to the physreg if there's one.)

[jonpa]

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

8626

That is guarded with a check to only do that for virtual registers...

[uweigand]

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

8626

Doh, of course, sorry for missing that.

Still, I think calling getRegForInlineAsmConstraint again here is redundant. The routine was already called in GetRegistersForValue, resulting in either a virtual register, in which case the RC is stored in the Flags, or else a physical register which is guaranteed to be a member of RC otherwise GetRegistersForValue will fail an assertion.

So we should check the register that was created for the matching index (which is in
AsmNodeOperands[CurOp+1] at this point); if it is a virtual register, get the RC from the flags, and if it is a physical register, use the RC this physreg is a member of. Does that make sense?

[jonpa]

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

8626

Ah, I was not quite aware of the fact that the phys-reg returned would also have a matching RC returned.

I also then think it would be best to call getRegForInlineAsmConstraint() once so I tried to pass an RC argument to it by reference so that the RC returned could be used in all the following places. The problem with this is however that targets fall back to the default TargetLowering::getRegForInlineAsmConstraint() which may return an unallocatable register class. I then tried demanding an allocatable register class in that method, but then physregs like CC become a problem since they don't have that. So to do this it seems then all targets must take care to return the right RC if there is not an allocatable one for a physreg.

What's more is that this wasn't actually working for an operand where the reg is a specified GR128 phys-reg. AddInlineAsmOperands will then call getNumRegisters() without an RC (and return 2 for i128). I tried returning just '1' here for a phys-reg, but that failed on other targets. Even so, InstrEmitter then fails to find the RC for the defined register, so SystemZTargetLowering::getRegClass() would have to return GR128 for MVT::Untyped...

I wonder:

• Do we want to also handle the explicit phys reg operands (at this point)?
• It is quite messy to first call getRegForInlineAsmConstraint() and then recompute the RC after that fact. Would it make sense to try to make TargetLowering::getRegForInlineAsmConstraint() return an allocatable RC and fix target cases of unallocatable physregs that show up in tests, and then just reuse that RC in throughout visitInlineAsm()? Alternatively, try first to return an allocatable RC and if there is none, return an unallocatable one.

If we are just handling virtual registers, maybe we could just get the RC from the flags per your suggestion...

[jonpa]

So we should check the register that was created for the matching index (which is inAsmNodeOperands[CurOp+1] at this point); if it is a virtual register, get the RC from the flags, and if it is a physical register, use the RC this physreg is a member of. Does that make sense?

Yes... I however found that we have to implement getRegClassFor(MVT::Untyped) to make inline assembly phys-regs work in InstrEmitter::EmitCopyFromReg(). If we do this we can use this also in SelectionDAGBuilder instead of the above. This seems to work as long as we only have one untyped regclass even though it is slightly confusing to not use the regclass returned by getRegForInlineAsmConstraint().

[uweigand]

In D100788#2766444, @jonpa wrote:

So we should check the register that was created for the matching index (which is inAsmNodeOperands[CurOp+1] at this point); if it is a virtual register, get the RC from the flags, and if it is a physical register, use the RC this physreg is a member of. Does that make sense?

Yes... I however found that we have to implement getRegClassFor(MVT::Untyped) to make inline assembly phys-regs work in InstrEmitter::EmitCopyFromReg(). If we do this we can use this also in SelectionDAGBuilder instead of the above. This seems to work as long as we only have one untyped regclass even though it is slightly confusing to not use the regclass returned by getRegForInlineAsmConstraint().

Can you elaborate? I thought InstrEmitter::EmitCopyFromReg() was supposed to also work with types that are not legal. It uses

SrcRC = TRI->getMinimalPhysRegClass(SrcReg, VT);

to get the RC for use with a physreg ...

I don't really like implementing getRegClassFor(MVT::Untyped) ... that seems a bit underspecified.

As I said, I still think the best option would be to remove the getRegClassFor call in the tied register case in visitInlineAsm, and instead retrieve the RC either from the OpFlags (if virtual reg) or via getMinimalPhysRegClass (if phys reg).

[jonpa]

Patch updated per review.

Can you elaborate? I thought InstrEmitter::EmitCopyFromReg() was supposed to also work with types that are not legal. It uses
SrcRC = TRI->getMinimalPhysRegClass(SrcReg, VT);
to get the RC for use with a physreg ...

It does that for SrcRC, but for DstRC it calls getRegClassFor(). Another way of solving this might be to check if the type is legal and if not then take SrcRC (see patch).

[uweigand]

In D100788#2769576, @jonpa wrote:

Patch updated per review.

Can you elaborate? I thought InstrEmitter::EmitCopyFromReg() was supposed to also work with types that are not legal. It uses
SrcRC = TRI->getMinimalPhysRegClass(SrcReg, VT);
to get the RC for use with a physreg ...

It does that for SrcRC, but for DstRC it calls getRegClassFor(). Another way of solving this might be to check if the type is legal and if not then take SrcRC (see patch).

Interesting. Note that for legal type, UseRC is always non-null here due to the lines above:

// Stick to the preferred register classes for legal types.
if (TLI->isTypeLegal(VT))
  UseRC = TLI->getRegClassFor(VT, Node->isDivergent());

so we really should have never arrived at the (second) getRegClassFor. Can you verify if that ever happens?

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

954

Moving Regs.front() out of the Regs.empty() check is probably not a good idea.

[jonpa]

...so we really should have never arrived at the (second) getRegClassFor. Can you verify if that ever happens?

Indeed, that seems to be the case - removed.

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

954

oops

[uweigand]

This already looks a lot simpler. I'm now wondering if it would be even better to change the getNumRegisters interface from (ValueVT, RC) to (ValueVT, RegisterVT). (Or I guess an Optional<RegisterVT> if we don't want to change all callers.) All callers of getNumRegisters already have the RegisterVT immediately available, and it allows the back end to make any distinction it needs just as well as a RC would.

[jonpa]

This already looks a lot simpler. I'm now wondering if it would be even better to change the getNumRegisters interface from (ValueVT, RC) to (ValueVT, RegisterVT). (Or I guess an Optional<RegisterVT> if we don't want to change all callers.) All callers of getNumRegisters already have the RegisterVT immediately available, and it allows the back end to make any distinction it needs just as well as a RC would.

Yes, even better! :-)

[uweigand]

Now it's looking mostly good to me; a couple of (mainly cosmetic) remaining comments inline.

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

8624–8625

Maybe avoid using AsmNodeOperands[CurOp+1] twice by taking the type of the TiedReg below?

8625–8626

The wording of the error message now no longer makes sense. In fact, I think with the new code it no longer can even happen for RC to be null, so I guess the whole error can be removed.

llvm/lib/Target/SystemZ/SystemZISelLowering.h

428

A brief comment would be helpful here.

[jonpa]

Updated per review.

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

8625–8626

Yes, since RC must be provided I guess this can't happen...

The RC is asserted in createVirtualRegister(), so no need to assert here.

[uweigand]

This version LGTM now. Thanks!

llvm/include/llvm/CodeGen/TargetLowering.h

#include "llvm/IR/ConstrainedOps.def"
   /// Return the number of registers that this ValueType will eventually
   /// require.
   ///
   /// This is one for any types promoted to live in larger registers, but may be
   /// more than one for types (like i64) that are split into pieces.  For types
   /// like i140, which are first promoted then expanded, it is the number of
   /// registers needed to hold all the bits of the original type.  For an i140
   /// on a 32 bit machine this means 5 registers.
-  unsigned getNumRegisters(LLVMContext &Context, EVT VT) const {
+  ///
+  /// RegisterVT may be passed as a way to override the default settings, for
+  /// instance with i128 inline assembly operands on SystemZ.
+  virtual unsigned
+  getNumRegisters(LLVMContext &Context, EVT VT,
+                  Optional<MVT> RegisterVT = None) const {
     if (VT.isSimple()) {
       assert((unsigned)VT.getSimpleVT().SimpleTy <
                 array_lengthof(NumRegistersForVT));
       return NumRegistersForVT[VT.getSimpleVT().SimpleTy];
     }
     if (VT.isVector()) {
       EVT VT1;
       MVT VT2;

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

EmitCopyFromReg(SDNode *Node, unsigned ResNo, bool IsClone, bool IsCloned,
 
   // Figure out the register class to create for the destreg.
   if (VRBase) {
     DstRC = MRI->getRegClass(VRBase);
   } else if (UseRC) {
     assert(TRI->isTypeLegalForClass(*UseRC, VT) &&
            "Incompatible phys register def and uses!");
     DstRC = UseRC;
-  } else {
-    DstRC = TLI->getRegClassFor(VT, Node->isDivergent());
-  }
+  } else
+    DstRC = SrcRC;
 
   // If all uses are reading from the src physical register and copying the
   // register is either impossible or very expensive, then don't create a copy.
   if (MatchReg && SrcRC->getCopyCost() < 0) {
     VRBase = SrcReg;
   } else {
     // Create the reg, emit the copy.
     VRBase = MRI->createVirtualRegister(DstRC);

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

else
     Chain = DAG.getNode(ISD::TokenFactor, dl, MVT::Other, Chains);
 }
 
 void RegsForValue::AddInlineAsmOperands(unsigned Code, bool HasMatching,
                                         unsigned MatchingIdx, const SDLoc &dl,
                                         SelectionDAG &DAG,
                                         std::vector<SDValue> &Ops) const {
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
 
   unsigned Flag = InlineAsm::getFlagWord(Code, Regs.size());
   if (HasMatching)
     Flag = InlineAsm::getFlagWordForMatchingOp(Flag, MatchingIdx);
   else if (!Regs.empty() && Register::isVirtualRegister(Regs.front())) {
     // Put the register class of the virtual registers in the flag word.  That
     // way, later passes can recompute register class constraints for inline
     // assembly as well as normal instructions.
     // Don't do this for tied operands that can use the regclass information
for (unsigned I = 0, E = ValueVTs.size(); I != E; ++I) {
           (Regs[I] != SP ||
            DAG.getMachineFunction().getFrameInfo().hasOpaqueSPAdjustment()) &&
           "If we clobbered the stack pointer, MFI should know about it.");
     }
     return;
   }
 
   for (unsigned Value = 0, Reg = 0, e = ValueVTs.size(); Value != e; ++Value) {
-    unsigned NumRegs = TLI.getNumRegisters(*DAG.getContext(), ValueVTs[Value]);
     MVT RegisterVT = RegVTs[Value];
+    unsigned NumRegs = TLI.getNumRegisters(*DAG.getContext(), ValueVTs[Value],
+                                           RegisterVT);
     for (unsigned i = 0; i != NumRegs; ++i) {
       assert(Reg < Regs.size() && "Mismatch in # registers expected");
       unsigned TheReg = Regs[Reg++];
       Ops.push_back(DAG.getRegister(TheReg, RegisterVT));
     }
   }
 }
 
static void GetRegistersForValue(SelectionDAG &DAG, const SDLoc &DL,
   if (!RC)
     return;
 
   // Get the actual register value type.  This is important, because the user
   // may have asked for (e.g.) the AX register in i32 type.  We need to
   // remember that AX is actually i16 to get the right extension.
   const MVT RegVT = *TRI.legalclasstypes_begin(*RC);
 
-  if (OpInfo.ConstraintVT != MVT::Other) {
+  if (OpInfo.ConstraintVT != MVT::Other && RegVT != MVT::Untyped) {
     // If this is an FP operand in an integer register (or visa versa), or more
     // generally if the operand value disagrees with the register class we plan
     // to stick it in, fix the operand type.
     //
     // If this is an input value, the bitcast to the new type is done now.
     // Bitcast for output value is done at the end of visitInlineAsm().
     if ((OpInfo.Type == InlineAsm::isOutput ||
          OpInfo.Type == InlineAsm::isInput) &&
static void GetRegistersForValue(SelectionDAG &DAG, const SDLoc &DL,
 
   EVT ValueVT = OpInfo.ConstraintVT;
   if (OpInfo.ConstraintVT == MVT::Other)
     ValueVT = RegVT;
 
   // Initialize NumRegs.
   unsigned NumRegs = 1;
   if (OpInfo.ConstraintVT != MVT::Other)
-    NumRegs = TLI.getNumRegisters(Context, OpInfo.ConstraintVT);
+    NumRegs = TLI.getNumRegisters(Context, OpInfo.ConstraintVT, RegVT);
 
   // If this is a constraint for a specific physical register, like {r17},
   // assign it now.
 
   // If this associated to a specific register, initialize iterator to correct
   // place. If virtual, make sure we have enough registers
 
   // Initialize iterator if necessary
case InlineAsm::isInput: {
           // Add (OpFlag&0xffff)>>3 registers to MatchedRegs.
           if (OpInfo.isIndirect) {
             // This happens on gcc/testsuite/gcc.dg/pr8788-1.c
             emitInlineAsmError(Call, "inline asm not supported yet: "
                                      "don't know how to handle tied "
                                      "indirect register inputs");
             return;
           }
 
-          MVT RegVT = AsmNodeOperands[CurOp+1].getSimpleValueType();
           SmallVector<unsigned, 4> Regs;
-
-          if (const TargetRegisterClass *RC = TLI.getRegClassFor(RegVT)) {
-            unsigned NumRegs = InlineAsm::getNumOperandRegisters(OpFlag);
-            MachineRegisterInfo &RegInfo =
-                DAG.getMachineFunction().getRegInfo();
-            for (unsigned i = 0; i != NumRegs; ++i)
-              Regs.push_back(RegInfo.createVirtualRegister(RC));
-          } else {
-            emitInlineAsmError(Call,
-                               "inline asm error: This value type register "
-                               "class is not natively supported!");
-            return;
-          }
+          MachineFunction &MF = DAG.getMachineFunction();
+          MachineRegisterInfo &MRI = MF.getRegInfo();
+          const TargetRegisterInfo &TRI = *MF.getSubtarget().getRegisterInfo();
+          RegisterSDNode *R = dyn_cast<RegisterSDNode>(AsmNodeOperands[CurOp+1]);
+          Register TiedReg = R->getReg();
+          MVT RegVT = R->getSimpleValueType(0);
+          const TargetRegisterClass *RC = TiedReg.isVirtual() ?
+            MRI.getRegClass(TiedReg) : TRI.getMinimalPhysRegClass(TiedReg);
+            Regs.push_back(MRI.createVirtualRegister(RC));
 
           RegsForValue MatchedRegs(Regs, RegVT, InOperandVal.getValueType());
 
           SDLoc dl = getCurSDLoc();
           // Use the produced MatchedRegs object to
           MatchedRegs.getCopyToRegs(InOperandVal, DAG, dl, Chain, &Flag, &Call);
           MatchedRegs.AddInlineAsmOperands(InlineAsm::Kind_RegUse,
                                            true, OpInfo.getMatchedOperand(), dl,

llvm/lib/Target/SystemZ/SystemZISelLowering.h

TargetLoweringBase::LegalizeTypeAction getPreferredVectorAction(MVT VT)
     //     so promoting the integers is less efficient.
     //
     // (c) there are no multiplication instructions for the widest integer
     //     type (v2i64).
     if (VT.getScalarSizeInBits() % 8 == 0)
       return TypeWidenVector;
     return TargetLoweringBase::getPreferredVectorAction(VT);
   }
+  unsigned
+  getNumRegisters(LLVMContext &Context, EVT VT,
+                  Optional<MVT> RegisterVT) const override {
+    // i128 inline assembly operand.
+    if (VT == MVT::i128 &&
+        RegisterVT.hasValue() && RegisterVT.getValue() == MVT::Untyped)
+      return 1;
+    return TargetLowering::getNumRegisters(Context, VT);
+  }
   bool isCheapToSpeculateCtlz() const override { return true; }
   bool preferZeroCompareBranch() const override { return true; }
   EVT getSetCCResultType(const DataLayout &DL, LLVMContext &,
                          EVT) const override;
   bool isFMAFasterThanFMulAndFAdd(const MachineFunction &MF,
                                   EVT VT) const override;
   bool isFPImmLegal(const APFloat &Imm, EVT VT,
                     bool ForCodeSize) const override;
public:
   SDValue LowerOperation(SDValue Op, SelectionDAG &DAG) const override;
   void LowerOperationWrapper(SDNode *N, SmallVectorImpl<SDValue> &Results,
                              SelectionDAG &DAG) const override;
   void ReplaceNodeResults(SDNode *N, SmallVectorImpl<SDValue>&Results,
                           SelectionDAG &DAG) const override;
   const MCPhysReg *getScratchRegisters(CallingConv::ID CC) const override;
   bool allowTruncateForTailCall(Type *, Type *) const override;
   bool mayBeEmittedAsTailCall(const CallInst *CI) const override;
+  bool splitValueIntoRegisterParts(SelectionDAG &DAG, const SDLoc &DL,
+                                   SDValue Val, SDValue *Parts,
+                                   unsigned NumParts, MVT PartVT,
+                                   Optional<CallingConv::ID> CC) const override;
+  SDValue
+  joinRegisterPartsIntoValue(SelectionDAG &DAG, const SDLoc &DL,
+                             const SDValue *Parts, unsigned NumParts,
+                             MVT PartVT, EVT ValueVT,
+                             Optional<CallingConv::ID> CC) const override;
   SDValue LowerFormalArguments(SDValue Chain, CallingConv::ID CallConv,
                                bool isVarArg,
                                const SmallVectorImpl<ISD::InputArg> &Ins,
                                const SDLoc &DL, SelectionDAG &DAG,
                                SmallVectorImpl<SDValue> &InVals) const override;
   SDValue LowerCall(CallLoweringInfo &CLI,
                     SmallVectorImpl<SDValue> &InVals) const override;
 

llvm/lib/Target/SystemZ/SystemZISelLowering.cpp

return DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, VA.getLocVT(), Value,
                        DAG.getConstant(0, DL, MVT::i32));
   case CCValAssign::Full:
     return Value;
   default:
     llvm_unreachable("Unhandled getLocInfo()");
   }
 }
 
+static SDValue lowerI128ToGR128(SelectionDAG &DAG, SDValue In) {
+  SDLoc DL(In);
+  SDValue Lo = DAG.getNode(ISD::EXTRACT_ELEMENT, DL, MVT::i64, In,
+                           DAG.getIntPtrConstant(0, DL));
+  SDValue Hi = DAG.getNode(ISD::EXTRACT_ELEMENT, DL, MVT::i64, In,
+                           DAG.getIntPtrConstant(1, DL));
+  SDNode *Pair = DAG.getMachineNode(SystemZ::PAIR128, DL,
+                                    MVT::Untyped, Hi, Lo);
+  return SDValue(Pair, 0);
+}
+
+static SDValue lowerGR128ToI128(SelectionDAG &DAG, SDValue In) {
+  SDLoc DL(In);
+  SDValue Hi = DAG.getTargetExtractSubreg(SystemZ::subreg_h64,
+                                          DL, MVT::i64, In);
+  SDValue Lo = DAG.getTargetExtractSubreg(SystemZ::subreg_l64,
+                                          DL, MVT::i64, In);
+  return DAG.getNode(ISD::BUILD_PAIR, DL, MVT::i128, Lo, Hi);
+}
+
+bool SystemZTargetLowering::splitValueIntoRegisterParts(
+    SelectionDAG &DAG, const SDLoc &DL, SDValue Val, SDValue *Parts,
+    unsigned NumParts, MVT PartVT, Optional<CallingConv::ID> CC) const {
+  EVT ValueVT = Val.getValueType();
+  assert((ValueVT != MVT::i128 ||
+          ((NumParts == 1 && PartVT == MVT::Untyped) ||
+           (NumParts == 2 && PartVT == MVT::i64))) &&
+         "Unknown handling of i128 value.");
+  if (ValueVT == MVT::i128 && NumParts == 1) {
+    // Inline assembly operand.
+    Parts[0] = lowerI128ToGR128(DAG, Val);
+    return true;
+  }
+  return false;
+}
+
+SDValue SystemZTargetLowering::joinRegisterPartsIntoValue(
+    SelectionDAG &DAG, const SDLoc &DL, const SDValue *Parts, unsigned NumParts,
+    MVT PartVT, EVT ValueVT, Optional<CallingConv::ID> CC) const {
+  assert((ValueVT != MVT::i128 ||
+          ((NumParts == 1 && PartVT == MVT::Untyped) ||
+           (NumParts == 2 && PartVT == MVT::i64))) &&
+         "Unknown handling of i128 value.");
+  if (ValueVT == MVT::i128 && NumParts == 1)
+    // Inline assembly operand.
+    return lowerGR128ToI128(DAG, Parts[0]);
+  return SDValue();
+}
+
 SDValue SystemZTargetLowering::LowerFormalArguments(
     SDValue Chain, CallingConv::ID CallConv, bool IsVarArg,
     const SmallVectorImpl<ISD::InputArg> &Ins, const SDLoc &DL,
     SelectionDAG &DAG, SmallVectorImpl<SDValue> &InVals) const {
   MachineFunction &MF = DAG.getMachineFunction();
   MachineFrameInfo &MFI = MF.getFrameInfo();
   MachineRegisterInfo &MRI = MF.getRegInfo();
   SystemZMachineFunctionInfo *FuncInfo =
case ISD::SRA:
     return lowerShift(Op, DAG, SystemZISD::VSRA_BY_SCALAR);
   default:
     llvm_unreachable("Unexpected node to lower");
   }
 }
 
 // Lower operations with invalid operand or result types (currently used
 // only for 128-bit integer types).
-
-static SDValue lowerI128ToGR128(SelectionDAG &DAG, SDValue In) {
-  SDLoc DL(In);
-  SDValue Lo = DAG.getNode(ISD::EXTRACT_ELEMENT, DL, MVT::i64, In,
-                           DAG.getIntPtrConstant(0, DL));
-  SDValue Hi = DAG.getNode(ISD::EXTRACT_ELEMENT, DL, MVT::i64, In,
-                           DAG.getIntPtrConstant(1, DL));
-  SDNode *Pair = DAG.getMachineNode(SystemZ::PAIR128, DL,
-                                    MVT::Untyped, Hi, Lo);
-  return SDValue(Pair, 0);
-}
-
-static SDValue lowerGR128ToI128(SelectionDAG &DAG, SDValue In) {
-  SDLoc DL(In);
-  SDValue Hi = DAG.getTargetExtractSubreg(SystemZ::subreg_h64,
-                                          DL, MVT::i64, In);
-  SDValue Lo = DAG.getTargetExtractSubreg(SystemZ::subreg_l64,
-                                          DL, MVT::i64, In);
-  return DAG.getNode(ISD::BUILD_PAIR, DL, MVT::i128, Lo, Hi);
-}
-
 void
 SystemZTargetLowering::LowerOperationWrapper(SDNode *N,
                                              SmallVectorImpl<SDValue> &Results,
                                              SelectionDAG &DAG) const {
   switch (N->getOpcode()) {
   case ISD::ATOMIC_LOAD: {
     SDLoc DL(N);
     SDVTList Tys = DAG.getVTList(MVT::Untyped, MVT::Other);

llvm/test/CodeGen/SystemZ/inline-asm-i128.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=s390x-linux-gnu -no-integrated-as < %s | FileCheck %s
+;
+; Test i128 (tied) operands.
+
+define i32 @fun0(i8* %p1, i32 signext %l1, i8* %p2, i32 signext %l2, i8 zeroext %pad) {
+; CHECK-LABEL: fun0:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    lgr %r0, %r5
+; CHECK-NEXT:    # kill: def $r4d killed $r4d def $r4q
+; CHECK-NEXT:    lgr %r1, %r3
+; CHECK-NEXT:    # kill: def $r2d killed $r2d def $r2q
+; CHECK-NEXT:    sllg %r5, %r6, 24
+; CHECK-NEXT:    rosbg %r5, %r0, 40, 63, 0
+; CHECK-NEXT:    risbg %r3, %r1, 40, 191, 0
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    clcl %r2, %r4
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    ogr %r3, %r5
+; CHECK-NEXT:    risbg %r0, %r3, 40, 191, 0
+; CHECK-NEXT:    ipm %r2
+; CHECK-NEXT:    afi %r2, -268435456
+; CHECK-NEXT:    srl %r2, 31
+; CHECK-NEXT:    br %r14
+entry:
+  %0 = ptrtoint i8* %p1 to i64
+  %1 = ptrtoint i8* %p2 to i64
+  %and5 = and i32 %l2, 16777215
+  %2 = zext i32 %and5 to i64
+  %conv7 = zext i8 %pad to i64
+  %shl = shl nuw nsw i64 %conv7, 24
+  %or = or i64 %shl, %2
+  %u1.sroa.0.0.insert.ext = zext i64 %0 to i128
+  %u1.sroa.0.0.insert.shift = shl nuw i128 %u1.sroa.0.0.insert.ext, 64
+  %3 = and i32 %l1, 16777215
+  %u1.sroa.0.0.insert.mask = zext i32 %3 to i128
+  %u1.sroa.0.0.insert.insert = or i128 %u1.sroa.0.0.insert.shift, %u1.sroa.0.0.insert.mask
+  %u2.sroa.5.0.insert.ext = zext i64 %or to i128
+  %u2.sroa.0.0.insert.ext = zext i64 %1 to i128
+  %u2.sroa.0.0.insert.shift = shl nuw i128 %u2.sroa.0.0.insert.ext, 64
+  %u2.sroa.0.0.insert.insert = or i128 %u2.sroa.0.0.insert.shift, %u2.sroa.5.0.insert.ext
+  %4 = tail call { i128, i128 } asm "clcl $0, $1", "=r,=r,0,1"(i128 %u1.sroa.0.0.insert.insert, i128 %u2.sroa.0.0.insert.insert)
+  %asmresult = extractvalue { i128, i128 } %4, 0
+  %asmresult11 = extractvalue { i128, i128 } %4, 1
+  %5 = or i128 %asmresult, %asmresult11
+  %6 = and i128 %5, 16777215
+  %7 = icmp eq i128 %6, 0
+  %land.ext = zext i1 %7 to i32
+  ret i32 %land.ext
+}
+
+; Test a phys-reg def.
+define void @fun1(i128* %Src, i128* %Dst) {
+; CHECK-LABEL: fun1:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    BLA %r4
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    stg %r5, 8(%r3)
+; CHECK-NEXT:    stg %r4, 0(%r3)
+; CHECK-NEXT:    br %r14
+entry:
+   %IAsm = call i128 asm "BLA $0", "={r4}"()
+  store volatile i128 %IAsm, i128* %Dst
+  ret void
+}
+
+; Test a phys-reg use.
+define void @fun2(i128* %Src, i128* %Dst) {
+; CHECK-LABEL: fun2:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    lg %r5, 8(%r2)
+; CHECK-NEXT:    lg %r4, 0(%r2)
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    BLA %r4
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    br %r14
+entry:
+  %L = load i128, i128* %Src
+  call void asm "BLA $0", "{r4}"(i128 %L)
+  ret void
+}
+
+; Test phys-reg use and phys-reg def.
+define void @fun3(i128* %Src, i128* %Dst) {
+; CHECK-LABEL: fun3:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    lg %r1, 8(%r2)
+; CHECK-NEXT:    lg %r0, 0(%r2)
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    BLA %r4, %r0
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    stg %r5, 8(%r3)
+; CHECK-NEXT:    stg %r4, 0(%r3)
+; CHECK-NEXT:    br %r14
+entry:
+  %L = load i128, i128* %Src
+  %IAsm = call i128 asm "BLA $0, $1", "={r4},{r0}"(i128 %L)
+  store volatile i128 %IAsm, i128* %Dst
+  ret void
+}
+
+; Test a tied phys-reg.
+define void @fun4(i128* %Src, i128* %Dst) {
+; CHECK-LABEL: fun4:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    lg %r5, 8(%r2)
+; CHECK-NEXT:    lg %r4, 0(%r2)
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    BLA %r4, %r4
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    stg %r5, 8(%r3)
+; CHECK-NEXT:    stg %r4, 0(%r3)
+; CHECK-NEXT:    br %r14
+entry:
+  %L = load i128, i128* %Src
+  %IAsm = call i128 asm "BLA $0, $1", "={r4},0"(i128 %L)
+  store volatile i128 %IAsm, i128* %Dst
+  ret void
+}