Experimental: inline assembly operands

[jonpa]

Summary

This was an attempt to let getRegForInlineAsmConstraint() return the regclass for an inline aseembly operand and then remember it and not recompute it.

The register class with this patch is encoded in all register operand Flags (also for physical regs), which causes some problems in target code that relies on the assumption that phys-regs do not have the reg-class encoded. All of those cases except one (X86FloatingPoint.cpp) seemed trivial to handle. Not sure if the X86 could be fixed without other changes (see patch comment in that file).

The motivation for this would be to simplify the handling of regclasses that are currently looked up again at a later point: The regclass is first carefully found in the default implementation of getRegForInlineAsmConstraint() but then later found again with TLI.getRegClassFor(MVT). This is confusing: which regclass computation is really correct? There could be a computation based on the type, or on the actual physreg. There are different regclasses possible for a phys-reg... Why not use the regclass that the target hook already returns...

Basically, it seemed right to let target override defaults for inline assembly operands in one place. This could also include NumParts. Given the problems with the targets, I am not sure if it is worthwhile... For the i128 operands on SystemZ, https://reviews.llvm.org/D100788 is a smaller and simpler patch even though it does not include the refactoring of this one.

[pengfei]

llvm/lib/Target/X86/X86FloatingPoint.cpp

1549–1551

Is this an existing bug for this pass, or affected by this patch? In which way, if it is affected?

1555

So this can be removed?

[jonpa]

llvm/lib/Target/X86/X86FloatingPoint.cpp

1549–1551

No, it's not a current bug. This (and some other files) relies on the fact that currently the RegClass is only encoded for virtual registers... This patch is experimental in encoding the returned RegClass (from getRegForInlineAsmConstraint) always (also for physregs), and that's why some problems show up.

Thanks for taking a look... Have anybody else thought about this or would desire this?

llvm/include/llvm/CodeGen/TargetLowering.h

#include "llvm/IR/ConstrainedOps.def"
   /// Return the number of registers that this ValueType will eventually
   /// require.
   ///
   /// This is one for any types promoted to live in larger registers, but may be
   /// more than one for types (like i64) that are split into pieces.  For types
   /// like i140, which are first promoted then expanded, it is the number of
   /// registers needed to hold all the bits of the original type.  For an i140
   /// on a 32 bit machine this means 5 registers.
-  unsigned getNumRegisters(LLVMContext &Context, EVT VT) const {
+  ///
+  /// RC is typically not needed, but it may be passed as a way to override
+  /// the default settings, for instance with i128 inline assembly operands
+  /// on SystemZ.
+  virtual unsigned
+  getNumRegisters(LLVMContext &Context, EVT VT,
+                  const TargetRegisterClass *RC = nullptr) const {
     if (VT.isSimple()) {
       assert((unsigned)VT.getSimpleVT().SimpleTy <
                 array_lengthof(NumRegistersForVT));
       return NumRegistersForVT[VT.getSimpleVT().SimpleTy];
     }
     if (VT.isVector()) {
       EVT VT1;
       MVT VT2;

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h

void getCopyToRegs(SDValue Val, SelectionDAG &DAG, const SDLoc &dl,
                      SDValue &Chain, SDValue *Flag, const Value *V = nullptr,
                      ISD::NodeType PreferredExtendType = ISD::ANY_EXTEND) const;
 
   /// Add this value to the specified inlineasm node operand list. This adds the
   /// code marker, matching input operand index (if applicable), and includes
   /// the number of values added into it.
   void AddInlineAsmOperands(unsigned Code, bool HasMatching,
                             unsigned MatchingIdx, const SDLoc &dl,
-                            SelectionDAG &DAG, std::vector<SDValue> &Ops) const;
+                            SelectionDAG &DAG, std::vector<SDValue> &Ops,
+                            const TargetRegisterClass *RC) const;
 
   /// Check if the total RegCount is greater than one.
   bool occupiesMultipleRegs() const {
     return std::accumulate(RegCount.begin(), RegCount.end(), 0) > 1;
   }
 
   /// Return a list of registers and their sizes.
   SmallVector<std::pair<unsigned, TypeSize>, 4> getRegsAndSizes() const;
 };
 
 } // end namespace llvm
 
 #endif // LLVM_LIB_CODEGEN_SELECTIONDAG_SELECTIONDAGBUILDER_H

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

if (NumRegs == 1 || Flag)
     Chain = Chains[NumRegs-1];
   else
     Chain = DAG.getNode(ISD::TokenFactor, dl, MVT::Other, Chains);
 }
 
 void RegsForValue::AddInlineAsmOperands(unsigned Code, bool HasMatching,
                                         unsigned MatchingIdx, const SDLoc &dl,
                                         SelectionDAG &DAG,
-                                        std::vector<SDValue> &Ops) const {
+                                        std::vector<SDValue> &Ops,
+                                        const TargetRegisterClass *RC) const {
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
 
   unsigned Flag = InlineAsm::getFlagWord(Code, Regs.size());
   if (HasMatching)
     Flag = InlineAsm::getFlagWordForMatchingOp(Flag, MatchingIdx);
-  else if (!Regs.empty() && Register::isVirtualRegister(Regs.front())) {
+  else if (!Regs.empty())
     // Put the register class of the virtual registers in the flag word.  That
     // way, later passes can recompute register class constraints for inline
     // assembly as well as normal instructions.
     // Don't do this for tied operands that can use the regclass information
     // from the def.
-    const MachineRegisterInfo &MRI = DAG.getMachineFunction().getRegInfo();
-    const TargetRegisterClass *RC = MRI.getRegClass(Regs.front());
     Flag = InlineAsm::getFlagWordForRegClass(Flag, RC->getID());
-  }
 
   SDValue Res = DAG.getTargetConstant(Flag, dl, MVT::i32);
   Ops.push_back(Res);
 
   if (Code == InlineAsm::Kind_Clobber) {
     // Clobbers should always have a 1:1 mapping with registers, and may
     // reference registers that have illegal (e.g. vector) types. Hence, we
     // shouldn't try to apply any sort of splitting logic to them.
     assert(Regs.size() == RegVTs.size() && Regs.size() == ValueVTs.size() &&
            "No 1:1 mapping from clobbers to regs?");
     Register SP = TLI.getStackPointerRegisterToSaveRestore();
     (void)SP;
     for (unsigned I = 0, E = ValueVTs.size(); I != E; ++I) {
       Ops.push_back(DAG.getRegister(Regs[I], RegVTs[I]));
       assert(
           (Regs[I] != SP ||
            DAG.getMachineFunction().getFrameInfo().hasOpaqueSPAdjustment()) &&
           "If we clobbered the stack pointer, MFI should know about it.");
     }
     return;
   }
 
   for (unsigned Value = 0, Reg = 0, e = ValueVTs.size(); Value != e; ++Value) {
-    unsigned NumRegs = TLI.getNumRegisters(*DAG.getContext(), ValueVTs[Value]);
+    // Pass RC since inline assembly operands may not follow the ordinary
+    // pattern of splitting.
+    unsigned NumRegs = TLI.getNumRegisters(*DAG.getContext(), ValueVTs[Value],
+                                           RC);
     MVT RegisterVT = RegVTs[Value];
     for (unsigned i = 0; i != NumRegs; ++i) {
       assert(Reg < Regs.size() && "Mismatch in # registers expected");
       unsigned TheReg = Regs[Reg++];
       Ops.push_back(DAG.getRegister(TheReg, RegisterVT));
     }
   }
 }
 
 SmallVector<std::pair<unsigned, TypeSize>, 4>
 RegsForValue::getRegsAndSizes() const {
   SmallVector<std::pair<unsigned, TypeSize>, 4> OutVec;
   unsigned I = 0;
   for (auto CountAndVT : zip_first(RegCount, RegVTs)) {
     unsigned RegCount = std::get<0>(CountAndVT);
     MVT RegisterVT = std::get<1>(CountAndVT);

 /// register allocator to handle the assignment process.  However, if the asm
 /// uses features that we can't model on machineinstrs, we have SDISel do the
 /// allocation.  This produces generally horrible, but correct, code.
 ///
 ///   OpInfo describes the operand
 ///   RefOpInfo describes the matching operand if any, the operand otherwise
 static void GetRegistersForValue(SelectionDAG &DAG, const SDLoc &DL,
                                  SDISelAsmOperandInfo &OpInfo,
-                                 SDISelAsmOperandInfo &RefOpInfo) {
+                                 SDISelAsmOperandInfo &RefOpInfo,
+                                 const TargetRegisterClass *&RC) {
   LLVMContext &Context = *DAG.getContext();
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
 
   MachineFunction &MF = DAG.getMachineFunction();
   SmallVector<unsigned, 4> Regs;
   const TargetRegisterInfo &TRI = *MF.getSubtarget().getRegisterInfo();
 
   // No work to do for memory operations.
   if (OpInfo.ConstraintType == TargetLowering::C_Memory)
     return;
 
   // If this is a constraint for a single physreg, or a constraint for a
   // register class, find it.
   unsigned AssignedReg;
-  const TargetRegisterClass *RC;
   std::tie(AssignedReg, RC) = TLI.getRegForInlineAsmConstraint(
       &TRI, RefOpInfo.ConstraintCode, RefOpInfo.ConstraintVT);
   // RC is unset only on failure. Return immediately.
   if (!RC)
     return;
 
   // Get the actual register value type.  This is important, because the user
   // may have asked for (e.g.) the AX register in i32 type.  We need to
   // remember that AX is actually i16 to get the right extension.
   const MVT RegVT = *TRI.legalclasstypes_begin(*RC);
 
-  if (OpInfo.ConstraintVT != MVT::Other) {
+  if (OpInfo.ConstraintVT != MVT::Other && RegVT != MVT::Untyped) {
     // If this is an FP operand in an integer register (or visa versa), or more
     // generally if the operand value disagrees with the register class we plan
     // to stick it in, fix the operand type.
     //
     // If this is an input value, the bitcast to the new type is done now.
     // Bitcast for output value is done at the end of visitInlineAsm().
     if ((OpInfo.Type == InlineAsm::isOutput ||
          OpInfo.Type == InlineAsm::isInput) &&
static void GetRegistersForValue(SelectionDAG &DAG, const SDLoc &DL,
 
   EVT ValueVT = OpInfo.ConstraintVT;
   if (OpInfo.ConstraintVT == MVT::Other)
     ValueVT = RegVT;
 
   // Initialize NumRegs.
   unsigned NumRegs = 1;
   if (OpInfo.ConstraintVT != MVT::Other)
-    NumRegs = TLI.getNumRegisters(Context, OpInfo.ConstraintVT);
+    NumRegs = TLI.getNumRegisters(Context, OpInfo.ConstraintVT, RC);
 
   // If this is a constraint for a specific physical register, like {r17},
   // assign it now.
 
   // If this associated to a specific register, initialize iterator to correct
   // place. If virtual, make sure we have enough registers
 
   // Initialize iterator if necessary
void SelectionDAGBuilder::visitInlineAsm(const CallBase &Call) {
   // Third pass: Loop over operands to prepare DAG-level operands.. As part of
   // this, assign virtual and physical registers for inputs and otput.
   for (SDISelAsmOperandInfo &OpInfo : ConstraintOperands) {
     // Assign Registers.
     SDISelAsmOperandInfo &RefOpInfo =
         OpInfo.isMatchingInputConstraint()
             ? ConstraintOperands[OpInfo.getMatchedOperand()]
             : OpInfo;
-    GetRegistersForValue(DAG, getCurSDLoc(), OpInfo, RefOpInfo);
+    const TargetRegisterClass *RC = nullptr;
+    GetRegistersForValue(DAG, getCurSDLoc(), OpInfo, RefOpInfo, RC);
 
     auto DetectWriteToReservedRegister = [&]() {
       const MachineFunction &MF = DAG.getMachineFunction();
       const TargetRegisterInfo &TRI = *MF.getSubtarget().getRegisterInfo();
       for (unsigned Reg : OpInfo.AssignedRegs.Regs) {
         if (Register::isPhysicalRegister(Reg) &&
             TRI.isInlineAsmReadOnlyReg(MF, Reg)) {
           const char *RegName = TRI.getName(Reg);
case InlineAsm::isOutput:
         if (DetectWriteToReservedRegister())
           return;
 
         // Add information to the INLINEASM node to know that this register is
         // set.
         OpInfo.AssignedRegs.AddInlineAsmOperands(
             OpInfo.isEarlyClobber ? InlineAsm::Kind_RegDefEarlyClobber
                                   : InlineAsm::Kind_RegDef,
-            false, 0, getCurSDLoc(), DAG, AsmNodeOperands);
+            false, 0, getCurSDLoc(), DAG, AsmNodeOperands, RC);
       }
       break;
 
     case InlineAsm::isInput: {
       SDValue InOperandVal = OpInfo.CallOperand;
 
       if (OpInfo.isMatchingInputConstraint()) {
         // If this is required to match an output register we have already set,
case InlineAsm::isInput: {
                                      "don't know how to handle tied "
                                      "indirect register inputs");
             return;
           }
 
           MVT RegVT = AsmNodeOperands[CurOp+1].getSimpleValueType();
           SmallVector<unsigned, 4> Regs;
 
-          if (const TargetRegisterClass *RC = TLI.getRegClassFor(RegVT)) {
+          unsigned RCID;
+          bool Succ = InlineAsm::hasRegClassConstraint(OpFlag, RCID);
+          assert(Succ && "No RC for tied def?"); (void)Succ;
+          const TargetRegisterInfo *TRI =
+            DAG.getMachineFunction().getSubtarget().getRegisterInfo();
+          if (const TargetRegisterClass *TiedRC = TRI->getRegClass(RCID)) {
             unsigned NumRegs = InlineAsm::getNumOperandRegisters(OpFlag);
             MachineRegisterInfo &RegInfo =
                 DAG.getMachineFunction().getRegInfo();
             for (unsigned i = 0; i != NumRegs; ++i)
-              Regs.push_back(RegInfo.createVirtualRegister(RC));
+              Regs.push_back(RegInfo.createVirtualRegister(TiedRC));
           } else {
             emitInlineAsmError(Call,
                                "inline asm error: This value type register "
                                "class is not natively supported!");
             return;
           }
 
           RegsForValue MatchedRegs(Regs, RegVT, InOperandVal.getValueType());
 
           SDLoc dl = getCurSDLoc();
           // Use the produced MatchedRegs object to
           MatchedRegs.getCopyToRegs(InOperandVal, DAG, dl, Chain, &Flag, &Call);
           MatchedRegs.AddInlineAsmOperands(InlineAsm::Kind_RegUse,
                                            true, OpInfo.getMatchedOperand(), dl,
-                                           DAG, AsmNodeOperands);
+                                           DAG, AsmNodeOperands, RC);
           break;
         }
 
         assert(InlineAsm::isMemKind(OpFlag) && "Unknown matching constraint!");
         assert(InlineAsm::getNumOperandRegisters(OpFlag) == 1 &&
                "Unexpected number of operands");
         // Add information to the INLINEASM node to know about this input.
         // See InlineAsm.h isUseOperandTiedToDef.
case InlineAsm::isInput: {
       // Treat indirect 'X' constraint as memory.
       if (OpInfo.ConstraintType == TargetLowering::C_Other &&
           OpInfo.isIndirect)
         OpInfo.ConstraintType = TargetLowering::C_Memory;
 
       if (OpInfo.ConstraintType == TargetLowering::C_Immediate ||
           OpInfo.ConstraintType == TargetLowering::C_Other) {
         std::vector<SDValue> Ops;
         TLI.LowerAsmOperandForConstraint(InOperandVal, OpInfo.ConstraintCode,
                                           Ops, DAG);
         if (Ops.empty()) {
           if (OpInfo.ConstraintType == TargetLowering::C_Immediate)
             if (isa<ConstantSDNode>(InOperandVal)) {
               emitInlineAsmError(Call, "value out of range for constraint '" +
                                            Twine(OpInfo.ConstraintCode) + "'");
               return;
             }
 
           emitInlineAsmError(Call,
                              "invalid operand for inline asm constraint '" +
                                  Twine(OpInfo.ConstraintCode) + "'");
           return;
         }
 
         // Add information to the INLINEASM node to know about this input.
         unsigned ResOpType =
           InlineAsm::getFlagWord(InlineAsm::Kind_Imm, Ops.size());
         AsmNodeOperands.push_back(DAG.getTargetConstant(
             ResOpType, getCurSDLoc(), TLI.getPointerTy(DAG.getDataLayout())));
         llvm::append_range(AsmNodeOperands, Ops);
         break;
       }
 
       if (OpInfo.ConstraintType == TargetLowering::C_Memory) {
         assert(OpInfo.isIndirect && "Operand must be indirect to be a mem!");
         assert(InOperandVal.getValueType() ==
                    TLI.getPointerTy(DAG.getDataLayout()) &&
                "Memory operands expect pointer values");
case InlineAsm::isInput: {
         return;
 
       SDLoc dl = getCurSDLoc();
 
       OpInfo.AssignedRegs.getCopyToRegs(InOperandVal, DAG, dl, Chain, &Flag,
                                         &Call);
 
       OpInfo.AssignedRegs.AddInlineAsmOperands(InlineAsm::Kind_RegUse, false, 0,
-                                               dl, DAG, AsmNodeOperands);
+                                               dl, DAG, AsmNodeOperands, RC);
       break;
     }
     case InlineAsm::isClobber:
       // Add the clobbered value to the operand list, so that the register
       // allocator is aware that the physreg got clobbered.
       if (!OpInfo.AssignedRegs.Regs.empty())
         OpInfo.AssignedRegs.AddInlineAsmOperands(InlineAsm::Kind_Clobber,
                                                  false, 0, getCurSDLoc(), DAG,
-                                                 AsmNodeOperands);
+                                                 AsmNodeOperands, RC);
       break;
     }
   }
 
   // Finish up input operands.  Set the input chain and add the flag last.
   AsmNodeOperands[InlineAsm::Op_InputChain] = Chain;
   if (Flag.getNode()) AsmNodeOperands.push_back(Flag);
 
auto handleRegAssign = [&](SDValue V) {
     //
     // This can also happen for a return value that disagrees with the register
     // class it is put in, eg. a double in a general-purpose register on a
     // 32-bit machine.
     if (ResultVT != V.getValueType() &&
         ResultVT.getSizeInBits() == V.getValueSizeInBits())
       V = DAG.getNode(ISD::BITCAST, getCurSDLoc(), ResultVT, V);
     else if (ResultVT != V.getValueType() && ResultVT.isInteger() &&
              V.getValueType().isInteger()) {
       // If a result value was tied to an input value, the computed result
       // may have a wider width than the expected result.  Extract the
       // relevant portion.
       V = DAG.getNode(ISD::TRUNCATE, getCurSDLoc(), ResultVT, V);
     }
     assert(ResultVT == V.getValueType() && "Asm result value mismatch!");
     ResultVTs.push_back(ResultVT);
     ResultValues.push_back(V);

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

for (const TargetRegisterClass *RC : RI->regclasses()) {
     for (const MCPhysReg &PR : *RC) {
       if (RegName.equals_lower(RI->getRegAsmName(PR))) {
         std::pair<unsigned, const TargetRegisterClass *> S =
             std::make_pair(PR, RC);
 
         // If this register class has the requested value type, return it,
         // otherwise keep searching and return the first class found
         // if no other is found which explicitly has the requested type.
-        if (RI->isTypeLegalForClass(*RC, VT))
+        if (RI->isTypeLegalForClass(*RC, VT) && RC->isAllocatable())
           return S;
         if (!R.second)
+          //            RC->isAllocatable() && !R.second->isAllocatable())
           R = S;
       }
     }
   }
 
   return R;
 }
 

llvm/lib/Target/ARM/ARMISelDAGToDAG.cpp

if (Kind != InlineAsm::Kind_RegUse && Kind != InlineAsm::Kind_RegDef
         && Kind != InlineAsm::Kind_RegDefEarlyClobber)
       continue;
 
     unsigned RC;
     bool HasRC = InlineAsm::hasRegClassConstraint(Flag, RC);
     if ((!IsTiedToChangedOp && (!HasRC || RC != ARM::GPRRegClassID))
         || NumRegs != 2)
       continue;
+    if (HasRC && NumRegs) {
+      // Check if RC was encoded while reg(s) is already assigned.
+      Register Reg = cast<RegisterSDNode>(N->getOperand(i + 1))->getReg();
+      if (Register::isPhysicalRegister(Reg))
+        continue;
+    }
 
     assert((i+2 < NumOps) && "Invalid number of operands in inline asm");
     SDValue V0 = N->getOperand(i+1);
     SDValue V1 = N->getOperand(i+2);
     unsigned Reg0 = cast<RegisterSDNode>(V0)->getReg();
     unsigned Reg1 = cast<RegisterSDNode>(V1)->getReg();
     SDValue PairedReg;
     MachineRegisterInfo &MRI = MF->getRegInfo();

llvm/lib/Target/Sparc/SparcISelDAGToDAG.cpp

if (Kind != InlineAsm::Kind_RegUse && Kind != InlineAsm::Kind_RegDef
         && Kind != InlineAsm::Kind_RegDefEarlyClobber)
       continue;
 
     unsigned RC;
     bool HasRC = InlineAsm::hasRegClassConstraint(Flag, RC);
     if ((!IsTiedToChangedOp && (!HasRC || RC != SP::IntRegsRegClassID))
         || NumRegs != 2)
       continue;
+    if (HasRC && NumRegs) {
+      // Check if RC was encoded while reg(s) is already assigned.
+      Register Reg = cast<RegisterSDNode>(N->getOperand(i + 1))->getReg();
+      if (Register::isPhysicalRegister(Reg))
+        continue;
+    }
 
     assert((i+2 < NumOps) && "Invalid number of operands in inline asm");
     SDValue V0 = N->getOperand(i+1);
     SDValue V1 = N->getOperand(i+2);
     unsigned Reg0 = cast<RegisterSDNode>(V0)->getReg();
     unsigned Reg1 = cast<RegisterSDNode>(V1)->getReg();
     SDValue PairedReg;
     MachineRegisterInfo &MRI = MF->getRegInfo();

llvm/lib/Target/SystemZ/SystemZISelLowering.h

TargetLoweringBase::LegalizeTypeAction getPreferredVectorAction(MVT VT)
     //     so promoting the integers is less efficient.
     //
     // (c) there are no multiplication instructions for the widest integer
     //     type (v2i64).
     if (VT.getScalarSizeInBits() % 8 == 0)
       return TypeWidenVector;
     return TargetLoweringBase::getPreferredVectorAction(VT);
   }
+  const TargetRegisterClass *
+  getRegClassFor(MVT VT, bool isDivergent = false) const override {
+    if (VT == MVT::Untyped)  // Needed for inline asm phys regs.
+      return &SystemZ::GR128BitRegClass;
+    return TargetLowering::getRegClassFor(VT);
+  }
+  unsigned
+  getNumRegisters(LLVMContext &Context, EVT VT,
+                  const TargetRegisterClass *RC = nullptr) const override {
+    if (VT == MVT::i128 && RC == &SystemZ::GR128BitRegClass)
+      return 1;
+    return TargetLowering::getNumRegisters(Context, VT);
+  }
   bool isCheapToSpeculateCtlz() const override { return true; }
   EVT getSetCCResultType(const DataLayout &DL, LLVMContext &,
                          EVT) const override;
   bool isFMAFasterThanFMulAndFAdd(const MachineFunction &MF,
                                   EVT VT) const override;
   bool isFPImmLegal(const APFloat &Imm, EVT VT,
                     bool ForCodeSize) const override;
   bool hasInlineStackProbe(MachineFunction &MF) const override;
public:
   SDValue LowerOperation(SDValue Op, SelectionDAG &DAG) const override;
   void LowerOperationWrapper(SDNode *N, SmallVectorImpl<SDValue> &Results,
                              SelectionDAG &DAG) const override;
   void ReplaceNodeResults(SDNode *N, SmallVectorImpl<SDValue>&Results,
                           SelectionDAG &DAG) const override;
   const MCPhysReg *getScratchRegisters(CallingConv::ID CC) const override;
   bool allowTruncateForTailCall(Type *, Type *) const override;
   bool mayBeEmittedAsTailCall(const CallInst *CI) const override;
+  bool splitValueIntoRegisterParts(SelectionDAG &DAG, const SDLoc &DL,
+                                   SDValue Val, SDValue *Parts,
+                                   unsigned NumParts, MVT PartVT,
+                                   Optional<CallingConv::ID> CC) const override;
+  SDValue
+  joinRegisterPartsIntoValue(SelectionDAG &DAG, const SDLoc &DL,
+                             const SDValue *Parts, unsigned NumParts,
+                             MVT PartVT, EVT ValueVT,
+                             Optional<CallingConv::ID> CC) const override;
   SDValue LowerFormalArguments(SDValue Chain, CallingConv::ID CallConv,
                                bool isVarArg,
                                const SmallVectorImpl<ISD::InputArg> &Ins,
                                const SDLoc &DL, SelectionDAG &DAG,
                                SmallVectorImpl<SDValue> &InVals) const override;
   SDValue LowerCall(CallLoweringInfo &CLI,
                     SmallVectorImpl<SDValue> &InVals) const override;
 

llvm/lib/Target/SystemZ/SystemZISelLowering.cpp

return DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, VA.getLocVT(), Value,
                        DAG.getConstant(0, DL, MVT::i32));
   case CCValAssign::Full:
     return Value;
   default:
     llvm_unreachable("Unhandled getLocInfo()");
   }
 }
 
+static SDValue lowerI128ToGR128(SelectionDAG &DAG, SDValue In) {
+  SDLoc DL(In);
+  SDValue Lo = DAG.getNode(ISD::EXTRACT_ELEMENT, DL, MVT::i64, In,
+                           DAG.getIntPtrConstant(0, DL));
+  SDValue Hi = DAG.getNode(ISD::EXTRACT_ELEMENT, DL, MVT::i64, In,
+                           DAG.getIntPtrConstant(1, DL));
+  SDNode *Pair = DAG.getMachineNode(SystemZ::PAIR128, DL,
+                                    MVT::Untyped, Hi, Lo);
+  return SDValue(Pair, 0);
+}
+
+static SDValue lowerGR128ToI128(SelectionDAG &DAG, SDValue In) {
+  SDLoc DL(In);
+  SDValue Hi = DAG.getTargetExtractSubreg(SystemZ::subreg_h64,
+                                          DL, MVT::i64, In);
+  SDValue Lo = DAG.getTargetExtractSubreg(SystemZ::subreg_l64,
+                                          DL, MVT::i64, In);
+  return DAG.getNode(ISD::BUILD_PAIR, DL, MVT::i128, Lo, Hi);
+}
+
+bool SystemZTargetLowering::splitValueIntoRegisterParts(
+    SelectionDAG &DAG, const SDLoc &DL, SDValue Val, SDValue *Parts,
+    unsigned NumParts, MVT PartVT, Optional<CallingConv::ID> CC) const {
+  EVT ValueVT = Val.getValueType();
+  assert((ValueVT != MVT::i128 ||
+          ((NumParts == 1 && PartVT == MVT::Untyped) ||
+           (NumParts == 2 && PartVT == MVT::i64))) &&
+         "Unknown handling of i128 value.");
+  if (ValueVT == MVT::i128 && NumParts == 1) {
+    // Inline assembly operand.
+    Parts[0] = lowerI128ToGR128(DAG, Val);
+    return true;
+  }
+  return false;
+}
+
+SDValue SystemZTargetLowering::joinRegisterPartsIntoValue(
+    SelectionDAG &DAG, const SDLoc &DL, const SDValue *Parts, unsigned NumParts,
+    MVT PartVT, EVT ValueVT, Optional<CallingConv::ID> CC) const {
+  assert((ValueVT != MVT::i128 ||
+          ((NumParts == 1 && PartVT == MVT::Untyped) ||
+           (NumParts == 2 && PartVT == MVT::i64))) &&
+         "Unknown handling of i128 value.");
+  if (ValueVT == MVT::i128 && NumParts == 1)
+    // Inline assembly operand.
+    return lowerGR128ToI128(DAG, Parts[0]);
+  return SDValue();
+}
+
 SDValue SystemZTargetLowering::LowerFormalArguments(
     SDValue Chain, CallingConv::ID CallConv, bool IsVarArg,
     const SmallVectorImpl<ISD::InputArg> &Ins, const SDLoc &DL,
     SelectionDAG &DAG, SmallVectorImpl<SDValue> &InVals) const {
   MachineFunction &MF = DAG.getMachineFunction();
   MachineFrameInfo &MFI = MF.getFrameInfo();
   MachineRegisterInfo &MRI = MF.getRegInfo();
   SystemZMachineFunctionInfo *FuncInfo =
case ISD::SRA:
     return lowerShift(Op, DAG, SystemZISD::VSRA_BY_SCALAR);
   default:
     llvm_unreachable("Unexpected node to lower");
   }
 }
 
 // Lower operations with invalid operand or result types (currently used
 // only for 128-bit integer types).
-
-static SDValue lowerI128ToGR128(SelectionDAG &DAG, SDValue In) {
-  SDLoc DL(In);
-  SDValue Lo = DAG.getNode(ISD::EXTRACT_ELEMENT, DL, MVT::i64, In,
-                           DAG.getIntPtrConstant(0, DL));
-  SDValue Hi = DAG.getNode(ISD::EXTRACT_ELEMENT, DL, MVT::i64, In,
-                           DAG.getIntPtrConstant(1, DL));
-  SDNode *Pair = DAG.getMachineNode(SystemZ::PAIR128, DL,
-                                    MVT::Untyped, Hi, Lo);
-  return SDValue(Pair, 0);
-}
-
-static SDValue lowerGR128ToI128(SelectionDAG &DAG, SDValue In) {
-  SDLoc DL(In);
-  SDValue Hi = DAG.getTargetExtractSubreg(SystemZ::subreg_h64,
-                                          DL, MVT::i64, In);
-  SDValue Lo = DAG.getTargetExtractSubreg(SystemZ::subreg_l64,
-                                          DL, MVT::i64, In);
-  return DAG.getNode(ISD::BUILD_PAIR, DL, MVT::i128, Lo, Hi);
-}
-
 void
 SystemZTargetLowering::LowerOperationWrapper(SDNode *N,
                                              SmallVectorImpl<SDValue> &Results,
                                              SelectionDAG &DAG) const {
   switch (N->getOpcode()) {
   case ISD::ATOMIC_LOAD: {
     SDLoc DL(N);
     SDVTList Tys = DAG.getVTList(MVT::Untyped, MVT::Other);

llvm/lib/Target/X86/X86FloatingPoint.cpp

for (unsigned i = InlineAsm::MIOp_FirstOperand, e = MI.getNumOperands();
         continue;
       const MachineOperand &MO = MI.getOperand(i + 1);
       if (!MO.isReg())
         continue;
       unsigned STReg = MO.getReg() - X86::FP0;
       if (STReg >= 8)
         continue;
 
-      // If the flag has a register class constraint, this must be an operand
-      // with constraint "f". Record its index and continue.
-      if (InlineAsm::hasRegClassConstraint(Flags, RCID)) {
+      // If the flag has an FP Stack registers regclass constraint, record
+      // its index and continue.  EXPERIMENTAL: This doesn't quite work since
+      // this is run after reg-alloc. Only virtual registers will have
+      // hasRegClassConstraint() currently, but there seems to be no good way
+      // to do the same check here while encoding the RC for all operands in
+      // SelectionDAGBuilder. Seems there would have to be a new bit added to
+      // Flags to indicate if the reg is originally virtual or physical..?
+      if (InlineAsm::hasRegClassConstraint(Flags, RCID) &&
+          (RCID == X86::RFP32RegClassID || RCID == X86::RFP64RegClassID ||
+           RCID == X86::RFP80RegClassID)) {
           FRegIdx.insert(i + 1);
           continue;
         }
 
       switch (InlineAsm::getKind(Flags)) {
       case InlineAsm::Kind_RegUse:
         STUses |= (1u << STReg);
         break;
       case InlineAsm::Kind_RegDef:
       case InlineAsm::Kind_RegDefEarlyClobber:
         STDefs |= (1u << STReg);

llvm/test/CodeGen/SystemZ/inline-asm-i128.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=s390x-linux-gnu -no-integrated-as < %s | FileCheck %s
+;
+; Test i128 (tied) operands.
+
+define i32 @clcl(i8* %p1, i32 signext %l1, i8* %p2, i32 signext %l2, i8 zeroext %pad) {
+; CHECK-LABEL: clcl:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    lgr %r0, %r5
+; CHECK-NEXT:    # kill: def $r4d killed $r4d def $r4q
+; CHECK-NEXT:    lgr %r1, %r3
+; CHECK-NEXT:    # kill: def $r2d killed $r2d def $r2q
+; CHECK-NEXT:    sllg %r5, %r6, 24
+; CHECK-NEXT:    rosbg %r5, %r0, 40, 63, 0
+; CHECK-NEXT:    risbg %r3, %r1, 40, 191, 0
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    clcl %r2, %r4
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    ogr %r3, %r5
+; CHECK-NEXT:    risbg %r0, %r3, 40, 191, 0
+; CHECK-NEXT:    ipm %r2
+; CHECK-NEXT:    afi %r2, -268435456
+; CHECK-NEXT:    srl %r2, 31
+; CHECK-NEXT:    br %r14
+entry:
+  %0 = ptrtoint i8* %p1 to i64
+  %1 = ptrtoint i8* %p2 to i64
+  %and5 = and i32 %l2, 16777215
+  %2 = zext i32 %and5 to i64
+  %conv7 = zext i8 %pad to i64
+  %shl = shl nuw nsw i64 %conv7, 24
+  %or = or i64 %shl, %2
+  %u1.sroa.0.0.insert.ext = zext i64 %0 to i128
+  %u1.sroa.0.0.insert.shift = shl nuw i128 %u1.sroa.0.0.insert.ext, 64
+  %3 = and i32 %l1, 16777215
+  %u1.sroa.0.0.insert.mask = zext i32 %3 to i128
+  %u1.sroa.0.0.insert.insert = or i128 %u1.sroa.0.0.insert.shift, %u1.sroa.0.0.insert.mask
+  %u2.sroa.5.0.insert.ext = zext i64 %or to i128
+  %u2.sroa.0.0.insert.ext = zext i64 %1 to i128
+  %u2.sroa.0.0.insert.shift = shl nuw i128 %u2.sroa.0.0.insert.ext, 64
+  %u2.sroa.0.0.insert.insert = or i128 %u2.sroa.0.0.insert.shift, %u2.sroa.5.0.insert.ext
+  %4 = tail call { i128, i128 } asm "clcl $0, $1", "=r,=r,0,1"(i128 %u1.sroa.0.0.insert.insert, i128 %u2.sroa.0.0.insert.insert)
+  %asmresult = extractvalue { i128, i128 } %4, 0
+  %asmresult11 = extractvalue { i128, i128 } %4, 1
+  %5 = or i128 %asmresult, %asmresult11
+  %6 = and i128 %5, 16777215
+  %7 = icmp eq i128 %6, 0
+  %land.ext = zext i1 %7 to i32
+  ret i32 %land.ext
+}
+
+; Test a tied phys-reg.
+define void @fun(i128* %Src, i128* %Dst) {
+entry:
+  %L = load i128, i128* %Src
+  %IAsm = call i128 asm "BLA $0, $1", "={r0},0"(i128 %L)
+  store volatile i128 %IAsm, i128* %Dst
+  ret void
+}

llvm/test/CodeGen/X86/callbr-asm-bb-exports.ll

 ; CHECK-NEXT: t12: ch = CopyToReg t0, Register:i32 %0, t10
 ; CHECK-NEXT: t6: i32,ch = CopyFromReg t0, Register:i32 %4
 ; CHECK-NEXT: t13: i32 = add t6, Constant:i32<1>
 ; CHECK-NEXT: t15: ch = CopyToReg t0, Register:i32 %1, t13
 ; CHECK-NEXT: t17: ch = TokenFactor t12, t15
 ; CHECK-NEXT: t2: i32,ch = CopyFromReg t0, Register:i32 %2
 ; CHECK-NEXT: t8: i32 = add t2, Constant:i32<4>
 ; CHECK-NEXT: t22: ch,glue = CopyToReg t17, Register:i32 %5, t8
-; CHECK-NEXT: t29: ch,glue = inlineasm_br t22, {{.*}}, t22:1
+; CHECK-NEXT: t31: ch,glue = inlineasm_br t22, {{.*}}, t22:1
 
 define i32 @test(i32 %a, i32 %b, i32 %c) {
 entry:
   %0 = add i32 %a, 4
   %1 = add i32 %b, 1
   %2 = add i32 %c, 1
   callbr void asm "xorl $0, $0; jmp ${1:l}", "r,X,~{dirflag},~{fpsr},~{flags}"(i32 %0, i8* blockaddress(@test, %fail)) to label %normal [label %fail]
 
 normal:
   ret i32 %1
 
 fail:
   ret i32 %2
 }

llvm/test/CodeGen/X86/inline-asm-avx512f-x-constraint.ll

 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx512f -stop-after=finalize-isel | FileCheck %s
 
 ; CHECK: %[[REG1:.*]]:vr512_0_15 = COPY %1
 ; CHECK: %[[REG2:.*]]:vr512_0_15 = COPY %2
-; CHECK: INLINEASM &"vpaddq\09$3, $2, $0 {$1}", 0 /* attdialect */, {{.*}}, def %{{.*}}, {{.*}}, %{{.*}}, {{.*}}, %[[REG1]], {{.*}}, %[[REG2]], 12 /* clobber */, implicit-def early-clobber $df, 12 /* clobber */, implicit-def early-clobber $fpsw, 12 /* clobber */, implicit-def early-clobber $eflags
+; CHECK: INLINEASM &"vpaddq\09$3, $2, $0 {$1}", 0 /* attdialect */, 7733258 /* regdef:VR512_0_15 */, def %3, 1179657 /* reguse:VK8WM */, %4, 7733257 /* reguse:VR512_0_15 */, %5, 7733257 /* reguse:VR512_0_15 */, %6, 3538956 /* clobber:DFCCR */, implicit-def early-clobber $df, 1376268 /* clobber:FPCCR */, implicit-def early-clobber $fpsw, 3473420 /* clobber:CCR */, implicit-def early-clobber $eflags
 
 define <8 x i64> @mask_Yk_i8(i8 signext %msk, <8 x i64> %x, <8 x i64> %y) {
 entry:
   %0 = tail call <8 x i64> asm "vpaddq\09$3, $2, $0 {$1}", "=x,^Yk,x,x,~{dirflag},~{fpsr},~{flags}"(i8 %msk, <8 x i64> %x, <8 x i64> %y)
   ret <8 x i64> %0
 }

llvm/test/CodeGen/X86/inline-asm-default-clobbers.ll

 ; RUN: llc < %s -mtriple=i686 -stop-after=finalize-isel | FileCheck %s
 
-; CHECK: INLINEASM &"", 1 /* sideeffect attdialect */, 12 /* clobber */, implicit-def early-clobber $df, 12 /* clobber */, implicit-def early-clobber $fpsw, 12 /* clobber */, implicit-def early-clobber $eflags
+; CHECK: INLINEASM &"", 1 /* sideeffect attdialect */, 3538956 /* clobber:DFCCR */, implicit-def early-clobber $df, 1376268 /* clobber:FPCCR */, implicit-def early-clobber $fpsw, 3473420 /* clobber:CCR */, implicit-def early-clobber $eflags
+
 define void @foo() {
 entry:
   call void asm sideeffect "", "~{dirflag},~{fpsr},~{flags}"()
   ret void
 }

llvm/test/CodeGen/X86/tail-dup-asm-goto.ll

define i8* @test1(i8** %arg1, i8* %arg2) {
   ; CHECK: bb.2.bb106:
   ; CHECK:   successors: %bb.3(0x80000000)
   ; CHECK:   ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
   ; CHECK:   CALL64pcrel32 @foo, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp
   ; CHECK:   ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
   ; CHECK: bb.3.bb110:
   ; CHECK:   successors: %bb.5(0x80000000), %bb.4(0x00000000)
   ; CHECK:   [[PHI:%[0-9]+]]:gr64 = PHI [[COPY]], %bb.2, [[MOV64rm]], %bb.1
-  ; CHECK:   INLINEASM_BR &"#$0 $1 $2", 9 /* sideeffect mayload attdialect */, 13 /* imm */, 42, 13 /* imm */, 0, 13 /* imm */, blockaddress(@test1, %ir-block.bb17.i.i.i), 12 /* clobber */, implicit-def early-clobber $df, 12 /* clobber */, implicit-def early-clobber $fpsw, 12 /* clobber */, implicit-def early-clobber $eflags
+  ; CHECK:   INLINEASM_BR &"#$0 $1 $2", 9 /* sideeffect mayload attdialect */, 13 /* imm */, 42, 13 /* imm */, 0, 13 /* imm */, blockaddress(@test1, %ir-block.bb17.i.i.i), 3538956 /* clobber:DFCCR */, implicit-def early-clobber $df, 1376268 /* clobber:FPCCR */, implicit-def early-clobber $fpsw, 3473420 /* clobber:CCR */, implicit-def early-clobber $eflags
   ; CHECK:   JMP_1 %bb.5
   ; CHECK: bb.4.bb17.i.i.i (address-taken):
   ; CHECK:   successors: %bb.5(0x80000000)
   ; CHECK: bb.5.kmem_cache_has_cpu_partial.exit:
   ; CHECK:   $rax = COPY [[PHI]]
   ; CHECK:   RET 0, $rax
 bb:
   %i28.i = load i8*, i8** %arg1, align 8