[AArch64][SME] Add support for arm_locally_streaming functions.

[sdesmalen]

Summary

Functions with aarch64_sme_pstatesm_body will emit a SMSTART at the start
of the function, and a SMSTOP at the end of the function, such that all
operations use the right value for vscale.

Because the placement of these nodes is critically important (i.e. no
vscale-dependent operations should be done before SMSTART has been issued),
we require glueing the CopyFromReg to the Entry node such that we can
insert the SMSTART as part of that glued chain.

More details about the SME attributes and design can be found
in D131562.

[aemerson]

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

1165

?

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

1285

I think we need a better understanding.

[sdesmalen]

Removed FIXME and work-around for constructor of SelectionDAG.

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

1165

Perhaps removing this without any alternative assert is a little crude, but I wasn't really sure what to test for instead.

The change here is that EntryToken now "becomes part of the schedule" because other instructions can be glued to it, even though the node itself isn't schedulable (it is always guaranteed to be the first instruction). I was hoping someone would have a suggestion here.

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

1285

You're right. The issue was that calling getVTList with more than one operand went through a different code-path which queries the VTListMap, which is initialised after EntryNode. By reordering these member variables (lexicographically) the issue goes away.

[aemerson]

Seems reasonable to me.

[sdesmalen]

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

1165

@efriedma just wanting a second opinion on this, are you aware of anything else I might need to worry about here?

I needed to remove this llvm_unreachable because I changed EntryToken to also have Glue, and when using that it makes the EntryToken part of the schedule (even though the node itself can't be scheduled).

[efriedma]

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

1165

This feels pretty isolated; the entry token basically has the same meaning during isel, and we're throwing it away after isel. The only possible side-effect I can think of is that maybe there's something assumes the entry token only has one successor.

That said, it feels like a sort of strange thing to be inserting smstart/smend this early... would it make sense to insert it as part of prologue/epilogue lowering instead?

[sdesmalen]

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

1165

I considered that early on but thought doing this at ISEL level would be easier because we get the benefits from the register allocator. Also, we already have the node/mechanisms in place for SelectionDAG, so adding it here seemed sensible.

For some simple example:

define <8 x i32> @foo(<8 x i32> %x) nounwind "aarch64_pstate_sm_body" {
  %y = add <8 x i32> %x, %x
  ret <8 x i32> %y
}

if we were to do this at point of Prologue/Epilogue lowering, we'd have the following input:

# *** IR Dump Before Prologue/Epilogue Insertion & Frame Finalization (prologepilog) ***:                                                                                                                                                                                                                                                                                   
# Machine code for function foo: NoPHIs, TracksLiveness, NoVRegs, TiedOpsRewritten, TracksDebugUserValues                                                                                                                                                                                                                                                                   
Function Live Ins: $q0, $q1                                                                                                                                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                                                                                                                            
bb.0 (%ir-block.0):                                                                                                                                                                                                                                                                                                                                                         
  liveins: $q0, $q1                                                                                                                                                                                                                                                                                                                                                         
  renamable $q0 = ADDv4i32 killed renamable $q0, renamable $q0                                                                                                                                                                                                                                                                                                              
  renamable $q1 = ADDv4i32 killed renamable $q1, renamable $q1                                                                                                                                                                                                                                                                                                              
  RET_ReallyLR implicit $q0, implicit $q1

We'd then first need to do analysis which registers are clobbered by the smstart/smstop, add extra spill slots for the input arguments passed in NEON/FP/SVE registers, and then add the spills/smstart/reloads for the prolgoue, and spills/smstop/reloads for the epilogue. The frame-lowering code is already quite complex. We'd need to end up with something like:

foo:                                    // @foo
// %bb.0:
        sub     sp, sp, #96
        stp     d15, d14, [sp, #32]
        stp     d13, d12, [sp, #48]
        stp     d11, d10, [sp, #64]
        stp     d9, d8, [sp, #80]
        stp     q0, q1, [sp]                    // input operands are clobbered by smstart, so spill to stack
        smstart sm
        ldr     q0, [sp]                        // reload the spilled operand register
        add     v2.4s, v0.4s, v0.4s
        ldr     q0, [sp, #16]               // reload the other spilled operand register
        add     v0.4s, v0.4s, v0.4s
        stp     q2, q0, [sp]                 // spill the results, because output operands are clobbered by smstop
        smstop  sm
        ldp     q0, q1, [sp]                 // reload the results
        ldp     d9, d8, [sp, #80]
        ldp     d11, d10, [sp, #64]
        ldp     d13, d12, [sp, #48]
        ldp     d15, d14, [sp, #32]
        add     sp, sp, #96
        ret

If we insert the smstart/smstop as part of ISEL the register allocator will insert all the spills and fills for free. And we may have some minor benefits from the scheduler which can rearrange code around the spills/fills for the input arguments.

llvm/include/llvm/CodeGen/SelectionDAG.h

class SelectionDAG {
 
   /// The function-level optimization remark emitter.  Used to emit remarks
   /// whenever manipulating the DAG.
   OptimizationRemarkEmitter *ORE;
 
   ProfileSummaryInfo *PSI = nullptr;
   BlockFrequencyInfo *BFI = nullptr;
 
+  /// List of non-single value types.
+  FoldingSet<SDVTListNode> VTListMap;
+
+  /// Pool allocation for misc. objects that are created once per SelectionDAG.
+  BumpPtrAllocator Allocator;
+
   /// The starting token.
   SDNode EntryNode;
 
   /// The root of the entire DAG.
   SDValue Root;
 
   /// A linked list of nodes in the current DAG.
   ilist<SDNode> AllNodes;
class SelectionDAG {
   /// This structure is used to memoize nodes, automatically performing
   /// CSE with existing nodes when a duplicate is requested.
   FoldingSet<SDNode> CSEMap;
 
   /// Pool allocation for machine-opcode SDNode operands.
   BumpPtrAllocator OperandAllocator;
   ArrayRecycler<SDUse> OperandRecycler;
 
-  /// Pool allocation for misc. objects that are created once per SelectionDAG.
-  BumpPtrAllocator Allocator;
-
   /// Tracks dbg_value and dbg_label information through SDISel.
   SDDbgInfo *DbgInfo;
 
   using CallSiteInfo = MachineFunction::CallSiteInfo;
   using CallSiteInfoImpl = MachineFunction::CallSiteInfoImpl;
 
   struct NodeExtraInfo {
     CallSiteInfo CSInfo;
private:
   SDNode *FindNodeOrInsertPos(const FoldingSetNodeID &ID, void *&InsertPos);
 
   /// Look up the node specified by ID in CSEMap.  If it exists, return it.  If
   /// not, return the insertion token that will make insertion faster.  Performs
   /// additional processing for constant nodes.
   SDNode *FindNodeOrInsertPos(const FoldingSetNodeID &ID, const SDLoc &DL,
                               void *&InsertPos);
 
-  /// List of non-single value types.
-  FoldingSet<SDVTListNode> VTListMap;
-
   /// Maps to auto-CSE operations.
   std::vector<CondCodeSDNode*> CondCodeNodes;
 
   std::vector<SDNode*> ValueTypeNodes;
   std::map<EVT, SDNode*, EVT::compareRawBits> ExtendedValueTypeNodes;
   StringMap<SDNode*> ExternalSymbols;
 
   std::map<std::pair<std::string, unsigned>, SDNode *> TargetExternalSymbols;

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

EmitSpecialNode(SDNode *Node, bool IsClone, bool IsCloned,
                 DenseMap<SDValue, Register> &VRBaseMap) {
   switch (Node->getOpcode()) {
   default:
 #ifndef NDEBUG
     Node->dump();
 #endif
     llvm_unreachable("This target-independent node should have been selected!");
   case ISD::EntryToken:
-    llvm_unreachable("EntryToken should have been excluded from the schedule!");
   case ISD::MERGE_VALUES:
   case ISD::TokenFactor: // fall thru
     break;
   case ISD::CopyToReg: {
     Register DestReg = cast<RegisterSDNode>(Node->getOperand(1))->getReg();
     SDValue SrcVal = Node->getOperand(2);
     if (Register::isVirtualRegister(DestReg) && SrcVal.isMachineOpcode() &&
         SrcVal.getMachineOpcode() == TargetOpcode::IMPLICIT_DEF) {

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

Type *Ty = VT == MVT::iPTR ?
                    VT.getTypeForEVT(*getContext());
 
   return getDataLayout().getABITypeAlign(Ty);
 }
 
 // EntryNode could meaningfully have debug info if we can find it...
 SelectionDAG::SelectionDAG(const TargetMachine &tm, CodeGenOpt::Level OL)
     : TM(tm), OptLevel(OL),
-      EntryNode(ISD::EntryToken, 0, DebugLoc(), getVTList(MVT::Other)),
+      EntryNode(ISD::EntryToken, 0, DebugLoc(), getVTList(MVT::Other, MVT::Glue)),
       Root(getEntryNode()) {
   InsertNode(&EntryNode);
   DbgInfo = new SDDbgInfo();
 }
 
 void SelectionDAG::init(MachineFunction &NewMF,
                         OptimizationRemarkEmitter &NewORE,
                         Pass *PassPtr, const TargetLibraryInfo *LibraryInfo,
                         LegacyDivergenceAnalysis * Divergence,
                         ProfileSummaryInfo *PSIin,
                         BlockFrequencyInfo *BFIin) {
   MF = &NewMF;
   SDAGISelPass = PassPtr;
   ORE = &NewORE;
   TLI = getSubtarget().getTargetLowering();

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

if (IsWin64)
       UseVarArgCC = isVarArg;
     CCAssignFn *AssignFn = CCAssignFnForCall(CallConv, UseVarArgCC);
     bool Res =
         AssignFn(i, ValVT, ValVT, CCValAssign::Full, Ins[i].Flags, CCInfo);
     assert(!Res && "Call operand has unhandled type");
     (void)Res;
   }
 
+  SMEAttrs Attrs(MF.getFunction());
+  bool IsLocallyStreaming =
+      !Attrs.hasStreamingInterface() && Attrs.hasStreamingBody();
+  assert(Chain.getOpcode() == ISD::EntryToken && "Unexpected Chain value");
+  SDValue Glue = Chain.getValue(1);
+
+  SmallVector<SDValue, 16> ArgValues;
   unsigned ExtraArgLocs = 0;
   for (unsigned i = 0, e = Ins.size(); i != e; ++i) {
     CCValAssign &VA = ArgLocs[i - ExtraArgLocs];
 
     if (Ins[i].Flags.isByVal()) {
       // Byval is used for HFAs in the PCS, but the system should work in a
       // non-compliant manner for larger structs.
       EVT PtrVT = getPointerTy(DAG.getDataLayout());
if (VA.isRegLoc()) {
       } else if (RegVT.isScalableVector()) {
         FuncInfo->setIsSVECC(true);
         RC = &AArch64::ZPRRegClass;
       } else
         llvm_unreachable("RegVT not supported by FORMAL_ARGUMENTS Lowering");
 
       // Transform the arguments in physical registers into virtual ones.
       Register Reg = MF.addLiveIn(VA.getLocReg(), RC);
+
+      if (IsLocallyStreaming) {
+        // LocallyStreamingFunctions must insert the SMSTART in the correct
+        // position, so we use Glue to ensure no instructions can be scheduled
+        // between the chain of:
+        //        t0: ch,glue = EntryNode
+        //      t1:  res,ch,glue = CopyFromReg
+        //     ...
+        //   tn: res,ch,glue = CopyFromReg t(n-1), ..
+        // t(n+1): ch, glue = SMSTART t0:0, ...., tn:2
+        // ^^^^^^
+        // This will be the new Chain/Root node.
+        ArgValue = DAG.getCopyFromReg(Chain, DL, Reg, RegVT, Glue);
+        Glue = ArgValue.getValue(2);
+      } else
         ArgValue = DAG.getCopyFromReg(Chain, DL, Reg, RegVT);
 
       // If this is an 8, 16 or 32-bit value, it is really passed promoted
       // to 64 bits.  Insert an assert[sz]ext to capture this, then
       // truncate to the right size.
       switch (VA.getLocInfo()) {
       default:
         llvm_unreachable("Unknown loc info!");
       case CCValAssign::Full:
if (VA.getLocInfo() == CCValAssign::Indirect) {
         }
       }
 
       InVals.push_back(ArgValue);
     }
   }
   assert((ArgLocs.size() + ExtraArgLocs) == Ins.size());
 
+  // Insert the SMSTART if this is a locally streaming function and
+  // make sure it is Glued to the last CopyFromReg value.
+  if (IsLocallyStreaming) {
+    const AArch64RegisterInfo *TRI = Subtarget->getRegisterInfo();
+    Chain = DAG.getNode(
+        AArch64ISD::SMSTART, DL, DAG.getVTList(MVT::Other, MVT::Glue),
+        {DAG.getRoot(),
+          DAG.getTargetConstant((int32_t)AArch64SVCR::SVCRSM, DL, MVT::i32),
+         DAG.getConstant(0, DL, MVT::i64), DAG.getConstant(1, DL, MVT::i64),
+         DAG.getRegisterMask(TRI->getSMStartStopCallPreservedMask()), Glue});
+    // Ensure that the SMSTART happens after the CopyWithChain such that its
+    // chain result is used.
+    for (unsigned I=0; I<InVals.size(); ++I) {
+      Register Reg = MF.getRegInfo().createVirtualRegister(
+          getRegClassFor(InVals[I].getValueType().getSimpleVT()));
+      SDValue X = DAG.getCopyToReg(Chain, DL, Reg, InVals[I]);
+      InVals[I] = DAG.getCopyFromReg(X, DL, Reg,
+                                     InVals[I].getValueType());
+    }
+  }
+
   // varargs
   if (isVarArg) {
     if (!Subtarget->isTargetDarwin() || IsWin64) {
       // The AAPCS variadic function ABI is identical to the non-variadic
       // one. As a result there may be more arguments in registers and we should
       // save them for future reference.
       // Win64 variadic functions also pass arguments in registers, but all float
       // arguments are passed in integer registers.
if (RegsUsed.count(VA.getLocReg())) {
           })->second;
       Bits = DAG.getNode(ISD::OR, DL, Bits.getValueType(), Bits, Arg);
     } else {
       RetVals.emplace_back(VA.getLocReg(), Arg);
       RegsUsed.insert(VA.getLocReg());
     }
   }
 
+  const AArch64RegisterInfo *TRI = Subtarget->getRegisterInfo();
+
+  // Emit SMSTOP before returning from a locally streaming function
+  SMEAttrs FuncAttrs(MF.getFunction());
+  if (FuncAttrs.hasStreamingBody() && !FuncAttrs.hasStreamingInterface()) {
+    Chain = DAG.getNode(
+        AArch64ISD::SMSTOP, DL, DAG.getVTList(MVT::Other, MVT::Glue), Chain,
+        DAG.getTargetConstant((int32_t)AArch64SVCR::SVCRSM, DL, MVT::i32),
+        DAG.getConstant(1, DL, MVT::i64), DAG.getConstant(0, DL, MVT::i64),
+        DAG.getRegisterMask(TRI->getSMStartStopCallPreservedMask()));
+    Flag = Chain.getValue(1);
+  }
+
   SmallVector<SDValue, 4> RetOps(1, Chain);
   for (auto &RetVal : RetVals) {
     Chain = DAG.getCopyToReg(Chain, DL, RetVal.first, RetVal.second, Flag);
     Flag = Chain.getValue(1);
     RetOps.push_back(
         DAG.getRegister(RetVal.first, RetVal.second.getValueType()));
   }
 
   // Windows AArch64 ABIs require that for returning structs by value we copy
   // the sret argument into X0 for the return.
   // We saved the argument into a virtual register in the entry block,
   // so now we copy the value out and into X0.
   if (unsigned SRetReg = FuncInfo->getSRetReturnReg()) {
     SDValue Val = DAG.getCopyFromReg(RetOps[0], DL, SRetReg,
                                      getPointerTy(MF.getDataLayout()));
 
     unsigned RetValReg = AArch64::X0;
     Chain = DAG.getCopyToReg(Chain, DL, RetValReg, Val, Flag);
     Flag = Chain.getValue(1);
 
     RetOps.push_back(
       DAG.getRegister(RetValReg, getPointerTy(DAG.getDataLayout())));
   }
 
-  const AArch64RegisterInfo *TRI = Subtarget->getRegisterInfo();
   const MCPhysReg *I = TRI->getCalleeSavedRegsViaCopy(&MF);
   if (I) {
     for (; *I; ++I) {
       if (AArch64::GPR64RegClass.contains(*I))
         RetOps.push_back(DAG.getRegister(*I, MVT::i64));
       else if (AArch64::FPR64RegClass.contains(*I))
         RetOps.push_back(DAG.getRegister(*I, MVT::getFloatingPointVT(64)));
       else

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

static void emitFrameOffsetAdj(MachineBasicBlock &MBB,
   case AArch64::ADDSXri:
   case AArch64::SUBXri:
   case AArch64::SUBSXri:
     MaxEncoding = 0xfff;
     ShiftSize = 12;
     break;
   case AArch64::ADDVL_XXI:
   case AArch64::ADDPL_XXI:
+  case AArch64::ADDSVL_XXI:
+  case AArch64::ADDSPL_XXI:
     MaxEncoding = 31;
     ShiftSize = 0;
     if (Offset < 0) {
       MaxEncoding = 32;
       Sign = -1;
       Offset = -Offset;
     }
     break;
   default:
     llvm_unreachable("Unsupported opcode");
   }
 
   // `Offset` can be in bytes or in "scalable bytes".
   int VScale = 1;
-  if (Opc == AArch64::ADDVL_XXI)
+  if (Opc == AArch64::ADDVL_XXI || Opc == AArch64::ADDSVL_XXI)
     VScale = 16;
-  else if (Opc == AArch64::ADDPL_XXI)
+  else if (Opc == AArch64::ADDPL_XXI || Opc == AArch64::ADDSPL_XXI)
     VScale = 2;
 
   // FIXME: If the offset won't fit in 24-bits, compute the offset into a
   // scratch register.  If DestReg is a virtual register, use it as the
   // scratch register; otherwise, create a new virtual register (to be
   // replaced by the scavenger at the end of PEI).  That case can be optimized
   // slightly if DestReg is SP which is always 16-byte aligned, so the scratch
   // register can be loaded with offset%8 and the add/sub can use an extending

 void llvm::emitFrameOffset(MachineBasicBlock &MBB,
                            MachineBasicBlock::iterator MBBI, const DebugLoc &DL,
                            unsigned DestReg, unsigned SrcReg,
                            StackOffset Offset, const TargetInstrInfo *TII,
                            MachineInstr::MIFlag Flag, bool SetNZCV,
                            bool NeedsWinCFI, bool *HasWinCFI,
                            bool EmitCFAOffset, StackOffset CFAOffset,
                            unsigned FrameReg) {
+  // If a function is marked as arm_locally_streaming, then the runtime value of
+  // vscale in the prologue/epilogue is different the runtime value of vscale
+  // in the function's body. To avoid having to consider multiple vscales,
+  // we can use `addsvl` to allocate any scalable stack-slots, which under
+  // most circumstances will be only locals, not callee-save slots.
+  const Function &F = MBB.getParent()->getFunction();
+  bool UseSVL = F.hasFnAttribute("aarch64_pstate_sm_body");
+
   int64_t Bytes, NumPredicateVectors, NumDataVectors;
   AArch64InstrInfo::decomposeStackOffsetForFrameOffsets(
       Offset, Bytes, NumPredicateVectors, NumDataVectors);
 
   // First emit non-scalable frame offsets, or a simple 'mov'.
   if (Bytes || (!Offset && SrcReg != DestReg)) {
     assert((DestReg != AArch64::SP || Bytes % 8 == 0) &&
            "SP increment/decrement not 8-byte aligned");
void llvm::emitFrameOffset(MachineBasicBlock &MBB,
 
   assert(!(SetNZCV && (NumPredicateVectors || NumDataVectors)) &&
          "SetNZCV not supported with SVE vectors");
   assert(!(NeedsWinCFI && (NumPredicateVectors || NumDataVectors)) &&
          "WinCFI not supported with SVE vectors");
 
   if (NumDataVectors) {
     emitFrameOffsetAdj(MBB, MBBI, DL, DestReg, SrcReg, NumDataVectors,
-                       AArch64::ADDVL_XXI, TII, Flag, NeedsWinCFI, nullptr,
-                       EmitCFAOffset, CFAOffset, FrameReg);
+                       UseSVL ? AArch64::ADDSVL_XXI : AArch64::ADDVL_XXI,
+                       TII, Flag, NeedsWinCFI, nullptr, EmitCFAOffset,
+                       CFAOffset, FrameReg);
     CFAOffset += StackOffset::getScalable(-NumDataVectors * 16);
     SrcReg = DestReg;
   }
 
   if (NumPredicateVectors) {
     assert(DestReg != AArch64::SP && "Unaligned access to SP");
     emitFrameOffsetAdj(MBB, MBBI, DL, DestReg, SrcReg, NumPredicateVectors,
-                       AArch64::ADDPL_XXI, TII, Flag, NeedsWinCFI, nullptr,
-                       EmitCFAOffset, CFAOffset, FrameReg);
+                       UseSVL ? AArch64::ADDSPL_XXI : AArch64::ADDPL_XXI,
+                       TII, Flag, NeedsWinCFI, nullptr, EmitCFAOffset,
+                       CFAOffset, FrameReg);
   }
 }
 
 MachineInstr *AArch64InstrInfo::foldMemoryOperandImpl(
     MachineFunction &MF, MachineInstr &MI, ArrayRef<unsigned> Ops,
     MachineBasicBlock::iterator InsertPt, int FrameIndex,
     LiveIntervals *LIS, VirtRegMap *VRM) const {
   // This is a bit of a hack. Consider this instruction:

llvm/test/CodeGen/AArch64/sme-get-pstatesm.ll

 ; CHECK-NEXT:    ret
   %pstate = call i64 @llvm.aarch64.sme.get.pstatesm()
   ret i64 %pstate
 }
 
 define i64 @get_pstatesm_locally_streaming() nounwind "aarch64_pstate_sm_body" {
 ; CHECK-LABEL: get_pstatesm_locally_streaming:
 ; CHECK:       // %bb.0:
+; CHECK-NEXT:    stp d15, d14, [sp, #-64]! // 16-byte Folded Spill
+; CHECK-NEXT:    stp d13, d12, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT:    stp d11, d10, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEXT:    stp d9, d8, [sp, #48] // 16-byte Folded Spill
+; CHECK-NEXT:    smstart sm
+; CHECK-NEXT:    smstop sm
+; CHECK-NEXT:    ldp d9, d8, [sp, #48] // 16-byte Folded Reload
 ; CHECK-NEXT:    mov w0, #1
+; CHECK-NEXT:    ldp d11, d10, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d13, d12, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d15, d14, [sp], #64 // 16-byte Folded Reload
 ; CHECK-NEXT:    ret
   %pstate = call i64 @llvm.aarch64.sme.get.pstatesm()
   ret i64 %pstate
 }
 
 define i64 @get_pstatesm_streaming_compatible() nounwind "aarch64_pstate_sm_compatible" {
 ; CHECK-LABEL: get_pstatesm_streaming_compatible:
 ; CHECK:       // %bb.0:

llvm/test/CodeGen/AArch64/sme-streaming-body.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sme -start-after=simplifycfg -enable-tail-merge=false  -verify-machineinstrs < %s | FileCheck %s
+
+declare void @normal_callee();
+declare void @streaming_callee() "aarch64_pstate_sm_enabled";
+declare void @streaming_compatible_callee() "aarch64_pstate_sm_compatible";
+
+define void @locally_streaming_caller_streaming_callee() "aarch64_pstate_sm_body" nounwind {
+; CHECK-LABEL: locally_streaming_caller_streaming_callee:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    stp d15, d14, [sp, #-80]! // 16-byte Folded Spill
+; CHECK-NEXT:    stp d13, d12, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT:    stp d11, d10, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEXT:    stp d9, d8, [sp, #48] // 16-byte Folded Spill
+; CHECK-NEXT:    str x30, [sp, #64] // 8-byte Folded Spill
+; CHECK-NEXT:    smstart sm
+; CHECK-NEXT:    bl streaming_compatible_callee
+; CHECK-NEXT:    bl streaming_compatible_callee
+; CHECK-NEXT:    smstop sm
+; CHECK-NEXT:    ldp d9, d8, [sp, #48] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d11, d10, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d13, d12, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr x30, [sp, #64] // 8-byte Folded Reload
+; CHECK-NEXT:    ldp d15, d14, [sp], #80 // 16-byte Folded Reload
+; CHECK-NEXT:    ret
+
+  call void @streaming_compatible_callee();
+  call void @streaming_compatible_callee();
+  ret void;
+}
+
+; Test that a streaming body and streaming interface, no smstart/smstop are emitted,
+; because the function already is in streaming mode upon entry.
+define void @streaming_and_locally_streaming_caller_streaming_callee() "aarch64_pstate_sm_enabled" "aarch64_pstate_sm_body" nounwind {
+; CHECK-LABEL: streaming_and_locally_streaming_caller_streaming_callee:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-NEXT:    bl streaming_callee
+; CHECK-NEXT:    bl streaming_callee
+; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-NEXT:    ret
+  call void @streaming_callee();
+  call void @streaming_callee();
+  ret void;
+}
+
+define void @locally_streaming_multiple_exit(i64 %cond) "aarch64_pstate_sm_body" nounwind {
+; CHECK-LABEL: locally_streaming_multiple_exit:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    stp d15, d14, [sp, #-64]! // 16-byte Folded Spill
+; CHECK-NEXT:    stp d13, d12, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT:    stp d11, d10, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEXT:    stp d9, d8, [sp, #48] // 16-byte Folded Spill
+; CHECK-NEXT:    smstart sm
+; CHECK-NEXT:    cmp x0, #1
+; CHECK-NEXT:    b.ne .LBB2_2
+; CHECK-NEXT:  // %bb.1: // %if.else
+; CHECK-NEXT:    smstop sm
+; CHECK-NEXT:    ldp d9, d8, [sp, #48] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d11, d10, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d13, d12, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d15, d14, [sp], #64 // 16-byte Folded Reload
+; CHECK-NEXT:    ret
+; CHECK-NEXT:  .LBB2_2: // %if.end
+; CHECK-NEXT:    smstop sm
+; CHECK-NEXT:    ldp d9, d8, [sp, #48] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d11, d10, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d13, d12, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d15, d14, [sp], #64 // 16-byte Folded Reload
+; CHECK-NEXT:    ret
+
+entry:
+  %tobool = icmp eq i64 %cond, 1
+  br i1 %tobool, label %if.else, label %if.end
+
+if.else:
+  ret void;
+
+if.end:
+  ret void;
+}
+
+; Do a fixed-width vector add on a NEON vector.
+; This tests that:
+; * Incoming vector in v0.d isn't clobbered by the change in streaming mode.
+; * Result vector is correctly preserved after smstop.
+define <2 x i64> @locally_streaming_caller_no_callee(<2 x i64> %a) "aarch64_pstate_sm_body" nounwind {
+; CHECK-LABEL: locally_streaming_caller_no_callee:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #80
+; CHECK-NEXT:    stp d15, d14, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT:    stp d13, d12, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEXT:    stp d11, d10, [sp, #48] // 16-byte Folded Spill
+; CHECK-NEXT:    stp d9, d8, [sp, #64] // 16-byte Folded Spill
+; CHECK-NEXT:    str q0, [sp] // 16-byte Folded Spill
+; CHECK-NEXT:    smstart sm
+; CHECK-NEXT:    adrp x8, .LCPI3_0
+; CHECK-NEXT:    ldr q1, [sp] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr q0, [x8, :lo12:.LCPI3_0]
+; CHECK-NEXT:    add v0.2d, v1.2d, v0.2d
+; CHECK-NEXT:    str q0, [sp] // 16-byte Folded Spill
+; CHECK-NEXT:    smstop sm
+; CHECK-NEXT:    ldp d9, d8, [sp, #64] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d11, d10, [sp, #48] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d13, d12, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d15, d14, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr q0, [sp] // 16-byte Folded Reload
+; CHECK-NEXT:    add sp, sp, #80
+; CHECK-NEXT:    ret
+
+  %add = add <2 x i64> %a, <i64 41, i64 42>;
+  ret <2 x i64> %add;
+}
+
+; Test that we use the interface (not the function's body) to determine what
+; streaming-mode to enter the callee. In this case the interface is normal, so
+; pstate.sm must be 0 on entry and is 0 upon return from the callee.
+define void @locally_streaming_caller_locally_streaming_callee() "aarch64_pstate_sm_body" nounwind {
+; CHECK-LABEL: locally_streaming_caller_locally_streaming_callee:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    stp d15, d14, [sp, #-80]! // 16-byte Folded Spill
+; CHECK-NEXT:    stp d13, d12, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT:    stp d11, d10, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEXT:    stp d9, d8, [sp, #48] // 16-byte Folded Spill
+; CHECK-NEXT:    str x30, [sp, #64] // 8-byte Folded Spill
+; CHECK-NEXT:    smstart sm
+; CHECK-NEXT:    smstop sm
+; CHECK-NEXT:    bl locally_streaming_caller_streaming_callee
+; CHECK-NEXT:    smstart sm
+; CHECK-NEXT:    smstop sm
+; CHECK-NEXT:    ldp d9, d8, [sp, #48] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d11, d10, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d13, d12, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr x30, [sp, #64] // 8-byte Folded Reload
+; CHECK-NEXT:    ldp d15, d14, [sp], #80 // 16-byte Folded Reload
+; CHECK-NEXT:    ret
+
+  call void @locally_streaming_caller_streaming_callee();
+  ret void;
+}
+
+;
+; Test that a locally streaming function correctly retains the
+; argument/result registers, because smstart/smstop instructions that are
+; inserted to implement the arm_locally_streaming attribute thrashes the
+; vector register contents.
+;
+
+define <2 x i64> @locally_streaming_caller_compatible_callee_vec_args_ret(<2 x i64> %a) "aarch64_pstate_sm_body" nounwind {
+; CHECK-LABEL: locally_streaming_caller_compatible_callee_vec_args_ret:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #96
+; CHECK-NEXT:    stp d15, d14, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT:    stp d13, d12, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEXT:    stp d11, d10, [sp, #48] // 16-byte Folded Spill
+; CHECK-NEXT:    stp d9, d8, [sp, #64] // 16-byte Folded Spill
+; CHECK-NEXT:    str x30, [sp, #80] // 8-byte Folded Spill
+; CHECK-NEXT:    str q0, [sp] // 16-byte Folded Spill
+; CHECK-NEXT:    smstart sm
+; CHECK-NEXT:    ldr q0, [sp] // 16-byte Folded Reload
+; CHECK-NEXT:    bl streaming_compatible_callee_vec_args_ret
+; CHECK-NEXT:    str q0, [sp] // 16-byte Folded Spill
+; CHECK-NEXT:    smstop sm
+; CHECK-NEXT:    ldp d9, d8, [sp, #64] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d11, d10, [sp, #48] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d13, d12, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d15, d14, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr q0, [sp] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr x30, [sp, #80] // 8-byte Folded Reload
+; CHECK-NEXT:    add sp, sp, #96
+; CHECK-NEXT:    ret
+  %res = call <2 x i64> @streaming_compatible_callee_vec_args_ret(<2 x i64> %a) "aarch64_pstate_sm_compatible"
+  ret <2 x i64> %res;
+}
+
+declare <2 x i64> @streaming_compatible_callee_vec_args_ret(<2 x i64>) "aarch64_pstate_sm_compatible"
+
+define {<2 x i64>, <2 x i64>} @locally_streaming_caller_compatible_callee_struct_arg_ret({<2 x i64>, <2 x i64>} %arg) "aarch64_pstate_sm_body" nounwind {
+; CHECK-LABEL: locally_streaming_caller_compatible_callee_struct_arg_ret:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #112
+; CHECK-NEXT:    stp d15, d14, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEXT:    stp d13, d12, [sp, #48] // 16-byte Folded Spill
+; CHECK-NEXT:    stp d11, d10, [sp, #64] // 16-byte Folded Spill
+; CHECK-NEXT:    stp d9, d8, [sp, #80] // 16-byte Folded Spill
+; CHECK-NEXT:    str x30, [sp, #96] // 8-byte Folded Spill
+; CHECK-NEXT:    str q1, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT:    smstart sm
+; CHECK-NEXT:    ldr q0, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    bl streaming_compatible_callee_vec_arg_struct_ret
+; CHECK-NEXT:    stp q1, q0, [sp] // 32-byte Folded Spill
+; CHECK-NEXT:    smstop sm
+; CHECK-NEXT:    ldp q1, q0, [sp] // 32-byte Folded Reload
+; CHECK-NEXT:    ldp d9, d8, [sp, #80] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d11, d10, [sp, #64] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d13, d12, [sp, #48] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d15, d14, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr x30, [sp, #96] // 8-byte Folded Reload
+; CHECK-NEXT:    add sp, sp, #112
+; CHECK-NEXT:    ret
+  %v1.arg = extractvalue {<2 x i64>, <2 x i64>} %arg, 1
+  %res = call {<2 x i64>, <2 x i64>} @streaming_compatible_callee_vec_arg_struct_ret(<2 x i64> %v1.arg) "aarch64_pstate_sm_compatible"
+  ret {<2 x i64>, <2 x i64>} %res;
+}
+
+declare {<2 x i64>, <2 x i64>} @streaming_compatible_callee_vec_arg_struct_ret(<2 x i64>) "aarch64_pstate_sm_compatible"
+
+; Test that we use `addsvl` for allocating any stack space for locals before `smstart`,
+; such that the correct amount of stack space is allocated.
+define void @locally_streaming_caller_alloca() nounwind "aarch64_pstate_sm_body" {
+; CHECK-LABEL: locally_streaming_caller_alloca:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    stp d15, d14, [sp, #-80]! // 16-byte Folded Spill
+; CHECK-NEXT:    stp d13, d12, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT:    stp d11, d10, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEXT:    stp d9, d8, [sp, #48] // 16-byte Folded Spill
+; CHECK-NEXT:    stp x29, x30, [sp, #64] // 16-byte Folded Spill
+; CHECK-NEXT:    addsvl sp, sp, #-1
+; CHECK-NEXT:    smstart sm
+; CHECK-NEXT:    mov x0, sp
+; CHECK-NEXT:    bl use_ptr
+; CHECK-NEXT:    smstop sm
+; CHECK-NEXT:    addsvl sp, sp, #1
+; CHECK-NEXT:    ldp x29, x30, [sp, #64] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d9, d8, [sp, #48] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d11, d10, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d13, d12, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d15, d14, [sp], #80 // 16-byte Folded Reload
+; CHECK-NEXT:    ret
+  %alloca = alloca <vscale x 4 x i32>
+  call void @use_ptr(ptr %alloca) "aarch64_pstate_sm_compatible"
+  ret void
+}
+
+declare void @use_ptr(ptr) "aarch64_pstate_sm_compatible"
+
+define double @call_to_intrinsic_without_chain(double %x) nounwind "aarch64_pstate_sm_body" {
+; CHECK-LABEL: call_to_intrinsic_without_chain:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    stp d15, d14, [sp, #-80]! // 16-byte Folded Spill
+; CHECK-NEXT:    stp d13, d12, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT:    stp d11, d10, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEXT:    stp d9, d8, [sp, #48] // 16-byte Folded Spill
+; CHECK-NEXT:    str x30, [sp, #64] // 8-byte Folded Spill
+; CHECK-NEXT:    str d0, [sp, #72] // 8-byte Folded Spill
+; CHECK-NEXT:    smstart sm
+; CHECK-NEXT:    smstop sm
+; CHECK-NEXT:    ldr d0, [sp, #72] // 8-byte Folded Reload
+; CHECK-NEXT:    bl cos
+; CHECK-NEXT:    str d0, [sp, #72] // 8-byte Folded Spill
+; CHECK-NEXT:    smstart sm
+; CHECK-NEXT:    smstop sm
+; CHECK-NEXT:    ldp d9, d8, [sp, #48] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d11, d10, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp d13, d12, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr d0, [sp, #72] // 8-byte Folded Reload
+; CHECK-NEXT:    ldr x30, [sp, #64] // 8-byte Folded Reload
+; CHECK-NEXT:    ldp d15, d14, [sp], #80 // 16-byte Folded Reload
+; CHECK-NEXT:    ret
+entry:
+  %0 = call fast double @llvm.cos.f64(double %x)
+  ret double %0
+}
+
+declare double @llvm.cos.f64(double)

llvm/test/CodeGen/AArch64/sve-fixed-length-frame-offests.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -debug-only=isel < %s 2>&1 | FileCheck %s
 
 ; REQUIRES: asserts
 
 target triple = "aarch64-unknown-linux-gnu"
 
 ; Ensure that only no offset frame indexes are folded into SVE load/stores when
 ; accessing fixed width objects.
 define void @foo(<8 x i64>* %a) #0 {
 ; CHECK-LABEL: foo:
 ; CHECK:       SelectionDAG has 14 nodes:
-; CHECK-NEXT:    t0: ch = EntryToken
+; CHECK-NEXT:    t0: ch,glue = EntryToken
 ; CHECK-NEXT:    t12: nxv2i1 = PTRUE_D TargetConstant:i32<31>
 ; CHECK-NEXT:    t2: i64,ch = CopyFromReg t0, Register:i64 %0
 ; CHECK-NEXT:    t18: nxv2i64,ch = LD1D_IMM<Mem:(volatile load (s512) from %ir.a)> t12, t2, TargetConstant:i64<0>, t0
 ; CHECK-NEXT:    t8: i64 = ADDXri TargetFrameIndex:i64<1>, TargetConstant:i32<0>, TargetConstant:i32<0>
 ; CHECK-NEXT:    t17: ch = ST1D_IMM<Mem:(volatile store (s512) into %ir.r0)> t18, t12, TargetFrameIndex:i64<0>, TargetConstant:i64<0>, t18:1
 ; CHECK-NEXT:    t16: ch = ST1D_IMM<Mem:(volatile store (s512) into %ir.r1)> t18, t12, t8, TargetConstant:i64<0>, t17
 ; CHECK-NEXT:    t10: ch = RET_ReallyLR t16
 ; CHECK-EMPTY:

llvm/test/CodeGen/AMDGPU/sdag-print-divergence.ll

 ; RUN: llc -march=amdgcn -mcpu=gfx900 -O0 -verify-machineinstrs < %s -debug-only=isel 2>&1 | FileCheck --check-prefixes=GCN,GCN-DEFAULT %s
 ; RUN: llc -march=amdgcn -mcpu=gfx900 -O0 -verify-machineinstrs < %s -debug-only=isel -dag-dump-verbose 2>&1 | FileCheck --check-prefixes=GCN,GCN-VERBOSE %s
 
 ; REQUIRES: asserts
 
 ; GCN-LABEL: === test_sdag_dump
 ; GCN: Initial selection DAG: %bb.0 'test_sdag_dump:entry'
 ; GCN: SelectionDAG has 10 nodes:
 
-; GCN-DEFAULT:  t0: ch = EntryToken
+; GCN-DEFAULT:  t0: ch,glue = EntryToken
 ; GCN-DEFAULT:  t2: f32,ch = CopyFromReg t0, Register:f32 %0
 ; GCN-DEFAULT:      t5: f32 = fadd t2, t2
 ; GCN-DEFAULT:      t4: f32,ch = CopyFromReg # D:1 t0, Register:f32 %1
 ; GCN-DEFAULT:    t6: f32 = fadd # D:1 t5, t4
 ; GCN-DEFAULT:  t8: ch,glue = CopyToReg # D:1 t0, Register:f32 $vgpr0, t6
 ; GCN-DEFAULT:  t9: ch = RETURN_TO_EPILOG # D:1 t8, Register:f32 $vgpr0, t8:1
 
-; GCN-VERBOSE:  t0: ch = EntryToken # D:0
+; GCN-VERBOSE:  t0: ch,glue = EntryToken # D:0
 ; GCN-VERBOSE:  t2: f32,ch = CopyFromReg [ORD=1] # D:0 t0, Register:f32 %0 # D:0
 ; GCN-VERBOSE:      t5: f32 = fadd [ORD=2] # D:0 t2, t2
 ; GCN-VERBOSE:      t4: f32,ch = CopyFromReg [ORD=1] # D:1 t0, Register:f32 %1 # D:0
 ; GCN-VERBOSE:    t6: f32 = fadd [ORD=3] # D:1 t5, t4
 ; GCN-VERBOSE:  t8: ch,glue = CopyToReg [ORD=4] # D:1 t0, Register:f32 $vgpr0 # D:0, t6
 ; GCN-VERBOSE:  t9: ch = RETURN_TO_EPILOG [ORD=4] # D:1 t8, Register:f32 $vgpr0 # D:0, t8:1
 
 define amdgpu_ps float @test_sdag_dump(float inreg %scalar, float %vector)  {
 entry:
   %sadd = fadd float %scalar, %scalar
   %ret = fadd float %sadd, %vector
   ret float %ret
 }
 
 declare i32 @llvm.amdgcn.workitem.id.x()

llvm/test/CodeGen/X86/callbr-asm-bb-exports.ll

 ; REQUIRES: asserts
 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -o /dev/null -debug-only=isel 2>&1 | FileCheck %s
 
 ; Make sure we emit the basic block exports and the TokenFactor before the
 ; inlineasm_br. Not sure how to get a MachineIR change so this reads the debug
 ; output from SelectionDAG.
 
-; CHECK: t0: ch = EntryToken
+; CHECK: t0: ch,glue = EntryToken
 ; CHECK-NEXT: t4: i32,ch = CopyFromReg t0, Register:i32 %3
 ; CHECK-NEXT: t10: i32 = add t4, Constant:i32<1>
 ; CHECK-NEXT: t12: ch = CopyToReg t0, Register:i32 %0, t10
 ; CHECK-NEXT: t6: i32,ch = CopyFromReg t0, Register:i32 %4
 ; CHECK-NEXT: t13: i32 = add t6, Constant:i32<1>
 ; CHECK-NEXT: t15: ch = CopyToReg t0, Register:i32 %1, t13
 ; CHECK-NEXT: t17: ch = TokenFactor t12, t15
 ; CHECK-NEXT: t2: i32,ch = CopyFromReg t0, Register:i32 %2

llvm/test/CodeGen/X86/merge-store-partially-alias-loads.ll

 ; REQUIRES: asserts
 ; RUN: llc -mtriple=x86_64-unknown-linux-gnu < %s | FileCheck -check-prefix=X86 %s
 ; RUN: llc -mtriple=x86_64-unknown-linux-gnu -debug-only=isel < %s -o /dev/null 2>&1 | FileCheck -check-prefix=DBGDAG %s
 
 ; It's OK to merge the load / store of the first 2 components, but
 ; they must not be placed on the same chain after merging.
 
 ; X86-LABEL: {{^}}merge_store_partial_overlap_load:
 ; X86-DAG: movzwl ([[BASEREG:%[a-z]+]]), %e[[LO2:[a-z]+]]
 ; X86-DAG: movzbl 2([[BASEREG]]), %e[[HI1:[a-z]]]
 
 ; X86-NEXT: movw %[[LO2]], 1([[BASEREG]])
 ; X86-NEXT: movb %[[HI1]]l, 3([[BASEREG]])
 ; X86-NEXT: retq
 
 ; DBGDAG-LABEL: Optimized legalized selection DAG: %bb.0 'merge_store_partial_overlap_load:'
-; DBGDAG: [[ENTRYTOKEN:t[0-9]+]]: ch = EntryToken
+; DBGDAG: [[ENTRYTOKEN:t[0-9]+]]: ch,glue = EntryToken
 ; DBGDAG-DAG: [[BASEPTR:t[0-9]+]]: i64,ch = CopyFromReg [[ENTRYTOKEN]],
 ; DBGDAG-DAG: [[ADDPTR:t[0-9]+]]: i64 = add {{(nuw )?}}[[BASEPTR]], Constant:i64<2>
 
 ; DBGDAG-DAG: [[LD2:t[0-9]+]]: i16,ch = load<(load (s16) from %ir.tmp81, align 1)> [[ENTRYTOKEN]], [[BASEPTR]], undef:i64
 ; DBGDAG-DAG: [[LD1:t[0-9]+]]: i8,ch = load<(load (s8) from %ir.tmp12)> [[ENTRYTOKEN]], [[ADDPTR]], undef:i64
 
 ; DBGDAG-DAG: [[ST1:t[0-9]+]]: ch = store<(store (s8) into %ir.tmp14)> [[ENTRYTOKEN]], [[LD1]], t{{[0-9]+}}, undef:i64
 ; DBGDAG-DAG: [[LOADTOKEN:t[0-9]+]]: ch = TokenFactor [[LD2]]:1, [[LD1]]:1

llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/amdgpu_isel.ll.expected

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=amdgcn-amd-amdhsa -stop-after=finalize-isel -debug-only=isel -o /dev/null %s 2>&1 | FileCheck %s
 
 define i64 @i64_test(i64 %i) nounwind readnone {
 ; CHECK-LABEL: i64_test:
 ; CHECK:       SelectionDAG has 9 nodes:
-; CHECK-NEXT:    t0: ch = EntryToken
+; CHECK-NEXT:    t0: ch,glue = EntryToken
 ; CHECK-NEXT:    t11: ch,glue = CopyToReg t0, Register:i32 $vgpr0, IMPLICIT_DEF:i32
 ; CHECK-NEXT:    t17: i32 = V_MOV_B32_e32 TargetConstant:i32<0>
 ; CHECK-NEXT:    t13: ch,glue = CopyToReg t11, Register:i32 $vgpr1, t17, t11:1
 ; CHECK-NEXT:    t14: ch = SI_RETURN Register:i32 $vgpr0, Register:i32 $vgpr1, t13, t13:1
 ; CHECK-EMPTY:
   %loc = alloca i64
   %j = load i64, i64 * %loc
   %r = add i64 %i, %j
   ret i64 %r
 }
 
 define i64 @i32_test(i32 %i) nounwind readnone {
 ; CHECK-LABEL: i32_test:
 ; CHECK:       SelectionDAG has 8 nodes:
 ; CHECK-NEXT:    t5: i32 = V_MOV_B32_e32 TargetConstant:i32<0>
-; CHECK-NEXT:    t0: ch = EntryToken
+; CHECK-NEXT:    t0: ch,glue = EntryToken
 ; CHECK-NEXT:    t7: ch,glue = CopyToReg t0, Register:i32 $vgpr0, t5
 ; CHECK-NEXT:    t9: ch,glue = CopyToReg t7, Register:i32 $vgpr1, t5, t7:1
 ; CHECK-NEXT:    t10: ch = SI_RETURN Register:i32 $vgpr0, Register:i32 $vgpr1, t9, t9:1
 ; CHECK-EMPTY:
   %loc = alloca i32
   %j = load i32, i32 * %loc
   %r = add i32 %i, %j
   %ext = zext i32 %r to i64
   ret i64 %ext
 }
 
 define i64 @i16_test(i16 %i) nounwind readnone {
 ; CHECK-LABEL: i16_test:
 ; CHECK:       SelectionDAG has 8 nodes:
 ; CHECK-NEXT:    t5: i32 = V_MOV_B32_e32 TargetConstant:i32<0>
-; CHECK-NEXT:    t0: ch = EntryToken
+; CHECK-NEXT:    t0: ch,glue = EntryToken
 ; CHECK-NEXT:    t7: ch,glue = CopyToReg t0, Register:i32 $vgpr0, t5
 ; CHECK-NEXT:    t9: ch,glue = CopyToReg t7, Register:i32 $vgpr1, t5, t7:1
 ; CHECK-NEXT:    t10: ch = SI_RETURN Register:i32 $vgpr0, Register:i32 $vgpr1, t9, t9:1
 ; CHECK-EMPTY:
   %loc = alloca i16
   %j = load i16, i16 * %loc
   %r = add i16 %i, %j
   %ext = zext i16 %r to i64
   ret i64 %ext
 }
 
 define i64 @i8_test(i8 %i) nounwind readnone {
 ; CHECK-LABEL: i8_test:
 ; CHECK:       SelectionDAG has 8 nodes:
 ; CHECK-NEXT:    t5: i32 = V_MOV_B32_e32 TargetConstant:i32<0>
-; CHECK-NEXT:    t0: ch = EntryToken
+; CHECK-NEXT:    t0: ch,glue = EntryToken
 ; CHECK-NEXT:    t7: ch,glue = CopyToReg t0, Register:i32 $vgpr0, t5
 ; CHECK-NEXT:    t9: ch,glue = CopyToReg t7, Register:i32 $vgpr1, t5, t7:1
 ; CHECK-NEXT:    t10: ch = SI_RETURN Register:i32 $vgpr0, Register:i32 $vgpr1, t9, t9:1
 ; CHECK-EMPTY:
   %loc = alloca i8
   %j = load i8, i8 * %loc
   %r = add i8 %i, %j
   %ext = zext i8 %r to i64
   ret i64 %ext
 }

llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/lanai_isel.ll.expected

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=lanai -stop-after=finalize-isel -debug-only=isel -o /dev/null %s 2>&1 | FileCheck %s
 
 define i64 @i64_test(i64 %i) nounwind readnone {
 ; CHECK-LABEL: i64_test:
 ; CHECK:       SelectionDAG has 22 nodes:
-; CHECK-NEXT:    t0: ch = EntryToken
+; CHECK-NEXT:    t0: ch,glue = EntryToken
 ; CHECK-NEXT:    t5: i32,ch = LDW_RI<Mem:(load (s32) from %fixed-stack.0)> TargetFrameIndex:i32<-2>, TargetConstant:i32<0>, TargetConstant:i32<0>, t0
 ; CHECK-NEXT:    t7: i32 = ADD_I_LO TargetFrameIndex:i32<0>, TargetConstant:i32<0>
 ; CHECK-NEXT:    t29: i32 = OR_I_LO t7, TargetConstant:i32<4>
 ; CHECK-NEXT:    t22: i32,ch = LDW_RI<Mem:(dereferenceable load (s32) from %ir.loc + 4, basealign 8)> t29, TargetConstant:i32<0>, TargetConstant:i32<0>, t0
 ; CHECK-NEXT:    t24: i32 = ADD_R t5, t22, TargetConstant:i32<0>
 ; CHECK-NEXT:    t3: i32,ch = LDW_RI<Mem:(load (s32) from %fixed-stack.1, align 8)> TargetFrameIndex:i32<-1>, TargetConstant:i32<0>, TargetConstant:i32<0>, t0
 ; CHECK-NEXT:    t19: i32,ch = LDW_RI<Mem:(dereferenceable load (s32) from %ir.loc, align 8)> TargetFrameIndex:i32<0>, TargetConstant:i32<0>, TargetConstant:i32<0>, t0
 ; CHECK-NEXT:    t25: i32 = ADD_R t3, t19, TargetConstant:i32<0>
 ; CHECK-NEXT:    t30: i32,glue = SFSUB_F_RR t24, t5
 ; CHECK-NEXT:    t31: i32 = SCC TargetConstant:i32<4>, t30:1
 ; CHECK-NEXT:    t28: i32 = ADD_R t25, t31, TargetConstant:i32<0>
 ; CHECK-NEXT:    t15: ch,glue = CopyToReg t0, Register:i32 $rv, t28
 ; CHECK-NEXT:    t17: ch,glue = CopyToReg t15, Register:i32 $r9, t24, t15:1
 ; CHECK-NEXT:    t18: ch = RET Register:i32 $rv, Register:i32 $r9, t17, t17:1
 ; CHECK-EMPTY:
   %loc = alloca i64
   %j = load i64, i64 * %loc
   %r = add i64 %i, %j
   ret i64 %r
 }
 
 define i64 @i32_test(i32 %i) nounwind readnone {
 ; CHECK-LABEL: i32_test:
 ; CHECK:       SelectionDAG has 14 nodes:
-; CHECK-NEXT:    t0: ch = EntryToken
+; CHECK-NEXT:    t0: ch,glue = EntryToken
 ; CHECK-NEXT:    t21: i32,ch = CopyFromReg t0, Register:i32 $r0
 ; CHECK-NEXT:    t13: ch,glue = CopyToReg t0, Register:i32 $rv, t21
 ; CHECK-NEXT:    t3: i32,ch = LDW_RI<Mem:(load (s32) from %fixed-stack.0, align 8)> TargetFrameIndex:i32<-1>, TargetConstant:i32<0>, TargetConstant:i32<0>, t0
 ; CHECK-NEXT:    t6: i32,ch = LDW_RI<Mem:(dereferenceable load (s32) from %ir.loc)> TargetFrameIndex:i32<0>, TargetConstant:i32<0>, TargetConstant:i32<0>, t0
 ; CHECK-NEXT:    t7: i32 = ADD_R t3, t6, TargetConstant:i32<0>
 ; CHECK-NEXT:    t15: ch,glue = CopyToReg t13, Register:i32 $r9, t7, t13:1
 ; CHECK-NEXT:    t16: ch = RET Register:i32 $rv, Register:i32 $r9, t15, t15:1
 ; CHECK-EMPTY:
   %loc = alloca i32
   %j = load i32, i32 * %loc
   %r = add i32 %i, %j
   %ext = zext i32 %r to i64
   ret i64 %ext
 }
 
 define i64 @i16_test(i16 %i) nounwind readnone {
 ; CHECK-LABEL: i16_test:
 ; CHECK:       SelectionDAG has 19 nodes:
-; CHECK-NEXT:    t0: ch = EntryToken
+; CHECK-NEXT:    t0: ch,glue = EntryToken
 ; CHECK-NEXT:    t33: i32,ch = CopyFromReg t0, Register:i32 $r0
 ; CHECK-NEXT:    t14: ch,glue = CopyToReg t0, Register:i32 $rv, t33
 ; CHECK-NEXT:    t1: i32 = ADD_I_LO TargetFrameIndex:i32<-1>, TargetConstant:i32<0>
 ; CHECK-NEXT:    t21: i32 = OR_I_LO t1, TargetConstant:i32<2>
 ; CHECK-NEXT:    t23: i32,ch = LDHz_RI<Mem:(load (s16) from %fixed-stack.0 + 2)> t21, TargetConstant:i32<0>, TargetConstant:i32<0>, t0
 ; CHECK-NEXT:    t22: i32,ch = LDHz_RI<Mem:(dereferenceable load (s16) from %ir.loc)> TargetFrameIndex:i32<0>, TargetConstant:i32<0>, TargetConstant:i32<0>, t0
 ; CHECK-NEXT:    t24: i32 = ADD_R t23, t22, TargetConstant:i32<0>
 ; CHECK-NEXT:    t27: i32 = AND_I_HI t24, TargetConstant:i32<0>
 ; CHECK-NEXT:    t16: ch,glue = CopyToReg t14, Register:i32 $r9, t27, t14:1
 ; CHECK-NEXT:    t28: i32 = TargetConstant<65535>
 ; CHECK-NEXT:    t17: ch = RET Register:i32 $rv, Register:i32 $r9, t16, t16:1
 ; CHECK-EMPTY:
   %loc = alloca i16
   %j = load i16, i16 * %loc
   %r = add i16 %i, %j
   %ext = zext i16 %r to i64
   ret i64 %ext
 }
 
 define i64 @i8_test(i8 %i) nounwind readnone {
 ; CHECK-LABEL: i8_test:
 ; CHECK:       SelectionDAG has 20 nodes:
-; CHECK-NEXT:    t0: ch = EntryToken
+; CHECK-NEXT:    t0: ch,glue = EntryToken
 ; CHECK-NEXT:    t33: i32,ch = CopyFromReg t0, Register:i32 $r0
 ; CHECK-NEXT:    t14: ch,glue = CopyToReg t0, Register:i32 $rv, t33
 ; CHECK-NEXT:    t1: i32 = ADD_I_LO TargetFrameIndex:i32<-1>, TargetConstant:i32<0>
 ; CHECK-NEXT:    t21: i32 = OR_I_LO t1, TargetConstant:i32<3>
 ; CHECK-NEXT:    t23: i32,ch = LDBz_RI<Mem:(load (s8) from %fixed-stack.0 + 3)> t21, TargetConstant:i32<0>, TargetConstant:i32<0>, t0
 ; CHECK-NEXT:    t22: i32,ch = LDBz_RI<Mem:(dereferenceable load (s8) from %ir.loc)> TargetFrameIndex:i32<0>, TargetConstant:i32<0>, TargetConstant:i32<0>, t0
 ; CHECK-NEXT:    t24: i32 = ADD_R t23, t22, TargetConstant:i32<0>
 ; CHECK-NEXT:    t26: i32 = SLI TargetConstant:i32<255>

llvm/test/tools/UpdateTestChecks/update_llc_test_checks/Inputs/x86_isel.ll.expected

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=x86_64 -stop-after=finalize-isel -debug-only=isel -o /dev/null %s 2>&1 | FileCheck %s --check-prefix=PIC
 ; RUN: llc -mtriple=x86_64-windows -stop-after=finalize-isel -debug-only=isel -o /dev/null %s 2>&1 | FileCheck %s --check-prefix=WIN
 
 define i64 @i64_test(i64 %i) nounwind readnone {
 ; PIC-LABEL: i64_test:
 ; PIC:       SelectionDAG has 12 nodes:
-; PIC-NEXT:    t0: ch = EntryToken
+; PIC-NEXT:    t0: ch,glue = EntryToken
 ; PIC-NEXT:    t2: i64,ch = CopyFromReg t0, Register:i64 %0
 ; PIC-NEXT:    t7: i64,i32,ch = ADD64rm<Mem:(dereferenceable load (s64) from %ir.loc)> t2, TargetFrameIndex:i64<0>, TargetConstant:i8<1>, Register:i64 $noreg, TargetConstant:i32<0>, Register:i16 $noreg, t0
 ; PIC-NEXT:    t10: ch,glue = CopyToReg t0, Register:i64 $rax, t7
 ; PIC-NEXT:    t11: ch = RET TargetConstant:i32<0>, Register:i64 $rax, t10, t10:1
 ; PIC-EMPTY:
 ;
 ; WIN-LABEL: i64_test:
 ; WIN:       SelectionDAG has 12 nodes:
-; WIN-NEXT:    t0: ch = EntryToken
+; WIN-NEXT:    t0: ch,glue = EntryToken
 ; WIN-NEXT:    t2: i64,ch = CopyFromReg t0, Register:i64 %0
 ; WIN-NEXT:    t7: i64,i32,ch = ADD64rm<Mem:(dereferenceable load (s64) from %ir.loc)> t2, TargetFrameIndex:i64<0>, TargetConstant:i8<1>, Register:i64 $noreg, TargetConstant:i32<0>, Register:i16 $noreg, t0
 ; WIN-NEXT:    t10: ch,glue = CopyToReg t0, Register:i64 $rax, t7
 ; WIN-NEXT:    t11: ch = RET TargetConstant:i32<0>, Register:i64 $rax, t10, t10:1
 ; WIN-EMPTY:
   %loc = alloca i64
   %j = load i64, i64 * %loc
   %r = add i64 %i, %j
   ret i64 %r
 }
 
 define i64 @i32_test(i32 %i) nounwind readnone {
 ; PIC-LABEL: i32_test:
 ; PIC:       SelectionDAG has 15 nodes:
-; PIC-NEXT:    t0: ch = EntryToken
+; PIC-NEXT:    t0: ch,glue = EntryToken
 ; PIC-NEXT:    t2: i32,ch = CopyFromReg t0, Register:i32 %0
 ; PIC-NEXT:    t7: i32,i32,ch = ADD32rm<Mem:(dereferenceable load (s32) from %ir.loc)> t2, TargetFrameIndex:i64<0>, TargetConstant:i8<1>, Register:i64 $noreg, TargetConstant:i32<0>, Register:i16 $noreg, t0
 ; PIC-NEXT:    t8: i64 = SUBREG_TO_REG TargetConstant:i64<0>, t7, TargetConstant:i32<6>
 ; PIC-NEXT:    t11: ch,glue = CopyToReg t0, Register:i64 $rax, t8
 ; PIC-NEXT:    t12: ch = RET TargetConstant:i32<0>, Register:i64 $rax, t11, t11:1
 ; PIC-EMPTY:
 ;
 ; WIN-LABEL: i32_test:
 ; WIN:       SelectionDAG has 15 nodes:
-; WIN-NEXT:    t0: ch = EntryToken
+; WIN-NEXT:    t0: ch,glue = EntryToken
 ; WIN-NEXT:    t2: i32,ch = CopyFromReg t0, Register:i32 %0
 ; WIN-NEXT:    t7: i32,i32,ch = ADD32rm<Mem:(dereferenceable load (s32) from %ir.loc)> t2, TargetFrameIndex:i64<0>, TargetConstant:i8<1>, Register:i64 $noreg, TargetConstant:i32<0>, Register:i16 $noreg, t0
 ; WIN-NEXT:    t8: i64 = SUBREG_TO_REG TargetConstant:i64<0>, t7, TargetConstant:i32<6>
 ; WIN-NEXT:    t11: ch,glue = CopyToReg t0, Register:i64 $rax, t8
 ; WIN-NEXT:    t12: ch = RET TargetConstant:i32<0>, Register:i64 $rax, t11, t11:1
 ; WIN-EMPTY:
   %loc = alloca i32
   %j = load i32, i32 * %loc
   %r = add i32 %i, %j
   %ext = zext i32 %r to i64
   ret i64 %ext
 }
 
 define i64 @i16_test(i16 %i) nounwind readnone {
 ; PIC-LABEL: i16_test:
 ; PIC:       SelectionDAG has 18 nodes:
-; PIC-NEXT:    t0: ch = EntryToken
+; PIC-NEXT:    t0: ch,glue = EntryToken
 ; PIC-NEXT:    t2: i32,ch = CopyFromReg t0, Register:i32 %0
 ; PIC-NEXT:    t3: i16 = EXTRACT_SUBREG t2, TargetConstant:i32<4>
 ; PIC-NEXT:    t8: i16,i32,ch = ADD16rm<Mem:(dereferenceable load (s16) from %ir.loc)> t3, TargetFrameIndex:i64<0>, TargetConstant:i8<1>, Register:i64 $noreg, TargetConstant:i32<0>, Register:i16 $noreg, t0
 ; PIC-NEXT:    t15: i32 = MOVZX32rr16 t8
 ; PIC-NEXT:    t9: i64 = SUBREG_TO_REG TargetConstant:i64<0>, t15, TargetConstant:i32<6>
 ; PIC-NEXT:    t12: ch,glue = CopyToReg t0, Register:i64 $rax, t9
 ; PIC-NEXT:    t13: ch = RET TargetConstant:i32<0>, Register:i64 $rax, t12, t12:1
 ; PIC-EMPTY:
 ;
 ; WIN-LABEL: i16_test:
 ; WIN:       SelectionDAG has 16 nodes:
-; WIN-NEXT:    t0: ch = EntryToken
+; WIN-NEXT:    t0: ch,glue = EntryToken
 ; WIN-NEXT:    t2: i16,ch = CopyFromReg t0, Register:i16 %0
 ; WIN-NEXT:    t7: i16,i32,ch = ADD16rm<Mem:(dereferenceable load (s16) from %ir.loc)> t2, TargetFrameIndex:i64<0>, TargetConstant:i8<1>, Register:i64 $noreg, TargetConstant:i32<0>, Register:i16 $noreg, t0
 ; WIN-NEXT:    t14: i32 = MOVZX32rr16 t7
 ; WIN-NEXT:    t8: i64 = SUBREG_TO_REG TargetConstant:i64<0>, t14, TargetConstant:i32<6>
 ; WIN-NEXT:    t11: ch,glue = CopyToReg t0, Register:i64 $rax, t8
 ; WIN-NEXT:    t12: ch = RET TargetConstant:i32<0>, Register:i64 $rax, t11, t11:1
 ; WIN-EMPTY:
   %loc = alloca i16
   %j = load i16, i16 * %loc
   %r = add i16 %i, %j
   %ext = zext i16 %r to i64
   ret i64 %ext
 }
 
 define i64 @i8_test(i8 %i) nounwind readnone {
 ; PIC-LABEL: i8_test:
 ; PIC:       SelectionDAG has 18 nodes:
-; PIC-NEXT:    t0: ch = EntryToken
+; PIC-NEXT:    t0: ch,glue = EntryToken
 ; PIC-NEXT:    t2: i32,ch = CopyFromReg t0, Register:i32 %0
 ; PIC-NEXT:    t3: i8 = EXTRACT_SUBREG t2, TargetConstant:i32<1>
 ; PIC-NEXT:    t8: i8,i32,ch = ADD8rm<Mem:(dereferenceable load (s8) from %ir.loc)> t3, TargetFrameIndex:i64<0>, TargetConstant:i8<1>, Register:i64 $noreg, TargetConstant:i32<0>, Register:i16 $noreg, t0
 ; PIC-NEXT:    t15: i32 = MOVZX32rr8 t8
 ; PIC-NEXT:    t9: i64 = SUBREG_TO_REG TargetConstant:i64<0>, t15, TargetConstant:i32<6>
 ; PIC-NEXT:    t12: ch,glue = CopyToReg t0, Register:i64 $rax, t9
 ; PIC-NEXT:    t13: ch = RET TargetConstant:i32<0>, Register:i64 $rax, t12, t12:1
 ; PIC-EMPTY:
 ;
 ; WIN-LABEL: i8_test:
 ; WIN:       SelectionDAG has 16 nodes:
-; WIN-NEXT:    t0: ch = EntryToken
+; WIN-NEXT:    t0: ch,glue = EntryToken
 ; WIN-NEXT:    t2: i8,ch = CopyFromReg t0, Register:i8 %0
 ; WIN-NEXT:    t7: i8,i32,ch = ADD8rm<Mem:(dereferenceable load (s8) from %ir.loc)> t2, TargetFrameIndex:i64<0>, TargetConstant:i8<1>, Register:i64 $noreg, TargetConstant:i32<0>, Register:i16 $noreg, t0
 ; WIN-NEXT:    t14: i32 = MOVZX32rr8 t7
 ; WIN-NEXT:    t8: i64 = SUBREG_TO_REG TargetConstant:i64<0>, t14, TargetConstant:i32<6>
 ; WIN-NEXT:    t11: ch,glue = CopyToReg t0, Register:i64 $rax, t8
 ; WIN-NEXT:    t12: ch = RET TargetConstant:i32<0>, Register:i64 $rax, t11, t11:1
 ; WIN-EMPTY:
   %loc = alloca i8
   %j = load i8, i8 * %loc
   %r = add i8 %i, %j
   %ext = zext i8 %r to i64
   ret i64 %ext
 }