[LiveDebugValues] Handle spill locations with a fixed and scalable component.

[sdesmalen]

Summary

This patch fixes the two LiveDebugValues implementations
(InstrRef/VarLoc)Based to handle cases where the StackOffset contains
both a fixed and scalable component.

This depends on the TargetRegisterInfo::prependOffsetExpression being
added in D90020. Feel free to leave comments on that patch if you have them.

[sdesmalen]

Rebased patch

[djtodoro]

llvm/lib/CodeGen/LiveDebugValues/VarLocBasedImpl.cpp

564

missing return false?

llvm/test/CodeGen/AArch64/live-debugvalues-sve.mir

49–53

no need for all these attributes

[sdesmalen]

• Refactored operator== to return false by default.
• Removed unnecessary attributes.

llvm/lib/CodeGen/LiveDebugValues/VarLocBasedImpl.cpp

564

Yes, good catch, thanks!

[djtodoro]

Thanks for addressing the comments.

llvm/lib/CodeGen/LiveDebugValues/InstrRefBasedImpl.cpp

231–234

We use tuples here because std::tie() takes l-values only?

llvm/lib/CodeGen/LiveDebugValues/VarLocBasedImpl.cpp

556

Do we use Loc.Hash for the case of Spill locations?

[jmorse]

Forgive my ignorance on scalable vectors; but is there definitely a scenario where the "Scalable" portion of the stack offset is nonzero, and if so could the test cover it please.

Right now I believe that for aggregate types:

• We can refer to portions of structs if they're split by SROA and promoted from there,
• The same for arrays that are completely promoted,
• All other aggregates are left on the stack, and we only describe their stack address.

As a result, the stack-location tracking in LiveDebugValues is only geared towards tracking the spills / restores of scalars that the register allocator introduces, and sometimes entire vectors. I have a fear that if this patch is trying to make elements within scalable vectors describable to debug-info, it might be aiming to do something that isn't supported.

This probably stems from not understanding scalable vectors; would you have any source examples of when this patch becomes necessary, just so that I can work through them?

[sdesmalen]

In D90046#2437220, @jmorse wrote:

Forgive my ignorance on scalable vectors; but is there definitely a scenario where the "Scalable" portion of the stack offset is nonzero, and if so could the test cover it please.

No worries! In the test-case in this patch the Scalable portion of the offset is actually non-zero:

# CHECK-DAG: ST1W_IMM renamable $z1, killed renamable $p{{[0-9]+}}, $fp, -[[#%u,OFFSET:]]
                                                                            ^^^^^^^^^^^

For this test OFFSET is '1' (so it is stored at FP - 1 * (length of scalable vector)). The 'length of scalable vector' is handled by the addressing mode of ST1W. So for 128bit vectors, it would store to FP - 16bytes, for 256 it would store to FP - 32bytes, etc. The corresponding StackOffset returned by TFI->getFrameIndexReference is {0 bytes, -16 scalable bytes}, and because SVE vectors are always a multiple of 16 bytes, this translates to using -1 for ST1W_IMM.

After the live-debugvalues pass, the debug-location for this value is now described by:

CHECK-DAG: DBG_VALUE $fp, 0, !{{[0-9]+}}, !DIExpression(DW_OP_constu, [[#mul(OFFSET,8)]], DW_OP_bregx, 46, 0, DW_OP_mul, DW_OP_minus),

Which describes the location FP - 8 * VG, where VG is the number of 64-bit 'vector granules'. For an 128bit vector, VG is 2, for 256 it is 4, etc, resulting in the same addresses calculated by the ST1W instruction.

Right now I believe that for aggregate types:

• We can refer to portions of structs if they're split by SROA and promoted from there,
• The same for arrays that are completely promoted,
• All other aggregates are left on the stack, and we only describe their stack address.

As a result, the stack-location tracking in LiveDebugValues is only geared towards tracking the spills / restores of scalars that the register allocator introduces, and sometimes entire vectors. I have a fear that if this patch is trying to make elements within scalable vectors describable to debug-info, it might be aiming to do something that isn't supported.

I don't think this is any different for scalable vectors. Specifically because (from the LangRef):

Scalable vectors cannot be global variables or members of structs or arrays because their size is unknown at compile time.

So I would expect only entire vectors to be tracked. It is also worth noting that the current use for this is with source-level scalable vector types. For SVE the only scalable types are defined in arm_sve.h which can only be used in conjunction with the intrinsics defined in the Arm C/C++ Language Extensions for SVE.

This probably stems from not understanding scalable vectors; would you have any source examples of when this patch becomes necessary, just so that I can work through them?

The following example hits the assert in VarLocBasedLDV::extractSpillBaseRegAndOffset that the StackOffset has no scalable component (which it now has, because the stack contains two SVE locals, at least the address of one of them must have a non-zero scalable offset).

// After applying D90020, build this example with:
//   clang -O1 -g -march=armv8-a+sve2 -S t.c -o -

#include <arm_sve.h>

svint32_t bar(svint32_t &);

svint32_t foo(svint32_t v, int N) {
  svint32_t local_v1 = v;
  svint32_t local_v2 = v;
  if (N > 0) {
    local_v1 = bar(local_v1);
    local_v2 = bar(local_v2);
  }
  return svadd_z(svptrue_b32(), local_v1, local_v2);
}

Hopefully that clarifies a few things, but don't hesitate to ask if you have more questions!

[sdesmalen]

llvm/lib/CodeGen/LiveDebugValues/InstrRefBasedImpl.cpp

231–234

Yes, that's right.

llvm/lib/CodeGen/LiveDebugValues/VarLocBasedImpl.cpp

556

We did before, because uint64_t Hash covered SpillLoc (which contains an unsigned (SpillBase) and int (SpillOffset)). But now that SpillOffset is a StackOffset instead of an int, uint64_t Hash no longer covers all those bits so we need to explicitly compare the two.

[jmorse]

Ah, I hadn't quite lined up in my head that the offset scales as the stack grows downwards. With that in mind, I think this sounds safe -- the scalable part of the stack offset is opaque as far as LiveDebugValues is concerned, and we can just pass it around.

Are there any scenarios where stack slots can be accessed with different scalable offsets? For example if stack slot colouring merged two slots for vectors of different scalable size (maybe it doesn't do that). If that happened, the StackOffset object would compare differently, and we might miss a spilt value being overwritten (see llvm/test/DebugInfo/MIR/X86/live-debug-values-stack-clobber.mir for a scalar example of overwrites).

The size of a VarLoc object increases from 72 bytes to ~80 bytes on amd64; this isn't ideal, but trying to micro-optimise it at this stage invites errors.

llvm/test/CodeGen/AArch64/live-debugvalues-sve.mir

149–151

I'd recommend also testing restoring the vector to a register, to ensure LiveDebugValues chops the spill-expression off correctly. VarLocBasedLDV should put in a DBG_VALUE after loading from the stack; you should need to clobber the stack location with some other value to force InstrRefBasedLDV to issue another DBG_VALUE.

[sdesmalen]

Are there any scenarios where stack slots can be accessed with different scalable offsets? For example if stack slot colouring merged two slots for vectors of different scalable size (maybe it doesn't do that). If that happened, the StackOffset object would compare differently, and we might miss a spilt value being overwritten (see llvm/test/DebugInfo/MIR/X86/live-debug-values-stack-clobber.mir for a scalar example of overwrites).

I guess that is possible. As long as both frame objects have the same Stack-ID, StackSlotColouring can merge them together. However, in that case, the sizes and offsets of those objects would both be scalable and comparing these sizes would be no different as they would be for non-scalable sizes.

The size of a VarLoc object increases from 72 bytes to ~80 bytes on amd64; this isn't ideal, but trying to micro-optimise it at this stage invites errors.

Yes, that is unfortunate. We'll need to accept some cost for returning structs instead of int/unsigned throughout the code-base to represent the scalable semantics (same for TypeSize and ElementCount).

llvm/test/CodeGen/AArch64/live-debugvalues-sve.mir

149–151

I tried this, but I'm not entirely sure the result is correct because isRestoreInstruction always returns false (TargetInstrInfo::getRestoreSize always returns 0 if this is not overriden by the target, which it is not for AArch64).

So when I do a reload from %stack.0 and then clobber that stack location, like this:

$z3 = IMPLICIT_DEF
renamable $z1 = LD1W_IMM renamable $p0, %stack.0, 0, debug-location !34 :: (load unknown-size from %stack.0, align 16)
ST1W_IMM renamable $z3, killed renamable $p0, %stack.0, 0 :: (store unknown-size into %stack.0, align 16)

LiveDebugValues creates DBG_VALUE $noreg, $noreg, !27, !DIExpression(), debug-location !30. It doesn't seem to recognize LD1W_IMM as a reload, and therefore doesn't see that we've just reloaded debug-value !27 back into $z1. The DIExpression is empty though.

Is that what you meant?

[jmorse]

I guess that is possible. As long as both frame objects have the same Stack-ID, StackSlotColouring can merge them together. However, in that case, the sizes and offsets of those objects would both be scalable and comparing these sizes would be no different as they would be for non-scalable sizes.

I imagine if they're merged, all the load and store instructions will refer to the same %stack.[0-9]+ frame index. So it's probably fine.

Generally LGTM with the extra CHECK mentioned inline.

llvm/test/CodeGen/AArch64/live-debugvalues-sve.mir

149–151

TargetInstrInfo::getRestoreSize always returns 0 if this is not overriden by the target, which it is not for AArch64

Oooofff, I wasn't aware of that. Making sure restores are correct are probably out of scope for this patch then.

LiveDebugValues creates DBG_VALUE $noreg,

That's part of the correct behaviour (terminating locations if they're overwritten on the stack), please do add CHECK lines for that, plus a "TODO" indicating this will change when AArch64 can detect loads.

[sdesmalen]

• Added extra CHECK line and TODO to test.

[sdesmalen]

Thanks for the review @jmorse!

[djtodoro]

lgtm, thanks!

[StephenTozer]

llvm/lib/CodeGen/LiveDebugValues/VarLocBasedImpl.cpp

576–578

The SpillOffset fixed and scalable arguments here are missing Other., so we're comparing the current VarLoc to itself - can you fix this in a followup patch?

[sdesmalen]

llvm/lib/CodeGen/LiveDebugValues/VarLocBasedImpl.cpp

576–578

Good spot, thanks for pointing out! Do you have any objection for me to just push the fix directly? (I wouldn't really know how to create a test for this, unless you have any suggestions?)

[jmorse]

llvm/lib/CodeGen/LiveDebugValues/VarLocBasedImpl.cpp

576–578

IMO just pushing up the fix is fine -- the developer policy permits it if it's "obvious" that the patch is correct.

llvm/lib/CodeGen/LiveDebugValues/InstrRefBasedImpl.cpp

 #include "llvm/IR/Function.h"
 #include "llvm/IR/Module.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/MC/MCRegisterInfo.h"
 #include "llvm/Pass.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/Debug.h"
+#include "llvm/Support/TypeSize.h"
 #include "llvm/Support/raw_ostream.h"
 #include <algorithm>
 #include <cassert>
 #include <cstdint>
 #include <functional>
 #include <queue>
 #include <tuple>
 #include <utility>
 #include <vector>
 #include <limits.h>
 #include <limits>
 
 #include "LiveDebugValues.h"
 
 using namespace llvm;
 
 #define DEBUG_TYPE "livedebugvalues"
 
ObserveAllStackops("observe-all-stack-ops", cl::Hidden,
                        cl::init(false));
 
 namespace {
 
 // The location at which a spilled value resides. It consists of a register and
 // an offset.
 struct SpillLoc {
   unsigned SpillBase;
-  int SpillOffset;
+  StackOffset SpillOffset;
   bool operator==(const SpillLoc &Other) const {
-    return std::tie(SpillBase, SpillOffset) ==
-           std::tie(Other.SpillBase, Other.SpillOffset);
+    return std::make_pair(SpillBase, SpillOffset) ==
+           std::make_pair(Other.SpillBase, Other.SpillOffset);
   }
   bool operator<(const SpillLoc &Other) const {
-    return std::tie(SpillBase, SpillOffset) <
-           std::tie(Other.SpillBase, Other.SpillOffset);
+    return std::make_tuple(SpillBase, SpillOffset.getFixed(),
+                    SpillOffset.getScalable()) <
+           std::make_tuple(Other.SpillBase, Other.SpillOffset.getFixed(),
+                    Other.SpillOffset.getScalable());
   }
 };
 
 class LocIdx {
   unsigned Location;
 
   // Default constructor is private, initializing to an illegal location number.
   // Use only for "not an entry" elements in IndexedMaps.
MachineInstrBuilder emitLoc(Optional<LocIdx> MLoc, const DebugVariable &Var,
     const DIExpression *Expr = Properties.DIExpr;
     if (!MLoc) {
       // No location -> DBG_VALUE $noreg
       MIB.addReg(0, RegState::Debug);
       MIB.addReg(0, RegState::Debug);
     } else if (LocIdxToLocID[*MLoc] >= NumRegs) {
       unsigned LocID = LocIdxToLocID[*MLoc];
       const SpillLoc &Spill = SpillLocs[LocID - NumRegs + 1];
+
+      auto *TRI = MF.getSubtarget().getRegisterInfo();
+      auto Decomp = TRI->decomposeStackOffsetForDwarf(Spill.SpillOffset);
       Expr = DIExpression::prepend(Expr, DIExpression::ApplyOffset,
-                                   Spill.SpillOffset);
+                                   Decomp.FixedOffset, Decomp.ScalableOffset,
+                                   *Decomp.ScaleReg);
       unsigned Base = Spill.SpillBase;
       MIB.addReg(Base, RegState::Debug);
       MIB.addImm(0);
     } else {
       unsigned LocID = LocIdxToLocID[*MLoc];
       MIB.addReg(LocID, RegState::Debug);
       if (Properties.Indirect)
         MIB.addImm(0);
InstrRefBasedLDV::extractSpillBaseRegAndOffset(const MachineInstr &MI) {
   auto MMOI = MI.memoperands_begin();
   const PseudoSourceValue *PVal = (*MMOI)->getPseudoValue();
   assert(PVal->kind() == PseudoSourceValue::FixedStack &&
          "Inconsistent memory operand in spill instruction");
   int FI = cast<FixedStackPseudoSourceValue>(PVal)->getFrameIndex();
   const MachineBasicBlock *MBB = MI.getParent();
   Register Reg;
   StackOffset Offset = TFI->getFrameIndexReference(*MBB->getParent(), FI, Reg);
-  assert(!Offset.getScalable() &&
-         "Frame offsets with a scalable component are not supported");
-  return {Reg, Offset.getFixed()};
+  return {Reg, Offset};
 }
 
 /// End all previous ranges related to @MI and start a new range from @MI
 /// if it is a DBG_VALUE instr.
 bool InstrRefBasedLDV::transferDebugValue(const MachineInstr &MI) {
   if (!MI.isDebugValue())
     return false;
 

llvm/lib/CodeGen/LiveDebugValues/VarLocBasedImpl.cpp

 #include "llvm/IR/Function.h"
 #include "llvm/IR/Module.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/MC/MCRegisterInfo.h"
 #include "llvm/Pass.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/Debug.h"
+#include "llvm/Support/TypeSize.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Target/TargetMachine.h"
 #include <algorithm>
 #include <cassert>
 #include <cstdint>
 #include <functional>
 #include <queue>
 #include <tuple>
private:
   using OptFragmentInfo = Optional<DIExpression::FragmentInfo>;
 
   /// A pair of debug variable and value location.
   struct VarLoc {
     // The location at which a spilled variable resides. It consists of a
     // register and an offset.
     struct SpillLoc {
       unsigned SpillBase;
-      int SpillOffset;
+      StackOffset SpillOffset;
       bool operator==(const SpillLoc &Other) const {
         return SpillBase == Other.SpillBase && SpillOffset == Other.SpillOffset;
       }
       bool operator!=(const SpillLoc &Other) const {
         return !(*this == Other);
       }
     };
 
enum VarLocKind {
       ImmediateKind,
       EntryValueKind,
       EntryValueBackupKind,
       EntryValueCopyBackupKind
     } Kind = InvalidKind;
 
     /// The value location. Stored separately to avoid repeatedly
     /// extracting it from MI.
-    union {
+    union LocUnion {
       uint64_t RegNo;
       SpillLoc SpillLocation;
       uint64_t Hash;
       int64_t Immediate;
       const ConstantFP *FPImm;
       const ConstantInt *CImm;
+      LocUnion() : Hash(0) {}
     } Loc;
 
     VarLoc(const MachineInstr &MI, LexicalScopes &LS)
         : Var(MI.getDebugVariable(), MI.getDebugExpression(),
               MI.getDebugLoc()->getInlinedAt()),
           Expr(MI.getDebugExpression()), MI(MI) {
-      static_assert((sizeof(Loc) == sizeof(uint64_t)),
-                    "hash does not cover all members of Loc");
       assert(MI.isDebugValue() && "not a DBG_VALUE");
       assert(MI.getNumOperands() == 4 && "malformed DBG_VALUE");
       if (int RegNo = isDbgValueDescribedByReg(MI)) {
         Kind = RegisterKind;
         Loc.RegNo = RegNo;
       } else if (MI.getDebugOperand(0).isImm()) {
         Kind = ImmediateKind;
         Loc.Immediate = MI.getDebugOperand(0).getImm();
static VarLoc CreateCopyLoc(const MachineInstr &MI, LexicalScopes &LS,
       assert(VL.Kind == RegisterKind);
       VL.Loc.RegNo = NewReg;
       return VL;
     }
 
     /// Take the variable described by DBG_VALUE MI, and create a VarLoc
     /// locating it in the specified spill location.
     static VarLoc CreateSpillLoc(const MachineInstr &MI, unsigned SpillBase,
-                                 int SpillOffset, LexicalScopes &LS) {
+                                 StackOffset SpillOffset, LexicalScopes &LS) {
       VarLoc VL(MI, LS);
       assert(VL.Kind == RegisterKind);
       VL.Kind = SpillLocKind;
       VL.Loc.SpillLocation = {SpillBase, SpillOffset};
       return VL;
     }
 
     /// Create a DBG_VALUE representing this VarLoc in the given function.
MachineInstr *BuildDbgValue(MachineFunction &MF) const {
         // Register locations are like the source DBG_VALUE, but with the
         // register number from this VarLoc.
         return BuildMI(MF, DbgLoc, IID, Indirect, Loc.RegNo, Var, DIExpr);
       case SpillLocKind: {
         // Spills are indirect DBG_VALUEs, with a base register and offset.
         // Use the original DBG_VALUEs expression to build the spilt location
         // on top of. FIXME: spill locations created before this pass runs
         // are not recognized, and not handled here.
+        auto *TRI = MF.getSubtarget().getRegisterInfo();
+        auto Decomp = TRI->decomposeStackOffsetForDwarf(
+            Loc.SpillLocation.SpillOffset);
         auto *SpillExpr = DIExpression::prepend(
-            DIExpr, DIExpression::ApplyOffset, Loc.SpillLocation.SpillOffset);
+            DIExpr, DIExpression::ApplyOffset, Decomp.FixedOffset,
+            Decomp.ScalableOffset, Decomp.ScaleReg);
         unsigned Base = Loc.SpillLocation.SpillBase;
         return BuildMI(MF, DbgLoc, IID, true, Base, Var, SpillExpr);
       }
       case ImmediateKind: {
         MachineOperand MO = MI.getDebugOperand(0);
         return BuildMI(MF, DbgLoc, IID, Indirect, MO, Var, DIExpr);
       }
       case EntryValueBackupKind:
void dump(const TargetRegisterInfo *TRI, raw_ostream &Out = dbgs()) const {
       case RegisterKind:
       case EntryValueKind:
       case EntryValueBackupKind:
       case EntryValueCopyBackupKind:
         Out << printReg(Loc.RegNo, TRI);
         break;
       case SpillLocKind:
         Out << printReg(Loc.SpillLocation.SpillBase, TRI);
-        Out << "[" << Loc.SpillLocation.SpillOffset << "]";
+        Out << "[" << Loc.SpillLocation.SpillOffset.getFixed() << " + "
+            << Loc.SpillLocation.SpillOffset.getScalable() << "x vscale"
+            << "]";
         break;
       case ImmediateKind:
         Out << Loc.Immediate;
         break;
       case InvalidKind:
         llvm_unreachable("Invalid VarLoc in dump method");
       }
 
       Out << ", \"" << Var.getVariable()->getName() << "\", " << *Expr << ", ";
       if (Var.getInlinedAt())
         Out << "!" << Var.getInlinedAt()->getMetadataID() << ")\n";
       else
         Out << "(null))";
 
       if (isEntryBackupLoc())
         Out << " (backup loc)\n";
       else
         Out << "\n";
     }
 #endif
 
     bool operator==(const VarLoc &Other) const {
-      return Kind == Other.Kind && Var == Other.Var &&
-             Loc.Hash == Other.Loc.Hash && Expr == Other.Expr;
+      if (Kind == Other.Kind && Var == Other.Var && Expr == Other.Expr) {
+        switch (Kind) {
+        case SpillLocKind:
+          return Loc.SpillLocation == Other.Loc.SpillLocation;
+        case RegisterKind:
+        case ImmediateKind:
+        case EntryValueKind:
+        case EntryValueBackupKind:
+        case EntryValueCopyBackupKind:
+          return Loc.Hash == Other.Loc.Hash;
+        default:
+          llvm_unreachable("Invalid kind");
+        }
+      }
     }
 
     /// This operator guarantees that VarLocs are sorted by Variable first.
     bool operator<(const VarLoc &Other) const {
+      switch (Kind) {
+      case SpillLocKind:
+        return std::make_tuple(Var, Kind, Loc.SpillLocation.SpillBase,
+                               Loc.SpillLocation.SpillOffset.getFixed(),
+                               Loc.SpillLocation.SpillOffset.getScalable(),
+                               Expr) <
+               std::make_tuple(
+                   Other.Var, Other.Kind, Other.Loc.SpillLocation.SpillBase,
+                   Loc.SpillLocation.SpillOffset.getFixed(),
+                   Loc.SpillLocation.SpillOffset.getScalable(), Other.Expr);
+      case RegisterKind:
+      case ImmediateKind:
+      case EntryValueKind:
+      case EntryValueBackupKind:
+      case EntryValueCopyBackupKind:
         return std::tie(Var, Kind, Loc.Hash, Expr) <
                std::tie(Other.Var, Other.Kind, Other.Loc.Hash, Other.Expr);
+      default:
+        llvm_unreachable("Invalid kind");
+      }
     }
   };
 
   /// VarLocMap is used for two things:
   /// 1) Assigning a unique LocIndex to a VarLoc. This LocIndex can be used to
   ///    virtually insert a VarLoc into a VarLocSet.
   /// 2) Given a LocIndex, look up the unique associated VarLoc.
   class VarLocMap {
VarLocBasedLDV::extractSpillBaseRegAndOffset(const MachineInstr &MI) {
   auto MMOI = MI.memoperands_begin();
   const PseudoSourceValue *PVal = (*MMOI)->getPseudoValue();
   assert(PVal->kind() == PseudoSourceValue::FixedStack &&
          "Inconsistent memory operand in spill instruction");
   int FI = cast<FixedStackPseudoSourceValue>(PVal)->getFrameIndex();
   const MachineBasicBlock *MBB = MI.getParent();
   Register Reg;
   StackOffset Offset = TFI->getFrameIndexReference(*MBB->getParent(), FI, Reg);
-  assert(!Offset.getScalable() &&
-         "Frame offsets with a scalable component are not supported");
-  return {Reg, Offset.getFixed()};
+  return {Reg, Offset};
 }
 
 /// Try to salvage the debug entry value if we encounter a new debug value
 /// describing the same parameter, otherwise stop tracking the value. Return
 /// true if we should stop tracking the entry value, otherwise return false.
 bool VarLocBasedLDV::removeEntryValue(const MachineInstr &MI,
                                        OpenRangesSet &OpenRanges,
                                        VarLocMap &VarLocIDs,

llvm/test/CodeGen/AArch64/live-debugvalues-sve.mir

+# RUN: llc -start-before=prologepilog -stop-after=livedebugvalues -o - %s | FileCheck %s
+# RUN: llc -start-before=prologepilog -stop-after=livedebugvalues  -experimental-debug-variable-locations -o - %s | FileCheck %s
+#
+# Test that the LiveDebugValues pass can correctly handle the address
+# of the SVE spill (at a scalable address location) which is expressed
+# with a complex DIExpression.
+#
+# CHECK: $z2 = ORR_ZZZ killed $z1, $z1
+# CHECK: $z1 = ORR_ZZZ killed $z0, $z0
+# CHECK-DAG: ST1W_IMM renamable $z1, killed renamable $p{{[0-9]+}}, $fp, -[[#%u,OFFSET:]], debug-location !{{[0-9]+}} :: (store unknown-size into %stack.0, align 16)
+# CHECK-DAG: DBG_VALUE $fp, 0, !{{[0-9]+}}, !DIExpression(DW_OP_constu, [[#mul(OFFSET,8)]], DW_OP_bregx, 46, 0, DW_OP_mul, DW_OP_minus),
+--- |
+  ; ModuleID = 't.c'
+  source_filename = "t.c"
+  target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
+  target triple = "aarch64-unknown-linux-gnu"
+
+  ; Function Attrs: nounwind
+  define dso_local <vscale x 8 x i32> @foo(<vscale x 4 x i32> %z0, <vscale x 4 x i32> %z1, i32 %c) local_unnamed_addr #0 !dbg !11 {
+  entry:
+    call void @llvm.dbg.value(metadata <vscale x 4 x i32> %z0, metadata !27, metadata !DIExpression()), !dbg !30
+    call void @llvm.dbg.value(metadata <vscale x 4 x i32> %z1, metadata !28, metadata !DIExpression()), !dbg !30
+    call void @llvm.dbg.value(metadata i32 %c, metadata !29, metadata !DIExpression()), !dbg !30
+    %tobool.not = icmp eq i32 %c, 0, !dbg !31
+    br i1 %tobool.not, label %if.end, label %if.then, !dbg !33
+
+  if.then:                                          ; preds = %entry
+    %call = call <vscale x 8 x i32> @bar(<vscale x 4 x i32> %z0, <vscale x 4 x i32> %z0, <vscale x 4 x i32> %z0, <vscale x 4 x i32> %z0, <vscale x 4 x i32> %z0, <vscale x 4 x i32> %z0, <vscale x 4 x i32> %z0, <vscale x 4 x i32> %z0, <vscale x 4 x i32> %z0) #4, !dbg !34
+    br label %return, !dbg !35
+
+  if.end:                                           ; preds = %entry
+    %0 = call <vscale x 8 x i32> @llvm.aarch64.sve.tuple.create2.nxv8i32.nxv4i32(<vscale x 4 x i32> %z1, <vscale x 4 x i32> %z0), !dbg !36
+    br label %return, !dbg !37
+
+  return:                                           ; preds = %if.end, %if.then
+    %retval.0 = phi <vscale x 8 x i32> [ %call, %if.then ], [ %0, %if.end ], !dbg !30
+    ret <vscale x 8 x i32> %retval.0, !dbg !38
+  }
+
+  declare !dbg !39 dso_local <vscale x 8 x i32> @bar(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32>) local_unnamed_addr #1
+
+  ; Function Attrs: nounwind readonly
+  declare <vscale x 8 x i32> @llvm.aarch64.sve.tuple.create2.nxv8i32.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>) #2
+
+  ; Function Attrs: nounwind readnone speculatable willreturn
+  declare void @llvm.dbg.value(metadata, metadata, metadata) #3
+
+  attributes #0 = { nounwind "disable-tail-calls"="false" "frame-pointer"="non-leaf" "less-precise-fpmad"="false" "min-legal-vector-width"="256" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+neon,+sve" "unsafe-fp-math"="false" "use-soft-float"="false" }
+  attributes #1 = { "disable-tail-calls"="false" "frame-pointer"="non-leaf" "less-precise-fpmad"="false" "no-infs-fp-math"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+neon,+sve" "unsafe-fp-math"="false" "use-soft-float"="false" }
+  attributes #2 = { nounwind readonly }
+  attributes #3 = { nounwind readnone speculatable willreturn }
+  attributes #4 = { nounwind }
+
+  !llvm.dbg.cu = !{!0}
+  !llvm.module.flags = !{!3, !4, !5, !6, !7, !8, !9}
+  !llvm.ident = !{!10}
+
+  !0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 12.0.0 (https://github.com/llvm/llvm-project.git b19275ba870a06c5ef0428af6264ffd28c7cde9e)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, splitDebugInlining: false, nameTableKind: None)
+  !1 = !DIFile(filename: "t.c", directory: "/tmp/")
+  !2 = !{}
+  !3 = !{i32 7, !"Dwarf Version", i32 4}
+  !4 = !{i32 2, !"Debug Info Version", i32 3}
+  !5 = !{i32 1, !"wchar_size", i32 4}
+  !6 = !{i32 1, !"branch-target-enforcement", i32 0}
+  !7 = !{i32 1, !"sign-return-address", i32 0}
+  !8 = !{i32 1, !"sign-return-address-all", i32 0}
+  !9 = !{i32 1, !"sign-return-address-with-bkey", i32 0}
+  !10 = !{!"clang version 12.0.0 (https://github.com/llvm/llvm-project.git b19275ba870a06c5ef0428af6264ffd28c7cde9e)"}
+  !11 = distinct !DISubprogram(name: "foo", scope: !1, file: !1, line: 6, type: !12, scopeLine: 6, flags: DIFlagPrototyped | DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !26)
+  !12 = !DISubroutineType(types: !13)
+  !13 = !{!14, !21, !21, !18}
+  !14 = !DIDerivedType(tag: DW_TAG_typedef, name: "svint32x2_t", file: !15, line: 59, baseType: !16)
+  !15 = !DIFile(filename: "lib/clang/12.0.0/include/arm_sve.h", directory: "/tmp")
+  !16 = !DIDerivedType(tag: DW_TAG_typedef, name: "__clang_svint32x2_t", file: !1, baseType: !17)
+  !17 = !DICompositeType(tag: DW_TAG_array_type, baseType: !18, flags: DIFlagVector, elements: !19)
+  !18 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+  !19 = !{!20}
+  !20 = !DISubrange(lowerBound: 0, upperBound: !DIExpression(DW_OP_constu, 4, DW_OP_bregx, 46, 0, DW_OP_mul, DW_OP_constu, 1, DW_OP_minus))
+  !21 = !DIDerivedType(tag: DW_TAG_typedef, name: "svint32_t", file: !15, line: 34, baseType: !22)
+  !22 = !DIDerivedType(tag: DW_TAG_typedef, name: "__SVInt32_t", file: !1, baseType: !23)
+  !23 = !DICompositeType(tag: DW_TAG_array_type, baseType: !18, flags: DIFlagVector, elements: !24)
+  !24 = !{!25}
+  !25 = !DISubrange(lowerBound: 0, upperBound: !DIExpression(DW_OP_constu, 2, DW_OP_bregx, 46, 0, DW_OP_mul, DW_OP_constu, 1, DW_OP_minus))
+  !26 = !{!27, !28, !29}
+  !27 = !DILocalVariable(name: "z0", arg: 1, scope: !11, file: !1, line: 6, type: !21)
+  !28 = !DILocalVariable(name: "z1", arg: 2, scope: !11, file: !1, line: 6, type: !21)
+  !29 = !DILocalVariable(name: "c", arg: 3, scope: !11, file: !1, line: 6, type: !18)
+  !30 = !DILocation(line: 0, scope: !11)
+  !31 = !DILocation(line: 7, column: 7, scope: !32)
+  !32 = distinct !DILexicalBlock(scope: !11, file: !1, line: 7, column: 7)
+  !33 = !DILocation(line: 7, column: 7, scope: !11)
+  !34 = !DILocation(line: 8, column: 12, scope: !32)
+  !35 = !DILocation(line: 8, column: 5, scope: !32)
+  !36 = !DILocation(line: 9, column: 10, scope: !11)
+  !37 = !DILocation(line: 9, column: 3, scope: !11)
+  !38 = !DILocation(line: 10, column: 1, scope: !11)
+  !39 = !DISubprogram(name: "bar", scope: !1, file: !1, line: 3, type: !40, flags: DIFlagPrototyped, spFlags: DISPFlagOptimized, retainedNodes: !2)
+  !40 = !DISubroutineType(types: !41)
+  !41 = !{!17, !23, !23, !23, !23, !23, !23, !23, !23, !23}
+
+...
+---
+name:            foo
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '$z0' }
+  - { reg: '$z1' }
+  - { reg: '$w0' }
+frameInfo:
+  maxAlignment:    16
+  adjustsStack:    true
+  hasCalls:        true
+  maxCallFrameSize: 0
+  savePoint:       '%bb.1'
+  restorePoint:    '%bb.1'
+stack:
+  - { id: 0, size: 16, alignment: 16, stack-id: sve-vec }
+machineFunctionInfo: {}
+body:             |
+  bb.0.entry:
+    successors: %bb.2(0x30000000), %bb.1(0x50000000)
+    liveins: $w0, $z0, $z1
+
+    DBG_VALUE $z0, $noreg, !27, !DIExpression(), debug-location !30
+    DBG_VALUE $z1, $noreg, !28, !DIExpression(), debug-location !30
+    DBG_VALUE $w0, $noreg, !29, !DIExpression(), debug-location !30
+    renamable $z2 = COPY $z1
+    renamable $z1 = COPY $z0
+    DBG_VALUE $w0, $noreg, !29, !DIExpression(), debug-location !30
+    DBG_VALUE $z2, $noreg, !28, !DIExpression(), debug-location !30
+    DBG_VALUE $z1, $noreg, !27, !DIExpression(), debug-location !30
+    CBZW killed renamable $w0, %bb.2, debug-location !33
+    B %bb.1, debug-location !33
+
+  bb.1.if.then:
+    liveins: $z1
+
+    ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp, debug-location !34
+    renamable $p0 = PTRUE_S 31, debug-location !34
+    $x0 = ADDXri %stack.0, 0, 0, debug-location !34
+    ST1W_IMM renamable $z1, killed renamable $p0, %stack.0, 0, debug-location !34 :: (store unknown-size into %stack.0, align 16)
+    $z0 = COPY renamable $z1, debug-location !34
+    $z2 = COPY renamable $z1, debug-location !34
+    $z3 = COPY renamable $z1, debug-location !34
+    $z4 = COPY renamable $z1, debug-location !34
+    $z5 = COPY renamable $z1, debug-location !34
+    $z6 = COPY renamable $z1, debug-location !34
+    $z7 = COPY renamable $z1, debug-location !34
+    BL @bar, csr_aarch64_sve_aapcs, implicit-def dead $lr, implicit $sp, implicit $z0, implicit $z1, implicit $z2, implicit $z3, implicit $z4, implicit $z5, implicit $z6, implicit $z7, implicit $x0, implicit-def $sp, implicit-def $z0, implicit-def $z1, debug-location !34
+    ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp, debug-location !34
+    renamable $z2 = COPY $z0, debug-location !34
+    B %bb.3, debug-location !35
+
+  bb.2.if.end:
+    liveins: $z1, $z2
+
+
+  bb.3.return:
+    liveins: $z1, $z2
+
+    $z0 = COPY killed renamable $z2, debug-location !38
+    RET_ReallyLR implicit $z0, implicit $z1, debug-location !38
+
+...