[DebugInfo] Produce variadic DBG_INSTR_REFs from ISel

[StephenTozer]

Summary

This patch modifies SelectionDAG and FastISel to produce DBG_INSTR_REFs with variadic expressions, and produce DBG_INSTR_REFs for debug values with variadic location expressions. The former essentially means just prepending DW_OP_LLVM_arg, 0 to the existing expression. The latter is achieved in MachineFunction::finalizeDebugInstrRefs and InstrEmitter::EmitDbgInstrRef.

[jmorse]

Generally looks good -- needs tests though.

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

833–835

Haven't these all been excluded due to IsInvalidOp?

[jmorse]

The "arg zero" form of DIExpression is going to be a very common case -- is it uniqued, and would it be worth storing a copy somewhere and just using that?

[StephenTozer]

In D133929#3796075, @jmorse wrote:

The "arg zero" form of DIExpression is going to be a very common case -- is it uniqued, and would it be worth storing a copy somewhere and just using that?

DIExpressions should be uniqued based on their expression elements, and the get should just be a map lookup underneath, so I don't think there should be any issues with duplicate storage.

[StephenTozer]

Rebased, fixed some errors when emitting undef values and update some tests accordingly.

[StephenTozer]

Added some test coverage for SelectionDAG+FastISel emission of variadic instruction references.

[Orlando]

This LGTM with a few inline questions. The addition of DW_OP_arg, 0 to all INSTR_REF expressions does add some visual noise but I personally prefer this. Given how convoluted debug instruction handling can get, I like the idea of continuing in this direction and minimising the number of special cases.

llvm/lib/CodeGen/MachineFunction.cpp

1138

👍

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

698

Possibly worth expanding the comment to mention that these will have been emitted as INSTR_REF by the code above if instr ref is enabled.

856–857

Looks like this one slipped through the cracks - can this TODO be done? IMO eliminating duplicated code is especially important around these fiddly areas.

922

Is there a benefit to preserving the number of operands (rather than just using DBG_VALUE $noreg) - does it strip assertions about DW_OP_arg usage or something if we don't?

[StephenTozer]

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

922

The only important part about this is that it doesn't invalidate the DIExpression - even for an undef debug value, I believe there are still some points at which having a mismatched number of debug operands with the DIExpression's arg operands is a problem. This could be resolved by also changing the expression, making it empty while any fragment info or other expression elements that apply to undef values - which sounds like something that should be done for all explicitly killed debug values, but happy to start the ball rolling here.

[StephenTozer]

Address comments (extracted MO selection for const debug operands, expanded comments in EmitDbgValue), add a function for setting a DIExpression "undef".

[Orlando]

LGTM

[StephenTozer]

Rebased onto latest main, added unit tests for new convertToUndefExpression, and converted some existing tests to test for the new fully-undef debug values that SelectionDAG produces.

[Orlando]

Still LGTM, nice unittest.

llvm/include/llvm/IR/DebugInfoMetadata.h

public:
   /// it cannot be a simple register location.
   bool isComplex() const;
 
   /// Return whether the evaluated expression makes use of a single location at
   /// the start of the expression, i.e. if it contains only a single
   /// DW_OP_LLVM_arg op as its first operand, or if it contains none.
   bool isSingleLocationExpression() const;
 
+  /// Removes all elements from \p Expr that do not apply to an undef debug
+  /// value, which includes every operator that computes the value/location on
+  /// the DWARF stack, including any DW_OP_LLVM_arg elements (making the result
+  /// of this function always a single-location expression) while leaving
+  /// everything that defines what the computed value applies to, i.e. the
+  /// fragment information.
+  static const DIExpression *convertToUndefExpression(const DIExpression *Expr);
+
   /// If \p Expr is a non-variadic expression (i.e. one that does not contain
   /// DW_OP_LLVM_arg), returns \p Expr converted to variadic form by adding a
   /// leading [DW_OP_LLVM_arg, 0] to the expression; otherwise returns \p Expr.
   static const DIExpression *
   convertToVariadicExpression(const DIExpression *Expr);
 
   /// If \p Expr is a valid single-location expression, i.e. it refers to only a
   /// single debug operand at the start of the expression, then return that

llvm/lib/CodeGen/MachineFunction.cpp

 #include "llvm/CodeGen/TargetSubtargetInfo.h"
 #include "llvm/CodeGen/WasmEHFuncInfo.h"
 #include "llvm/CodeGen/WinEHFuncInfo.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/IR/Attributes.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/Constant.h"
 #include "llvm/IR/DataLayout.h"
-#include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/DerivedTypes.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/GlobalValue.h"
 #include "llvm/IR/Instruction.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/Metadata.h"
 #include "llvm/IR/Module.h"
 #include "llvm/IR/ModuleSlotTracker.h"

 }
 
 void MachineFunction::finalizeDebugInstrRefs() {
   auto *TII = getSubtarget().getInstrInfo();
 
   auto MakeUndefDbgValue = [&](MachineInstr &MI) {
     const MCInstrDesc &RefII = TII->get(TargetOpcode::DBG_VALUE_LIST);
     MI.setDesc(RefII);
-    MI.getDebugOperand(0).setReg(0);
+    MI.setDebugValueUndef();
   };
 
   DenseMap<Register, DebugInstrOperandPair> ArgDbgPHIs;
   for (auto &MBB : *this) {
     for (auto &MI : MBB) {
-      if (!MI.isDebugRef() || !MI.getDebugOperand(0).isReg())
+      if (!MI.isDebugRef())
         continue;
 
-      Register Reg = MI.getDebugOperand(0).getReg();
+      bool IsValidRef = true;
+
+      for (MachineOperand &MO : MI.debug_operands()) {
+        if (!MO.isReg())
+          continue;
+
+        Register Reg = MO.getReg();
 
         // Some vregs can be deleted as redundant in the meantime. Mark those
         // as DBG_VALUE $noreg. Additionally, some normal instructions are
         // quickly deleted, leaving dangling references to vregs with no def.
         if (Reg == 0 || !RegInfo->hasOneDef(Reg)) {
-        MakeUndefDbgValue(MI);
-        continue;
+          IsValidRef = false;
+          break;
         }
-      // Only convert Expr to variadic form when we're sure we're emitting a
-      // complete instruction reference.
-      MI.getDebugExpressionOp().setMetadata(
-          DIExpression::convertToVariadicExpression(MI.getDebugExpression()));
 
         assert(Reg.isVirtual());
         MachineInstr &DefMI = *RegInfo->def_instr_begin(Reg);
 
         // If we've found a copy-like instruction, follow it back to the
         // instruction that defines the source value, see salvageCopySSA docs
         // for why this is important.
         if (DefMI.isCopyLike() || TII->isCopyInstr(DefMI)) {
           auto Result = salvageCopySSA(DefMI, ArgDbgPHIs);
-        MI.getDebugOperand(0).ChangeToDbgInstrRef(Result.first, Result.second);
+          MO.ChangeToDbgInstrRef(Result.first, Result.second);
         } else {
           // Otherwise, identify the operand number that the VReg refers to.
           unsigned OperandIdx = 0;
-        for (const auto &MO : DefMI.operands()) {
-          if (MO.isReg() && MO.isDef() && MO.getReg() == Reg)
+          for (const auto &DefMO : DefMI.operands()) {
+            if (DefMO.isReg() && DefMO.isDef() && DefMO.getReg() == Reg)
               break;
             ++OperandIdx;
           }
           assert(OperandIdx < DefMI.getNumOperands());
 
           // Morph this instr ref to point at the given instruction and operand.
           unsigned ID = DefMI.getDebugInstrNum();
-        MI.getDebugOperand(0).ChangeToDbgInstrRef(ID, OperandIdx);
+          MO.ChangeToDbgInstrRef(ID, OperandIdx);
+        }
       }
+
+      if (!IsValidRef)
+        MakeUndefDbgValue(MI);
     }
   }
 }
 
 bool MachineFunction::useDebugInstrRef() const {
   // Disable instr-ref at -O0: it's very slow (in compile time). We can still
   // have optimized code inlined into this unoptimized code, however with
   // fewer and less aggressive optimizations happening, coverage and accuracy

llvm/lib/CodeGen/SelectionDAG/FastISel.cpp

case Intrinsic::dbg_declare: {
 
     if (Op) {
       assert(DI->getVariable()->isValidLocationForIntrinsic(MIMD.getDL()) &&
              "Expected inlined-at fields to agree");
       if (UseInstrRefDebugInfo && Op->isReg()) {
         // If using instruction referencing, produce this as a DBG_INSTR_REF,
         // to be later patched up by finalizeDebugInstrRefs. Tack a deref onto
         // the expression, we don't have an "indirect" flag in DBG_INSTR_REF.
-        auto *NewExpr =
-           DIExpression::prepend(DI->getExpression(), DIExpression::DerefBefore);
+        SmallVector<uint64_t, 3> Ops(
+            {dwarf::DW_OP_LLVM_arg, 0, dwarf::DW_OP_deref});
+        auto *NewExpr = DIExpression::prependOpcodes(DI->getExpression(), Ops);
         BuildMI(*FuncInfo.MBB, FuncInfo.InsertPt, MIMD.getDL(),
                 TII.get(TargetOpcode::DBG_INSTR_REF), /*IsIndirect*/ false, *Op,
                 DI->getVariable(), NewExpr);
       } else {
         // A dbg.declare describes the address of a source variable, so lower it
         // into an indirect DBG_VALUE.
         BuildMI(*FuncInfo.MBB, FuncInfo.InsertPt, MIMD.getDL(),
                 TII.get(TargetOpcode::DBG_VALUE), /*IsIndirect*/ true, *Op,
if (!V || isa<UndefValue>(V) || DI->hasArgList()) {
       } else {
         // If using instruction referencing, produce this as a DBG_INSTR_REF,
         // to be later patched up by finalizeDebugInstrRefs.
         SmallVector<MachineOperand, 1> MOs({MachineOperand::CreateReg(
             /* Reg */ Reg, /* isDef */ false, /* isImp */ false,
             /* isKill */ false, /* isDead */ false,
             /* isUndef */ false, /* isEarlyClobber */ false,
             /* SubReg */ 0, /* isDebug */ true)});
+        SmallVector<uint64_t, 2> Ops({dwarf::DW_OP_LLVM_arg, 0});
+        auto *NewExpr = DIExpression::prependOpcodes(DI->getExpression(), Ops);
         BuildMI(*FuncInfo.MBB, FuncInfo.InsertPt, MIMD.getDL(),
                 TII.get(TargetOpcode::DBG_INSTR_REF), /*IsIndirect*/ false, MOs,
-                DI->getVariable(), DI->getExpression());
+                DI->getVariable(), NewExpr);
       }
     } else {
       // We don't know how to handle other cases, so we drop.
       LLVM_DEBUG(dbgs() << "Dropping debug info for " << *DI << "\n");
     }
     return true;
   }
   case Intrinsic::dbg_label: {

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.h

public:
   /// if there is no variable location; alternately a half-formed DBG_INSTR_REF
   /// that refers to a virtual register and is corrected later in isel.
   MachineInstr *EmitDbgInstrRef(SDDbgValue *SD,
                                 DenseMap<SDValue, Register> &VRBaseMap);
 
   /// Emit a DBG_VALUE $noreg, indicating a variable has no location.
   MachineInstr *EmitDbgNoLocation(SDDbgValue *SD);
 
+  /// Emit a DBG_VALUE_LIST from the operands to SDDbgValue.
+  MachineInstr *EmitDbgValueList(SDDbgValue *SD,
+                                 DenseMap<SDValue, Register> &VRBaseMap);
+
   /// Emit a DBG_VALUE from the operands to SDDbgValue.
   MachineInstr *EmitDbgValueFromSingleOp(SDDbgValue *SD,
                                     DenseMap<SDValue, Register> &VRBaseMap);
 
   /// Generate machine instruction for a dbg_label node.
   MachineInstr *EmitDbgLabel(SDDbgLabel *SD);
 
   /// EmitNode - Generate machine code for a node and needed dependencies.

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

 }
 
 /// EmitDbgValue - Generate machine instruction for a dbg_value node.
 ///
 MachineInstr *
 InstrEmitter::EmitDbgValue(SDDbgValue *SD,
                            DenseMap<SDValue, Register> &VRBaseMap) {
   MDNode *Var = SD->getVariable();
-  MDNode *Expr = SD->getExpression();
   DebugLoc DL = SD->getDebugLoc();
   assert(cast<DILocalVariable>(Var)->isValidLocationForIntrinsic(DL) &&
          "Expected inlined-at fields to agree");
 
   SD->setIsEmitted();
 
   ArrayRef<SDDbgOperand> LocationOps = SD->getLocationOps();
   assert(!LocationOps.empty() && "dbg_value with no location operands?");
 
   if (SD->isInvalidated())
     return EmitDbgNoLocation(SD);
 
-  // Emit variadic dbg_value nodes as DBG_VALUE_LIST.
-  if (SD->isVariadic()) {
-    // DBG_VALUE_LIST := "DBG_VALUE_LIST" var, expression, loc (, loc)*
-    const MCInstrDesc &DbgValDesc = TII->get(TargetOpcode::DBG_VALUE_LIST);
-    // Build the DBG_VALUE_LIST instruction base.
-    auto MIB = BuildMI(*MF, DL, DbgValDesc);
-    MIB.addMetadata(Var);
-    MIB.addMetadata(Expr);
-    AddDbgValueLocationOps(MIB, DbgValDesc, LocationOps, VRBaseMap);
-    return &*MIB;
-  }
-
   // Attempt to produce a DBG_INSTR_REF if we've been asked to.
-  // We currently exclude the possibility of instruction references for
-  // variadic nodes; if at some point we enable them, this should be moved
-  // above the variadic block.
   if (EmitDebugInstrRefs)
     if (auto *InstrRef = EmitDbgInstrRef(SD, VRBaseMap))
       return InstrRef;
 
+  // Emit variadic dbg_value nodes as DBG_VALUE_LIST if they have not been
+  // emitted as instruction references.
+  if (SD->isVariadic())
+    return EmitDbgValueList(SD, VRBaseMap);
+
+  // Emit single-location dbg_value nodes as DBG_VALUE if they have not been
+  // emitted as instruction references.
   return EmitDbgValueFromSingleOp(SD, VRBaseMap);
 }
 
+MachineOperand GetMOForConstDbgOp(const SDDbgOperand &Op) {
+  const Value *V = Op.getConst();
+  if (const ConstantInt *CI = dyn_cast<ConstantInt>(V)) {
+    if (CI->getBitWidth() > 64)
+      return MachineOperand::CreateCImm(CI);
+    return MachineOperand::CreateImm(CI->getSExtValue());
+  }
+  if (const ConstantFP *CF = dyn_cast<ConstantFP>(V))
+    return MachineOperand::CreateFPImm(CF);
+  // Note: This assumes that all nullptr constants are zero-valued.
+  if (isa<ConstantPointerNull>(V))
+    return MachineOperand::CreateImm(0);
+  // Undef or unhandled value type, so return an undef operand.
+  return MachineOperand::CreateReg(
+      /* Reg */ 0U, /* isDef */ false, /* isImp */ false,
+      /* isKill */ false, /* isDead */ false,
+      /* isUndef */ false, /* isEarlyClobber */ false,
+      /* SubReg */ 0, /* isDebug */ true);
+}
+
 void InstrEmitter::AddDbgValueLocationOps(
     MachineInstrBuilder &MIB, const MCInstrDesc &DbgValDesc,
     ArrayRef<SDDbgOperand> LocationOps,
     DenseMap<SDValue, Register> &VRBaseMap) {
   for (const SDDbgOperand &Op : LocationOps) {
     switch (Op.getKind()) {
     case SDDbgOperand::FRAMEIX:
       MIB.addFrameIndex(Op.getFrameIx());
case SDDbgOperand::SDNODE: {
       // in all cases would be very fragile; this is a safeguard for any
       // that were missed.
       if (VRBaseMap.count(V) == 0)
         MIB.addReg(0U); // undef
       else
         AddOperand(MIB, V, (*MIB).getNumOperands(), &DbgValDesc, VRBaseMap,
                    /*IsDebug=*/true, /*IsClone=*/false, /*IsCloned=*/false);
     } break;
-    case SDDbgOperand::CONST: {
-      const Value *V = Op.getConst();
-      if (const ConstantInt *CI = dyn_cast<ConstantInt>(V)) {
-        if (CI->getBitWidth() > 64)
-          MIB.addCImm(CI);
-        else
-          MIB.addImm(CI->getSExtValue());
-      } else if (const ConstantFP *CF = dyn_cast<ConstantFP>(V)) {
-        MIB.addFPImm(CF);
-      } else if (isa<ConstantPointerNull>(V)) {
-        // Note: This assumes that all nullptr constants are zero-valued.
-        MIB.addImm(0);
-      } else {
-        // Could be an Undef. In any case insert an Undef so we can see what we
-        // dropped.
-        MIB.addReg(0U);
-      }
-    } break;
+    case SDDbgOperand::CONST:
+      MIB.add(GetMOForConstDbgOp(Op));
+      break;
     }
   }
 }
 
 MachineInstr *
 InstrEmitter::EmitDbgInstrRef(SDDbgValue *SD,
                               DenseMap<SDValue, Register> &VRBaseMap) {
-  assert(!SD->isVariadic());
-  SDDbgOperand DbgOperand = SD->getLocationOps()[0];
   MDNode *Var = SD->getVariable();
   const DIExpression *Expr = (DIExpression *)SD->getExpression();
   DebugLoc DL = SD->getDebugLoc();
   const MCInstrDesc &RefII = TII->get(TargetOpcode::DBG_INSTR_REF);
 
-  // Handle variable locations that don't actually depend on the instructions
-  // in the program: constants and stack locations.
-  if (DbgOperand.getKind() == SDDbgOperand::FRAMEIX ||
-      DbgOperand.getKind() == SDDbgOperand::CONST)
+  // Returns true if the given operand is not a legal debug operand for a
+  // DBG_INSTR_REF.
+  auto IsInvalidOp = [](SDDbgOperand DbgOp) {
+    return DbgOp.getKind() == SDDbgOperand::FRAMEIX;
+  };
+  // Returns true if the given operand is not itself an instruction reference
+  // but is a legal debug operand for a DBG_INSTR_REF.
+  auto IsNonInstrRefOp = [](SDDbgOperand DbgOp) {
+    return DbgOp.getKind() == SDDbgOperand::CONST;
+  };
+
+  // If this variable location does not depend on any instructions or contains
+  // any stack locations, produce it as a standard debug value instead.
+  if (any_of(SD->getLocationOps(), IsInvalidOp) ||
+      all_of(SD->getLocationOps(), IsNonInstrRefOp)) {
+    if (SD->isVariadic())
+      return EmitDbgValueList(SD, VRBaseMap);
     return EmitDbgValueFromSingleOp(SD, VRBaseMap);
+  }
 
   // Immediately fold any indirectness from the LLVM-IR intrinsic into the
   // expression:
-  if (SD->isIndirect()) {
-    std::vector<uint64_t> Elts = {dwarf::DW_OP_deref};
-    Expr = DIExpression::append(Expr, Elts);
-  }
+  if (SD->isIndirect())
+    Expr = DIExpression::append(Expr, dwarf::DW_OP_deref);
+  // If this is not already a variadic expression, it must be modified to become
+  // one.
+  if (!SD->isVariadic())
+    Expr = DIExpression::convertToVariadicExpression(Expr);
+
+  SmallVector<MachineOperand> MOs;
 
   // It may not be immediately possible to identify the MachineInstr that
   // defines a VReg, it can depend for example on the order blocks are
   // emitted in. When this happens, or when further analysis is needed later,
   // produce an instruction like this:
   //
-  //    DBG_INSTR_REF %0:gr64, 0, !123, !456
+  //    DBG_INSTR_REF !123, !456, %0:gr64
   //
   // i.e., point the instruction at the vreg, and patch it up later in
   // MachineFunction::finalizeDebugInstrRefs.
-  auto EmitHalfDoneInstrRef = [&](unsigned VReg) -> MachineInstr * {
-    SmallVector<MachineOperand, 1> MOs({MachineOperand::CreateReg(
+  auto AddVRegOp = [&](unsigned VReg) {
+    MOs.push_back(MachineOperand::CreateReg(
         /* Reg */ VReg, /* isDef */ false, /* isImp */ false,
         /* isKill */ false, /* isDead */ false,
         /* isUndef */ false, /* isEarlyClobber */ false,
-        /* SubReg */ 0, /* isDebug */ true)});
-    return BuildMI(*MF, DL, RefII, false, MOs, Var, Expr);
+        /* SubReg */ 0, /* isDebug */ true));
   };
+  unsigned OpCount = SD->getLocationOps().size();
+  for (unsigned OpIdx = 0; OpIdx < OpCount; ++OpIdx) {
+    SDDbgOperand DbgOperand = SD->getLocationOps()[OpIdx];
 
     // Try to find both the defined register and the instruction defining it.
     MachineInstr *DefMI = nullptr;
     unsigned VReg;
 
     if (DbgOperand.getKind() == SDDbgOperand::VREG) {
       VReg = DbgOperand.getVReg();
 
       // No definition means that block hasn't been emitted yet. Leave a vreg
       // reference to be fixed later.
-    if (!MRI->hasOneDef(VReg))
-      return EmitHalfDoneInstrRef(VReg);
+      if (!MRI->hasOneDef(VReg)) {
+        AddVRegOp(VReg);
+        continue;
+      }
 
       DefMI = &*MRI->def_instr_begin(VReg);
-  } else {
-    assert(DbgOperand.getKind() == SDDbgOperand::SDNODE);
+    } else if (DbgOperand.getKind() == SDDbgOperand::SDNODE) {
       // Look up the corresponding VReg for the given SDNode, if any.
       SDNode *Node = DbgOperand.getSDNode();
       SDValue Op = SDValue(Node, DbgOperand.getResNo());
       DenseMap<SDValue, Register>::iterator I = VRBaseMap.find(Op);
       // No VReg -> produce a DBG_VALUE $noreg instead.
-    if (I==VRBaseMap.end())
-      return EmitDbgNoLocation(SD);
+      if (I == VRBaseMap.end())
+        break;
 
       // Try to pick out a defining instruction at this point.
       VReg = getVR(Op, VRBaseMap);
 
       // Again, if there's no instruction defining the VReg right now, fix it up
       // later.
-    if (!MRI->hasOneDef(VReg))
-      return EmitHalfDoneInstrRef(VReg);
+      if (!MRI->hasOneDef(VReg)) {
+        AddVRegOp(VReg);
+        continue;
+      }
 
       DefMI = &*MRI->def_instr_begin(VReg);
+    } else {
+      assert(DbgOperand.getKind() == SDDbgOperand::CONST);
+      MOs.push_back(GetMOForConstDbgOp(DbgOperand));
+      continue;
     }
 
     // Avoid copy like instructions: they don't define values, only move them.
-  // Leave a virtual-register reference until it can be fixed up later, to find
-  // the underlying value definition.
-  if (DefMI->isCopyLike() || TII->isCopyInstr(*DefMI))
-    return EmitHalfDoneInstrRef(VReg);
+    // Leave a virtual-register reference until it can be fixed up later, to
+    // find the underlying value definition.
+    if (DefMI->isCopyLike() || TII->isCopyInstr(*DefMI)) {
+      AddVRegOp(VReg);
+      continue;
+    }
 
     // Find the operand number which defines the specified VReg.
     unsigned OperandIdx = 0;
     for (const auto &MO : DefMI->operands()) {
       if (MO.isReg() && MO.isDef() && MO.getReg() == VReg)
         break;
       ++OperandIdx;
     }
     assert(OperandIdx < DefMI->getNumOperands());
 
-  // Only convert Expr to variadic form when we're sure we're emitting a
-  // complete instruction reference.
-  if (!SD->isVariadic())
-    Expr = DIExpression::convertToVariadicExpression(Expr);
     // Make the DBG_INSTR_REF refer to that instruction, and that operand.
     unsigned InstrNum = DefMI->getDebugInstrNum();
-  SmallVector<MachineOperand, 1> MOs(
-      {MachineOperand::CreateDbgInstrRef(InstrNum, OperandIdx)});
+    MOs.push_back(MachineOperand::CreateDbgInstrRef(InstrNum, OperandIdx));
+  }
+
+  // If we haven't created a valid MachineOperand for every DbgOp, abort and
+  // produce an undef DBG_VALUE.
+  if (MOs.size() != OpCount)
+    return EmitDbgNoLocation(SD);
+
   return BuildMI(*MF, DL, RefII, false, MOs, Var, Expr);
 }
 
 MachineInstr *InstrEmitter::EmitDbgNoLocation(SDDbgValue *SD) {
   // An invalidated SDNode must generate an undef DBG_VALUE: although the
   // original value is no longer computed, earlier DBG_VALUEs live ranges
   // must not leak into later code.
+  DIVariable *Var = SD->getVariable();
+  const DIExpression *Expr =
+      DIExpression::convertToUndefExpression(SD->getExpression());
+  DebugLoc DL = SD->getDebugLoc();
+  const MCInstrDesc &Desc = TII->get(TargetOpcode::DBG_VALUE);
+  return BuildMI(*MF, DL, Desc, false, 0U, Var, Expr);
+}
+
+MachineInstr *
+InstrEmitter::EmitDbgValueList(SDDbgValue *SD,
+                               DenseMap<SDValue, Register> &VRBaseMap) {
   MDNode *Var = SD->getVariable();
-  MDNode *Expr = SD->getExpression();
+  DIExpression *Expr = SD->getExpression();
   DebugLoc DL = SD->getDebugLoc();
-  auto MIB = BuildMI(*MF, DL, TII->get(TargetOpcode::DBG_VALUE));
-  MIB.addReg(0U);
-  MIB.addReg(0U);
+  // DBG_VALUE_LIST := "DBG_VALUE_LIST" var, expression, loc (, loc)*
+  const MCInstrDesc &DbgValDesc = TII->get(TargetOpcode::DBG_VALUE_LIST);
+  // Build the DBG_VALUE_LIST instruction base.
+  auto MIB = BuildMI(*MF, DL, DbgValDesc);
   MIB.addMetadata(Var);
   MIB.addMetadata(Expr);
+  AddDbgValueLocationOps(MIB, DbgValDesc, SD->getLocationOps(), VRBaseMap);
   return &*MIB;
 }
 
 MachineInstr *
 InstrEmitter::EmitDbgValueFromSingleOp(SDDbgValue *SD,
                                        DenseMap<SDValue, Register> &VRBaseMap) {
   MDNode *Var = SD->getVariable();
   DIExpression *Expr = SD->getExpression();
   DebugLoc DL = SD->getDebugLoc();
   const MCInstrDesc &II = TII->get(TargetOpcode::DBG_VALUE);
 
   assert(SD->getLocationOps().size() == 1 &&
          "Non variadic dbg_value should have only one location op");
 
   // See about constant-folding the expression.
   // Copy the location operand in case we replace it.

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

if (Reg.isVirtual() && MF.useDebugInstrRef()) {
           /* isUndef */ false, /* isEarlyClobber */ false,
           /* SubReg */ 0, /* isDebug */ true)});
 
       auto *NewDIExpr = FragExpr;
       // We don't have an "Indirect" field in DBG_INSTR_REF, fold that into
       // the DIExpression.
       if (Indirect)
         NewDIExpr = DIExpression::prepend(FragExpr, DIExpression::DerefBefore);
+      SmallVector<uint64_t, 2> Ops({dwarf::DW_OP_LLVM_arg, 0});
+      NewDIExpr = DIExpression::prependOpcodes(NewDIExpr, Ops);
       return BuildMI(MF, DL, Inst, false, MOs, Variable, NewDIExpr);
     } else {
       // Create a completely standard DBG_VALUE.
       auto &Inst = TII->get(TargetOpcode::DBG_VALUE);
       return BuildMI(MF, DL, Inst, Indirect, Reg, Variable, FragExpr);
     }
   };
 

llvm/lib/IR/DebugInfoMetadata.cpp

if (ExprOpBegin->getOp() == dwarf::DW_OP_LLVM_arg)
     ++ExprOpBegin;
 
   return !std::any_of(ExprOpBegin, ExprOpEnd, [](auto Op) {
     return Op.getOp() == dwarf::DW_OP_LLVM_arg;
   });
 }
 
 const DIExpression *
+DIExpression::convertToUndefExpression(const DIExpression *Expr) {
+  SmallVector<uint64_t, 3> UndefOps;
+  if (auto FragmentInfo = Expr->getFragmentInfo()) {
+    UndefOps.append({dwarf::DW_OP_LLVM_fragment, FragmentInfo->OffsetInBits,
+                     FragmentInfo->SizeInBits});
+  }
+  return DIExpression::get(Expr->getContext(), UndefOps);
+}
+
+const DIExpression *
 DIExpression::convertToVariadicExpression(const DIExpression *Expr) {
   if (any_of(Expr->expr_ops(), [](auto ExprOp) {
         return ExprOp.getOp() == dwarf::DW_OP_LLVM_arg;
       }))
     return Expr;
   SmallVector<uint64_t> NewOps;
   NewOps.reserve(Expr->getNumElements() + 2);
   NewOps.append({dwarf::DW_OP_LLVM_arg, 0});

llvm/test/CodeGen/X86/pr57673.ll

-; RUN: llc -mtriple=x86_64-unknown-unknown -mcpu=x86-64 -stop-after=x86-optimize-LEAs < %s | FileCheck %s
+; RUN: llc -mtriple=x86_64-unknown-unknown -mcpu=x86-64 -stop-after=x86-optimize-LEAs -experimental-debug-variable-locations=false < %s \
+; RUN:   | FileCheck %s --check-prefix=NORMAL
+; RUN: llc -mtriple=x86_64-unknown-unknown -mcpu=x86-64 -stop-after=x86-optimize-LEAs -experimental-debug-variable-locations < %s \
+; RUN:   | FileCheck %s --check-prefix=INSTRREF
 
 ; The LEA optimization pass used to crash on this testcase.
 
 ; This test case used to trigger:
 ;
 ;   assert(MRI->use_empty(LastVReg) &&
 ;          "The LEA's def register must have no uses");
 
 ; CHECK:     LEA64r
 ; CHECK-NOT: LEA64r
-; CHECK:     DBG_VALUE_LIST
+; NORMAL:    DBG_VALUE_LIST
+; INSTRREF:  DBG_INSTR_REF
 
 target triple = "x86_64-unknown-linux-gnu"
 
 %t10 = type { ptr, [32 x i8] }
 
 define void @foo() {
 bb_entry:
   %tmp11 = alloca [0 x [0 x i32]], i32 0, align 4

llvm/test/DebugInfo/X86/arg-dbg-value-list.ll

-;; FIXME: We currently don't do any special bookkeeping for unused args used by
-;; variadic dbg_values. When/if we support them, the DBG_VALUE_LIST should be
-;; updated accordingly.
-
-; RUN: llc %s -start-after=codegenprepare -stop-before=finalize-isel -o - | FileCheck %s
-
-;; Check that unused argument values are handled the same way for variadic
-;; dbg_values as non-variadics.
-
-; CHECK: ![[A:[0-9]+]] = !DILocalVariable(name: "a",
-; CHECK: ![[B:[0-9]+]] = !DILocalVariable(name: "b",
-; CHECK: ![[C:[0-9]+]] = !DILocalVariable(name: "c",
-
-; CHECK: DBG_VALUE $ecx, $noreg, ![[A]], !DIExpression(), debug-location
-; CHECK: DBG_VALUE $edx, $noreg, ![[B]], !DIExpression(), debug-location
-; CHECK: DBG_VALUE $noreg, $noreg, ![[C]], !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus, DW_OP_stack_value), debug-location
-
-target triple = "x86_64-pc-windows-msvc19.16.27034"
-define dso_local i32 @"?foo@@YAHHH@Z"(i32 %a, i32 %b) local_unnamed_addr !dbg !8 {
-entry:
-  call void @llvm.dbg.value(metadata i32 %a, metadata !14, metadata !DIExpression()), !dbg !17
-  call void @llvm.dbg.value(metadata i32 %b, metadata !15, metadata !DIExpression()), !dbg !17
-  call void @llvm.dbg.value(metadata !DIArgList(i32 %a, i32 %b), metadata !16, metadata !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus, DW_OP_stack_value)), !dbg !17
-  ret i32 0
-}
-
-declare void @llvm.dbg.value(metadata, metadata, metadata)
-
-!llvm.dbg.cu = !{!0}
-!llvm.module.flags = !{!3, !4, !5, !6}
-!llvm.ident = !{!7}
-
-!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !1, producer: "clang version 11.0.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, nameTableKind: None)
-!1 = !DIFile(filename: "test.cpp", directory: "/")
-!2 = !{}
-!3 = !{i32 2, !"CodeView", i32 1}
-!4 = !{i32 2, !"Debug Info Version", i32 3}
-!5 = !{i32 1, !"wchar_size", i32 2}
-!6 = !{i32 7, !"PIC Level", i32 2}
-!7 = !{!"clang version 11.0.0"}
-!8 = distinct !DISubprogram(name: "foo", linkageName: "?foo@@YAHHH@Z", scope: !9, file: !9, line: 1, type: !10, scopeLine: 1, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !13)
-!9 = !DIFile(filename: "test.cpp", directory: "/")
-!10 = !DISubroutineType(types: !11)
-!11 = !{!12, !12, !12}
-!12 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!13 = !{!14, !15, !16}
-!14 = !DILocalVariable(name: "a", arg: 1, scope: !8, file: !9, line: 1, type: !12)
-!15 = !DILocalVariable(name: "b", arg: 2, scope: !8, file: !9, line: 1, type: !12)
-!16 = !DILocalVariable(name: "c", scope: !8, file: !9, line: 2, type: !12)
-!17 = !DILocation(line: 0, scope: !8)
+;; FIXME: We currently don't do any special bookkeeping for unused args used by
+;; variadic dbg_values. When/if we support them, the DBG_VALUE_LIST should be
+;; updated accordingly.
+
+; RUN: llc %s -start-after=codegenprepare --experimental-debug-variable-locations=false -stop-before=finalize-isel -o - | FileCheck %s
+; RUN: llc %s -start-after=codegenprepare --experimental-debug-variable-locations -stop-before=finalize-isel -o - | FileCheck %s
+
+;; Check that unused argument values are handled the same way for variadic
+;; dbg_values as non-variadics.
+
+; CHECK: ![[A:[0-9]+]] = !DILocalVariable(name: "a",
+; CHECK: ![[B:[0-9]+]] = !DILocalVariable(name: "b",
+; CHECK: ![[C:[0-9]+]] = !DILocalVariable(name: "c",
+
+; CHECK: DBG_VALUE $ecx, $noreg, ![[A]], !DIExpression(), debug-location
+; CHECK: DBG_VALUE $edx, $noreg, ![[B]], !DIExpression(), debug-location
+; CHECK: DBG_VALUE $noreg, $noreg, ![[C]], !DIExpression(), debug-location
+
+target triple = "x86_64-pc-windows-msvc19.16.27034"
+define dso_local i32 @"?foo@@YAHHH@Z"(i32 %a, i32 %b) local_unnamed_addr !dbg !8 {
+entry:
+  call void @llvm.dbg.value(metadata i32 %a, metadata !14, metadata !DIExpression()), !dbg !17
+  call void @llvm.dbg.value(metadata i32 %b, metadata !15, metadata !DIExpression()), !dbg !17
+  call void @llvm.dbg.value(metadata !DIArgList(i32 %a, i32 %b), metadata !16, metadata !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus, DW_OP_stack_value)), !dbg !17
+  ret i32 0
+}
+
+declare void @llvm.dbg.value(metadata, metadata, metadata)
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3, !4, !5, !6}
+!llvm.ident = !{!7}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !1, producer: "clang version 11.0.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, nameTableKind: None)
+!1 = !DIFile(filename: "test.cpp", directory: "/")
+!2 = !{}
+!3 = !{i32 2, !"CodeView", i32 1}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = !{i32 1, !"wchar_size", i32 2}
+!6 = !{i32 7, !"PIC Level", i32 2}
+!7 = !{!"clang version 11.0.0"}
+!8 = distinct !DISubprogram(name: "foo", linkageName: "?foo@@YAHHH@Z", scope: !9, file: !9, line: 1, type: !10, scopeLine: 1, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !13)
+!9 = !DIFile(filename: "test.cpp", directory: "/")
+!10 = !DISubroutineType(types: !11)
+!11 = !{!12, !12, !12}
+!12 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!13 = !{!14, !15, !16}
+!14 = !DILocalVariable(name: "a", arg: 1, scope: !8, file: !9, line: 1, type: !12)
+!15 = !DILocalVariable(name: "b", arg: 2, scope: !8, file: !9, line: 1, type: !12)
+!16 = !DILocalVariable(name: "c", scope: !8, file: !9, line: 2, type: !12)
+!17 = !DILocation(line: 0, scope: !8)

llvm/test/DebugInfo/X86/dbg-val-list-undef.ll

-; RUN: llc %s -start-after=codegenprepare -stop-before=finalize-isel -o - | FileCheck %s
-
-;; %y is unused and cannot (FIXME: currently) be salvaged. Ensure that the
-;; variadic dbg_value using %y becomes undef.
-
-; CHECK: ![[C:[0-9]+]] = !DILocalVariable(name: "c",
-; CHECK: DBG_VALUE $noreg, $noreg, ![[C]], !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus, DW_OP_stack_value),  debug-location
-
-target triple = "x86_64-pc-windows-msvc19.16.27034"
-define dso_local i32 @"?foo@@YAHHH@Z"(i32 %a, i32 %b) local_unnamed_addr !dbg !8 {
-entry:
-  %x = add i32 %a, %b
-  %y = mul i32 %x, %b
-  call void @llvm.dbg.value(metadata !DIArgList(i32 %x, i32 %y), metadata !16, metadata !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus, DW_OP_stack_value)), !dbg !17
-  ret i32 %x
-}
-
-declare void @llvm.dbg.value(metadata, metadata, metadata)
-
-!llvm.dbg.cu = !{!0}
-!llvm.module.flags = !{!3, !4, !5, !6}
-!llvm.ident = !{!7}
-
-!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !1, producer: "clang version 11.0.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, nameTableKind: None)
-!1 = !DIFile(filename: "test.cpp", directory: "/")
-!2 = !{}
-!3 = !{i32 2, !"CodeView", i32 1}
-!4 = !{i32 2, !"Debug Info Version", i32 3}
-!5 = !{i32 1, !"wchar_size", i32 2}
-!6 = !{i32 7, !"PIC Level", i32 2}
-!7 = !{!"clang version 11.0.0"}
-!8 = distinct !DISubprogram(name: "foo", linkageName: "?foo@@YAHHH@Z", scope: !9, file: !9, line: 1, type: !10, scopeLine: 1, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !13)
-!9 = !DIFile(filename: "test.cpp", directory: "/")
-!10 = !DISubroutineType(types: !11)
-!11 = !{!12, !12, !12}
-!12 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!13 = !{!14, !15, !16}
-!14 = !DILocalVariable(name: "a", arg: 1, scope: !8, file: !9, line: 1, type: !12)
-!15 = !DILocalVariable(name: "b", arg: 2, scope: !8, file: !9, line: 1, type: !12)
-!16 = !DILocalVariable(name: "c", scope: !8, file: !9, line: 2, type: !12)
-!17 = !DILocation(line: 0, scope: !8)
+; RUN: llc %s -start-after=codegenprepare --experimental-debug-variable-locations=false -stop-before=finalize-isel -o - | FileCheck %s
+; RUN: llc %s -start-after=codegenprepare --experimental-debug-variable-locations -stop-before=finalize-isel -o - | FileCheck %s
+
+;; %y is unused and cannot (FIXME: currently) be salvaged. Ensure that the
+;; variadic dbg_value using %y becomes undef.
+
+; CHECK: ![[C:[0-9]+]] = !DILocalVariable(name: "c",
+; CHECK: DBG_VALUE $noreg, $noreg, ![[C]], !DIExpression(), debug-location
+
+target triple = "x86_64-pc-windows-msvc19.16.27034"
+define dso_local i32 @"?foo@@YAHHH@Z"(i32 %a, i32 %b) local_unnamed_addr !dbg !8 {
+entry:
+  %x = add i32 %a, %b
+  %y = mul i32 %x, %b
+  call void @llvm.dbg.value(metadata !DIArgList(i32 %x, i32 %y), metadata !16, metadata !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus, DW_OP_stack_value)), !dbg !17
+  ret i32 %x
+}
+
+declare void @llvm.dbg.value(metadata, metadata, metadata)
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3, !4, !5, !6}
+!llvm.ident = !{!7}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !1, producer: "clang version 11.0.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, nameTableKind: None)
+!1 = !DIFile(filename: "test.cpp", directory: "/")
+!2 = !{}
+!3 = !{i32 2, !"CodeView", i32 1}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = !{i32 1, !"wchar_size", i32 2}
+!6 = !{i32 7, !"PIC Level", i32 2}
+!7 = !{!"clang version 11.0.0"}
+!8 = distinct !DISubprogram(name: "foo", linkageName: "?foo@@YAHHH@Z", scope: !9, file: !9, line: 1, type: !10, scopeLine: 1, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !13)
+!9 = !DIFile(filename: "test.cpp", directory: "/")
+!10 = !DISubroutineType(types: !11)
+!11 = !{!12, !12, !12}
+!12 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!13 = !{!14, !15, !16}
+!14 = !DILocalVariable(name: "a", arg: 1, scope: !8, file: !9, line: 1, type: !12)
+!15 = !DILocalVariable(name: "b", arg: 2, scope: !8, file: !9, line: 1, type: !12)
+!16 = !DILocalVariable(name: "c", scope: !8, file: !9, line: 2, type: !12)
+!17 = !DILocation(line: 0, scope: !8)

llvm/test/DebugInfo/X86/instr-ref-sdag-empty-vreg.ll

 ; RUN: llc %s -o - -stop-before=finalize-isel \
 ; RUN:  -experimental-debug-variable-locations \
 ; RUN: | FileCheck %s
 
 ;; The multiply by zero of %width below causes %width to become unused very late
 ;; in SelectionDAG. As a result, DBG_INSTR_REFs are produced that refer to a
 ;; vreg that is never defined, which risks a crash. Check that we don't crash,
 ;; and produce an empty variable location.
 
-; CHECK: DBG_VALUE_LIST {{.+}}, $noreg
+; CHECK: DBG_VALUE_LIST {{.+}}, $noreg
+; CHECK: DBG_VALUE_LIST {{.+}}, $noreg, 4
 
 target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-unknown"
 
 %class.Color = type { i8 }
 
 define hidden void @_Z14drawXZWideLineR4Vec3RK5Colorf(ptr %color, float %width) local_unnamed_addr !dbg !7 {
 cond.false.i:
   br label %_ZN4Vec39normalizeEv.exit, !dbg !12
 
 _ZN4Vec39normalizeEv.exit:                        ; preds = %cond.false.i
   call void @llvm.dbg.value(metadata float %width, metadata !11, metadata !DIExpression()), !dbg !12
+  call void @llvm.dbg.value(metadata !DIArgList(float %width, i32 4), metadata !11, metadata !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_mul, DW_OP_stack_value)), !dbg !12
   %mul.i = fmul float %width, 0.000000e+00, !dbg !12
   ret void, !dbg !12
 }
 
 ; Function Attrs: nofree nosync nounwind readnone speculatable willreturn
 declare void @llvm.dbg.value(metadata, metadata, metadata)
 
 !llvm.dbg.cu = !{!0}

llvm/test/DebugInfo/X86/instr-ref-selectiondag.ll

 
 ; Test that SelectionDAG produces DBG_VALUEs normally, but DBG_INSTR_REFs when
 ; asked.
 
 ; NORMAL-LABEL: name: foo
 
 ; NORMAL:      %[[REG0:[0-9]+]]:gr32 = ADD32rr
 ; NORMAL-NEXT: DBG_VALUE %[[REG0]]
+; NORMAL-NEXT: DBG_VALUE_LIST {{.+}}, %[[REG0]], 2
 ; NORMAL-NEXT: %[[REG1:[0-9]+]]:gr32 = ADD32rr
 ; NORMAL-NEXT: DBG_VALUE %[[REG1]]
+; NORMAL-NEXT: DBG_VALUE_LIST {{.+}}, %[[REG1]], %[[REG0]]
 
 ; Note that I'm baking in an assumption of one-based ordering here. We could
 ; capture and check for the instruction numbers, we'd rely on machine verifier
 ; ensuring there were no duplicates.
 
 ; INSTRREF-LABEL: name: foo
 
 ; INSTRREF:      ADD32rr
 ; INSTRREF-SAME: debug-instr-number 1
 ; INSTRREF-NEXT: DBG_INSTR_REF {{.+}}, dbg-instr-ref(1, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF {{.+}}, dbg-instr-ref(1, 0), 2
 ; INSTRREF-NEXT: ADD32rr
 ; INSTRREF-SAME: debug-instr-number 2
 ; INSTRREF-NEXT: DBG_INSTR_REF {{.+}}, dbg-instr-ref(2, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF {{.+}}, dbg-instr-ref(2, 0), dbg-instr-ref(1, 0)
 
-; Test that fast-isel will produce DBG_INSTR_REFs too.
+; Test that fast-isel will produce DBG_INSTR_REFs too, except for debug values
+; using DIArgList, which is not supported in FastIsel.
 
 ; FASTISEL-INSTRREF-LABEL: name: foo
 
 ; FASTISEL-INSTRREF:      ADD32rr
 ; FASTISEL-INSTRREF-SAME: debug-instr-number 1
 ; FASTISEL-INSTRREF-NEXT: DBG_INSTR_REF {{.+}}, dbg-instr-ref(1, 0)
+; FASTISEL-INSTRREF-NEXT: DBG_VALUE $noreg, {{.+}}
 ; FASTISEL-INSTRREF-NEXT: ADD32rr
 ; FASTISEL-INSTRREF-SAME: debug-instr-number 2
 ; FASTISEL-INSTRREF-NEXT: DBG_INSTR_REF {{.+}}, dbg-instr-ref(2, 0)
+; FASTISEL-INSTRREF-NEXT: DBG_VALUE $noreg, {{.+}}
 
 @glob32 = global i32 0
 @glob16 = global i16 0
 @glob8 = global i8 0
 
 declare void @llvm.dbg.value(metadata, metadata, metadata)
 declare void @llvm.dbg.declare(metadata, metadata, metadata)
 
 define i32 @foo(i32 %bar, i32 %baz, i32 %qux) !dbg !7 {
 entry:
   %0 = add i32 %bar, %baz, !dbg !14
   call void @llvm.dbg.value(metadata i32 %0, metadata !13, metadata !DIExpression()), !dbg !14
+  call void @llvm.dbg.value(metadata !DIArgList(i32 %0, i32 2), metadata !13, metadata !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus, DW_OP_stack_value)), !dbg !14
   %1 = add i32 %0, %qux
   call void @llvm.dbg.value(metadata i32 %1, metadata !13, metadata !DIExpression()), !dbg !14
+  call void @llvm.dbg.value(metadata !DIArgList(i32 %1, i32 %0), metadata !13, metadata !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus, DW_OP_stack_value)), !dbg !14
   ret i32 %1, !dbg !14
 }
 
 ; In the code below, isel produces a large number of copies between subregisters
 ; to represent the gradually decreasing width of the argument. This gets
 ; optimized away into three stores, but it's an objective of the instruction
 ; referencing design that COPYs are not numbered: they move values, not define
 ; them. Test that nothing is numbered, and instead that appropriate

 shoes:
   %moo = trunc i64 %foo to i16
   br label %slippers
 }
 
 ;; Test for dbg.declare of non-stack-slot Values. These turn up with NRVO and 
 ;; other ABI scenarios where something is technically in memory, but we don't
 ;; refer to it relative to the stack pointer. We refer to these either with an
-;; indirect DBG_VAUE, or a DBG_INSTR_REF with DW_OP_deref prepended.
+;; indirect DBG_VALUE, or a DBG_INSTR_REF with DW_OP_deref prepended.
 ;;
 ;; Test an inlined dbg.declare in a different scope + block, to test behaviours
 ;; where the debug intrinsic isn't in the first block. The normal-mode DBG_VALUE
 ;; is hoisted into the entry block for that. This is fine because the variable
 ;; location is never re-assigned. (FIXME: do we scope-trim / fail-to-propagate
 ;; these hoisted locations later?).
 
 ; NORMAL-LABEL: name: qux

llvm/test/DebugInfo/X86/invalidated-dbg-value-is-undef.ll

 ; RUN: llc %s --stop-before=finalize-isel -o - | FileCheck %s --implicit-check-not=DBG_VALUE
 
 ;; Check that when a debug value is invalidated during Instruction Selection,
 ;; we produce an undef DBG_VALUE/DBG_VALUE_LIST.
 
 ; CHECK-LABEL: body:
 
-; CHECK: DBG_VALUE $noreg, $noreg, ![[VAR:[0-9]+]]
+; CHECK: DBG_VALUE $noreg, $noreg, ![[VAR:[0-9]+]], !DIExpression()
 
 target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 @intel_pmu_enable_bts_config = external dso_local local_unnamed_addr global i32, align 4, !dbg !0
 
 define dso_local i32 @intel_pmu_enable_bts() local_unnamed_addr !dbg !16 {
 entry:

llvm/test/DebugInfo/X86/safestack-byval.ll

 ; Test dwarf codegen for DILocalVariable of a byval function argument that
 ; points to neither an argument nor an alloca. This kind of IR is generated by
 ; SafeStack for unsafe byval arguments.
-; RUN: llc -mtriple=x86_64-unknown-unknown -stop-after finalize-isel %s -o - | FileCheck %s
+; RUN: llc -mtriple=x86_64-unknown-unknown --experimental-debug-variable-locations=false -stop-after finalize-isel %s -o - | FileCheck %s --check-prefixes=CHECK,NORMAL
+; RUN: llc -mtriple=x86_64-unknown-unknown --experimental-debug-variable-locations -stop-after finalize-isel %s -o - | FileCheck %s --check-prefixes=CHECK,INSTRREF
 
 ; This was built by compiling the following source with SafeStack and
 ; simplifying the result a little.
 ; struct S {
 ;   int a[100];
 ; };
 ;
 ; int f(S zzz, unsigned long len) {
 ;   return zzz.a[len];
 ; }
 
-; CHECK: ![[ZZZ:.*]] = !DILocalVariable(name: "zzz",
-; CHECK: DBG_VALUE {{.*}} ![[ZZZ]], !DIExpression(DW_OP_deref, DW_OP_constu, 400, DW_OP_minus)
+; CHECK:    ![[ZZZ:.*]] = !DILocalVariable(name: "zzz",
+; NORMAL:   DBG_VALUE {{.*}} ![[ZZZ]], !DIExpression(DW_OP_deref, DW_OP_constu, 400, DW_OP_minus)
+; INSTRREF: DBG_INSTR_REF ![[ZZZ]], !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_deref, DW_OP_constu, 400, DW_OP_minus, DW_OP_deref)
 
 %struct.S = type { [100 x i32] }
 
 @__safestack_unsafe_stack_ptr = external thread_local(initialexec) global ptr
 
 ; Function Attrs: norecurse nounwind readonly safestack uwtable
 define i32 @_Z1f1Sm(ptr byval(%struct.S) nocapture readonly align 8 %zzz, i64 %len) #0 !dbg !12 {
 entry:

llvm/unittests/IR/MetadataTest.cpp

#define EXPECT_INVALID_FRAGMENT(Offset, Size, ...)                             \
   // deref in the expression).
   EXPECT_INVALID_FRAGMENT(0, 32, dwarf::DW_OP_deref, dwarf::DW_OP_plus_uconst,
                           2, dwarf::DW_OP_stack_value);
 
 #undef EXPECT_VALID_FRAGMENT
 #undef EXPECT_INVALID_FRAGMENT
 }
 
+TEST_F(DIExpressionTest, convertToUndefExpression) {
+#define EXPECT_UNDEF_OPS_EQUAL(TestExpr, Expected)                             \
+  do {                                                                         \
+    const DIExpression *Undef =                                                \
+        DIExpression::convertToUndefExpression(TestExpr);                      \
+    EXPECT_EQ(Undef, Expected);                                                \
+  } while (false)
+#define GET_EXPR(...) DIExpression::get(Context, {__VA_ARGS__})
+
+  // Expressions which are single-location and non-complex should be unchanged.
+  EXPECT_UNDEF_OPS_EQUAL(GET_EXPR(), GET_EXPR());
+  EXPECT_UNDEF_OPS_EQUAL(GET_EXPR(dwarf::DW_OP_LLVM_fragment, 0, 32),
+                         GET_EXPR(dwarf::DW_OP_LLVM_fragment, 0, 32));
+
+  // Variadic expressions should become single-location.
+  EXPECT_UNDEF_OPS_EQUAL(GET_EXPR(dwarf::DW_OP_LLVM_arg, 0), GET_EXPR());
+  EXPECT_UNDEF_OPS_EQUAL(
+      GET_EXPR(dwarf::DW_OP_LLVM_arg, 0, dwarf::DW_OP_LLVM_fragment, 32, 32),
+      GET_EXPR(dwarf::DW_OP_LLVM_fragment, 32, 32));
+  EXPECT_UNDEF_OPS_EQUAL(GET_EXPR(dwarf::DW_OP_LLVM_arg, 0,
+                                  dwarf::DW_OP_LLVM_arg, 1, dwarf::DW_OP_mul),
+                         GET_EXPR());
+  EXPECT_UNDEF_OPS_EQUAL(GET_EXPR(dwarf::DW_OP_LLVM_arg, 0,
+                                  dwarf::DW_OP_LLVM_arg, 1, dwarf::DW_OP_mul,
+                                  dwarf::DW_OP_LLVM_fragment, 64, 32),
+                         GET_EXPR(dwarf::DW_OP_LLVM_fragment, 64, 32));
+
+  // Any stack-computing ops should be removed.
+  EXPECT_UNDEF_OPS_EQUAL(GET_EXPR(dwarf::DW_OP_plus_uconst, 8), GET_EXPR());
+  EXPECT_UNDEF_OPS_EQUAL(
+      GET_EXPR(dwarf::DW_OP_plus_uconst, 8, dwarf::DW_OP_LLVM_fragment, 0, 16),
+      GET_EXPR(dwarf::DW_OP_LLVM_fragment, 0, 16));
+  EXPECT_UNDEF_OPS_EQUAL(GET_EXPR(dwarf::DW_OP_constu, 24, dwarf::DW_OP_shra),
+                         GET_EXPR());
+  EXPECT_UNDEF_OPS_EQUAL(GET_EXPR(dwarf::DW_OP_constu, 24, dwarf::DW_OP_shra,
+                                  dwarf::DW_OP_LLVM_fragment, 8, 16),
+                         GET_EXPR(dwarf::DW_OP_LLVM_fragment, 8, 16));
+  EXPECT_UNDEF_OPS_EQUAL(GET_EXPR(dwarf::DW_OP_deref), GET_EXPR());
+  EXPECT_UNDEF_OPS_EQUAL(
+      GET_EXPR(dwarf::DW_OP_deref, dwarf::DW_OP_LLVM_fragment, 16, 16),
+      GET_EXPR(dwarf::DW_OP_LLVM_fragment, 16, 16));
+  EXPECT_UNDEF_OPS_EQUAL(GET_EXPR(dwarf::DW_OP_constu, 4, dwarf::DW_OP_minus),
+                         GET_EXPR());
+  EXPECT_UNDEF_OPS_EQUAL(GET_EXPR(dwarf::DW_OP_constu, 4, dwarf::DW_OP_minus,
+                                  dwarf::DW_OP_LLVM_fragment, 24, 16),
+                         GET_EXPR(dwarf::DW_OP_LLVM_fragment, 24, 16));
+
+  // Stack-value operators are also not preserved.
+  EXPECT_UNDEF_OPS_EQUAL(
+      GET_EXPR(dwarf::DW_OP_plus_uconst, 8, dwarf::DW_OP_stack_value),
+      GET_EXPR());
+  EXPECT_UNDEF_OPS_EQUAL(GET_EXPR(dwarf::DW_OP_plus_uconst, 8,
+                                  dwarf::DW_OP_stack_value,
+                                  dwarf::DW_OP_LLVM_fragment, 32, 16),
+                         GET_EXPR(dwarf::DW_OP_LLVM_fragment, 32, 16));
+
+#undef EXPECT_UNDEF_OPS_EQUAL
+#undef GET_EXPR
+}
+
 TEST_F(DIExpressionTest, convertToVariadicExpression) {
 #define EXPECT_CONVERT_IS_NOOP(TestExpr)                                       \
   do {                                                                         \
     const DIExpression *Variadic =                                             \
         DIExpression::convertToVariadicExpression(TestExpr);                   \
     EXPECT_EQ(Variadic, TestExpr);                                             \
   } while (false)
 #define EXPECT_VARIADIC_OPS_EQUAL(TestExpr, Expected)                          \