[DebugInfo] Process DBG_VALUE_LIST in LiveDebugVariables

[StephenTozer]

Summary

Continuing the work discussed in the RFC[0], this patch implements DBG_VALUE_LIST handling in the LiveDebugVariables pass.

This is one of the larger pass changes involved with handling DBG_VALUE_LIST, second only to LiveDebugValues, as this pass is involved quite in-depth with debug values. The majority of the changes made here are relatively simple, however, largely falling into two categories:

The largest logical changes take place in computeIntervals, extendDef, and addDefsFromCopies; when extending the def of a DBG_VALUE_LIST the live ranges of every used register must be considered, and when such a def is killed by more than one of its used registers being killed at the same time it is necessary to find valid copies of all of those registers to create a new def with.

Many of the remaining changes concern the interface of DebugVariableValue, which has been modified to support DBG_VALUE_LIST. Besides using a list of location numbers instead of just one, some of the functions have also changed; changeLocNo now requires the old location number that is being replaced, remapLocNos is used when more than one location needs to be changed at once, and decrementLocNosAfterPivot is used to decrement location numbers when a location is deleted and the array indices shift. Also, the constructor of DebugVariableValue is used to remove any duplicate locations in the location list; these are manageable in the instructions themselves, but cause problems with the kind of in-depth analysis that LiveDebugVariables does.

[0] http://lists.llvm.org/pipermail/llvm-dev/2020-February/139376.html

[StephenTozer]

Remove unused MF attributes.

[Orlando]

Hi, sorry for leaving this for so long! I think it will take me a couple of passes to fully understand it, but for now I left some inline nits/questions. Please feel free to leave the nit changes until later revisions.

I'm hoping you can clarify something about addDefsFromCopies for me. IIUC without the patch we are adding defs from copies at kill points if possible. And with the patch a def is only added on the _first_ kill. Is that right?

llvm/lib/CodeGen/LiveDebugVariables.cpp

200–1

It looks like this class was trying to be small on purpose, and it is now relatively much larger. I don't know if it will make a significant impact or not, have you managed to run any kind of performance tests with this patch yet?

200–1

Please can you update this comment, and the one for addDefsFromCopies?

200–1

Is it not possible to do this instead?

LLVM_DEBUG(dbgs() << "\t[" << Start << ';' << Stop << "):";
           DbgValue.printLocNos(dbgs()));

201–202

IMO it'd be nice to make these default = false and remove the default constructor.

202

IIRC you mentioned that DBG_VALUE_LIST don't have the 'IsIndirect' field. It may be worth adding an assert here that !(WasIndirect && WasList)?

202–205

This if looks a bit gnarly to me. I would personally prefer a couple of simpler early exit checks instead, though I'm not sure if there's anything in the style guide to mandate it.

[StephenTozer]

Rebased onto recent master; no functional changes from rebase.

Also changed DbgVariableValue to hold a pointer to a vector of location numbers, instead of containing a SmallVector of the same. This change was necessary to reduce the size of the DbgVariableValue class so that it can be used with an IntervalMap without error. This solution, however, is most likely not optimal in terms of efficiency. As such it is likely that this implementation will need a change; I have uploaded it as-is for two reasons: firstly to ensure that we have a working stack of patches available immediately, and secondly to allow review for the current implementation as most of the changes will not be affected by a future fix unless it involves a total rewrite of LiveDebugVariables.

[StephenTozer]

In D83895#2225768, @Orlando wrote:

I'm hoping you can clarify something about addDefsFromCopies for me. IIUC without the patch we are adding defs from copies at kill points if possible. And with the patch a def is only added on the _first_ kill. Is that right?

That's what it says on the tin, but after actually examining the code that calls addDefsFromCopies it appears that we never passed more than one kill point before this change, hence the slight rewrite. More specifically, addDefsFromCopies is only called in one place; in that place, the Kills vector is declared locally and then passed to extendDef immediately before that call, and extendDef always returns immediately after adding a single item to Kills, meaning that it's really more of an optional than a vector.

llvm/lib/CodeGen/LiveDebugVariables.cpp

200–1

I did notice this when changing the class, and I share the concerns. Performance testing this implementation is in-progress, but I imagine that if the size is a concern then it should be sufficient to shrink the size of the array. In most cases we only actually need one element, so compared to the old class size size of 12 bytes with 64-bit pointers, this would be 21 bytes - an increase, but hopefully not drastic. If this turns out to still cause a major performance hit though, then it will probably need a more drastic redesign. I'm optimistic here however, as LiveDebugVariables is not usually a performance bottleneck (compared to LiveDebugValues).

200–1

When the ostream returned by dbgs() falls out of scope, it adds a banner (with a newline) and flushes output; we only want this to happen once for the entire print statement here, so we need to assign it to a local variable so that we can use the same object between statements.

[StephenTozer]

Rebased onto recent master, fixed test failures.

[StephenTozer]

Ping.

[jmorse]

Looking good -- I think there are some test gaps, is there coverage for:

• extendDef for a DBG_VALUE_VAR -- As I understand the -movements test, it's designed for range splitting?
• DBG_VALUE_VAR with a $noreg operand isn't propagated / split
• Reduction / replaceArg of duplicate DBG_VALUE_VAR operands

In general I think this is good -- my greatest concern is using a std::vector for DbgVariableValue, and the fact that it's cloned whenever a modification takes place. This is going to introduce a lot more allocations in the common-case, including the number-pivoting. As I understand from the history, the problem is LiveInterval having to store a huge object? Is there a plan for solving / optimising this after this patch lands? I also saw std::move is applied to DbgVariableValue in a few places, does it need an rvalue constructor for that to be effective? (I've no idea).

There's an "isReal" flag that doesn't seem to do anything and isn't documented, is this necessary?

Finally, there's the business with SpillOffset being added to expressions in insertDebugValue, but as shown in the tests, there isn't an offset in any of the spilt expressions. That'd be what I expect, because the stack layout is determined when PEI runs. Can we ditch the offset adding code and assert that it's alway zero at this stage? (Not necessary for this patch to land).

llvm/lib/CodeGen/LiveDebugVariables.cpp

200–1

Can we make these Registers now that we're not bit-packing?

200–1

The previous method had the benefit that only those elements above the pivot point were re-written -- this will rewrite all of them. Is there an easy / obvious optimisation? I reckon we can land with this and optimise later if it turns out to be a significant burden.

200–1

Can we not use an ArrayRef for the operands reference?

202–205

Seconded; 'tis nasty

204

IMO, should be a SmallVector rather than a std::vector, avoids a few additional allocs.

208

Mega-nit: a more specific name like "KilledLocIntervals" would help readability IMO.

[StephenTozer]

llvm/lib/CodeGen/LiveDebugVariables.cpp

200–1

Possibly - I'm not sure what the actual performance cost of setValueUnchecked is - I assume it'd be equivalent to a lookup, i.e. small. The performance cost of decrementLocNosAfterPivot shouldn't be too large, but copying the full object every time might be bad. The easiest way to handle this is probably to create a new method to check if(DbgValue.getMaxLocNo() > LocNo), or more efficiently (since this would be its only use) if(DbgValue.hasLocNoGreaterThan(LocNo)); it's still a more expensive check than getLocNo() since it requires iteration and comparison, but there's no way around that without caching, which isn't an option when we need to minimize object size.

200–1

These ones are actually indices, not Registers.

202–205

I think so as well, but I'm not sure I prefer the idea of having the same checks spread out over multiple lines with this repeated between all of them:

if (...) {
  LLVM_DEBUG(dbgs() << "Can't handle " << MI);
  return false;
}

[StephenTozer]

Address review comments: remove unused isReal, add hasLocNoGreaterThan to prevent unnecessary modifications, replace messy vector storage of LocNo with simple C-style array, clean up messy if condition, use default-size SmallVectors where possible, renaming and formatting.

Still TODO: Add extra tests, prevent unnecessary cloning of arrays by reusing existing ones where possible.

llvm/lib/CodeGen/LiveDebugVariables.cpp

 
 enum : unsigned { UndefLocNo = ~0U };
 
 namespace {
 /// Describes a debug variable value by location number and expression along
 /// with some flags about the original usage of the location.
 class DbgVariableValue {
 public:
-  DbgVariableValue(unsigned LocNo, bool WasIndirect,
-                   const DIExpression &Expression)
-      : LocNo(LocNo), WasIndirect(WasIndirect), Expression(&Expression) {
-    assert(getLocNo() == LocNo && "location truncation");
+  DbgVariableValue(ArrayRef<unsigned> NewLocs, bool WasIndirect, bool WasList,
+                   const DIExpression &Expr)
+      : WasIndirect(WasIndirect), WasList(WasList), Expression(&Expr) {
+    assert(!(WasIndirect && WasList) &&
+           "DBG_VALUE_LISTs should not be indirect.");
+    SmallVector<unsigned> LocNoVec;
+    for (unsigned LocNo : NewLocs) {
+      auto It = find(LocNoVec, LocNo);
+      if (It == LocNoVec.end())
+        LocNoVec.push_back(LocNo);
+      else {
+        // Loc duplicates an element in LocNos; replace references to Op
+        // with references to the duplicating element.
+        unsigned OpIdx = LocNoVec.size();
+        unsigned DuplicatingIdx = std::distance(LocNoVec.begin(), It);
+        Expression =
+            DIExpression::replaceArg(Expression, OpIdx, DuplicatingIdx);
+      }
+    }
+    // A debug value referencing 64+ unique machine locations is very likely
+    // to be the result of a bug earlier in the pipeline. If by some means this
+    // limit is validly reached, then we can add a byte to the size of
+    // LocNoCount.
+    assert(LocNoVec.size() < 64 &&
+           "debug value containing 64+ unique machine locations is not "
+           "supported by Live Debug Variables");
+    LocNoCount = LocNoVec.size();
+    if (LocNoCount > 0) {
+      LocNos.reset(new unsigned[LocNoCount]());
+      std::copy(LocNoVec.begin(), LocNoVec.end(), loc_nos_begin());
+    }
   }
 
-  DbgVariableValue() : LocNo(0), WasIndirect(0) {}
+  DbgVariableValue() : LocNoCount(0), WasIndirect(0), WasList(0) {}
+  DbgVariableValue(const DbgVariableValue &Other)
+      : LocNoCount(Other.LocNoCount), WasIndirect(Other.getWasIndirect()),
+        WasList(Other.getWasList()), Expression(Other.getExpression()) {
+    if (Other.getLocNoCount()) {
+      LocNos.reset(new unsigned[Other.getLocNoCount()]);
+      std::copy(Other.loc_nos_begin(), Other.loc_nos_end(), loc_nos_begin());
+    }
+  }
+
+  DbgVariableValue &operator=(const DbgVariableValue &Other) {
+    if (this == &Other)
+      return *this;
+    if (Other.getLocNoCount()) {
+      LocNos.reset(new unsigned[Other.getLocNoCount()]);
+      std::copy(Other.loc_nos_begin(), Other.loc_nos_end(), loc_nos_begin());
+    } else {
+      LocNos.release();
+    }
+    LocNoCount = Other.getLocNoCount();
+    WasIndirect = Other.getWasIndirect();
+    WasList = Other.getWasList();
+    Expression = Other.getExpression();
+    return *this;
+  }
 
   const DIExpression *getExpression() const { return Expression; }
-  unsigned getLocNo() const {
-    // Fix up the undef location number, which gets truncated.
-    return LocNo == INT_MAX ? UndefLocNo : LocNo;
+  uint8_t getLocNoCount() const { return LocNoCount; }
+  bool containsLocNo(unsigned LocNo) const {
+    return is_contained(loc_nos(), LocNo);
   }
   bool getWasIndirect() const { return WasIndirect; }
-  bool isUndef() const { return getLocNo() == UndefLocNo; }
+  bool getWasList() const { return WasList; }
+  bool isUndef() const { return LocNoCount == 0 || containsLocNo(UndefLocNo); }
 
-  DbgVariableValue changeLocNo(unsigned NewLocNo) const {
-    return DbgVariableValue(NewLocNo, WasIndirect, *Expression);
+  DbgVariableValue decrementLocNosAfterPivot(unsigned Pivot) const {
+    SmallVector<unsigned, 4> NewLocNos;
+    for (unsigned LocNo : loc_nos())
+      NewLocNos.push_back(LocNo != UndefLocNo && LocNo > Pivot ? LocNo - 1
+                                                               : LocNo);
+    return DbgVariableValue(NewLocNos, WasIndirect, WasList, *Expression);
+  }
+
+  DbgVariableValue remapLocNos(ArrayRef<unsigned> LocNoMap) const {
+    SmallVector<unsigned> NewLocNos;
+    for (unsigned LocNo : loc_nos())
+      // Undef values don't exist in locations (and thus not in LocNoMap
+      // either) so skip over them. See getLocationNo().
+      NewLocNos.push_back(LocNo == UndefLocNo ? UndefLocNo : LocNoMap[LocNo]);
+    return DbgVariableValue(NewLocNos, WasIndirect, WasList, *Expression);
+  }
+
+  DbgVariableValue changeLocNo(unsigned OldLocNo, unsigned NewLocNo) const {
+    SmallVector<unsigned> NewLocNos;
+    NewLocNos.assign(loc_nos_begin(), loc_nos_end());
+    auto OldLocIt = find(NewLocNos, OldLocNo);
+    assert(OldLocIt != NewLocNos.end() && "Old location must be present.");
+    *OldLocIt = NewLocNo;
+    return DbgVariableValue(NewLocNos, WasIndirect, WasList, *Expression);
+  }
+
+  bool hasLocNoGreaterThan(unsigned LocNo) const {
+    return any_of(loc_nos(),
+                  [LocNo](unsigned ThisLocNo) { return ThisLocNo > LocNo; });
+  }
+
+  void printLocNos(llvm::raw_ostream &OS) const {
+    for (const unsigned &Loc : loc_nos())
+      OS << (&Loc == loc_nos_begin() ? " " : ", ") << Loc;
   }
 
   friend inline bool operator==(const DbgVariableValue &LHS,
                                 const DbgVariableValue &RHS) {
-    return LHS.LocNo == RHS.LocNo && LHS.WasIndirect == RHS.WasIndirect &&
-           LHS.Expression == RHS.Expression;
+    if (std::tie(LHS.LocNoCount, LHS.WasIndirect, LHS.WasList,
+                 LHS.Expression) !=
+        std::tie(RHS.LocNoCount, RHS.WasIndirect, RHS.WasList, RHS.Expression))
+      return false;
+    return std::equal(LHS.loc_nos_begin(), LHS.loc_nos_end(),
+                      RHS.loc_nos_begin());
   }
 
   friend inline bool operator!=(const DbgVariableValue &LHS,
                                 const DbgVariableValue &RHS) {
     return !(LHS == RHS);
   }
 
+  unsigned *loc_nos_begin() { return LocNos.get(); }
+  const unsigned *loc_nos_begin() const { return LocNos.get(); }
+  unsigned *loc_nos_end() { return LocNos.get() + LocNoCount; }
+  const unsigned *loc_nos_end() const { return LocNos.get() + LocNoCount; }
+  ArrayRef<unsigned> loc_nos() const {
+    return ArrayRef<unsigned>(LocNos.get(), LocNoCount);
+  }
+
 private:
-  unsigned LocNo : 31;
-  unsigned WasIndirect : 1;
+  // IntervalMap requires the value object to be very small, to the extent
+  // that we do not have enough room for an std::vector. Using a C-style array
+  // (with a unique_ptr wrapper for convenience) allows us to optimize for this
+  // specific case by packing the array size into only 6 bits (it is highly
+  // unlikely that any debug value will need 64+ locations).
+  std::unique_ptr<unsigned[]> LocNos;
+  uint8_t LocNoCount : 6;
+  bool WasIndirect : 1;
+  bool WasList : 1;
   const DIExpression *Expression = nullptr;
 };
 } // namespace
 
 /// Map of where a user value is live to that value.
 using LocMap = IntervalMap<SlotIndex, DbgVariableValue, 4>;
 
 /// Map of stack slot offsets for spilled locations.
class UserValue {
 
   /// Set of interval start indexes that have been trimmed to the
   /// lexical scope.
   SmallSet<SlotIndex, 2> trimmedDefs;
 
   /// Insert a DBG_VALUE into MBB at Idx for DbgValue.
   void insertDebugValue(MachineBasicBlock *MBB, SlotIndex StartIdx,
                         SlotIndex StopIdx, DbgVariableValue DbgValue,
-                        bool Spilled, unsigned SpillOffset, LiveIntervals &LIS,
+                        ArrayRef<bool> LocSpills,
+                        ArrayRef<unsigned> SpillOffsets, LiveIntervals &LIS,
                         const TargetInstrInfo &TII,
                         const TargetRegisterInfo &TRI,
                         BlockSkipInstsMap &BBSkipInstsMap);
 
   /// Replace OldLocNo ranges with NewRegs ranges where NewRegs
   /// is live. Returns true if any changes were made.
   bool splitLocation(unsigned OldLocNo, ArrayRef<Register> NewRegs,
                      LiveIntervals &LIS);
unsigned getLocationNo(const MachineOperand &LocMO) {
     return locations.size() - 1;
   }
 
   /// Remove (recycle) a location number. If \p LocNo still is used by the
   /// locInts nothing is done.
   void removeLocationIfUnused(unsigned LocNo) {
     // Bail out if LocNo still is used.
     for (LocMap::const_iterator I = locInts.begin(); I.valid(); ++I) {
-      DbgVariableValue DbgValue = I.value();
-      if (DbgValue.getLocNo() == LocNo)
+      const DbgVariableValue &DbgValue = I.value();
+      if (DbgValue.containsLocNo(LocNo))
         return;
     }
     // Remove the entry in the locations vector, and adjust all references to
     // location numbers above the removed entry.
     locations.erase(locations.begin() + LocNo);
     for (LocMap::iterator I = locInts.begin(); I.valid(); ++I) {
-      DbgVariableValue DbgValue = I.value();
-      if (!DbgValue.isUndef() && DbgValue.getLocNo() > LocNo)
-        I.setValueUnchecked(DbgValue.changeLocNo(DbgValue.getLocNo() - 1));
+      const DbgVariableValue &DbgValue = I.value();
+      if (DbgValue.hasLocNoGreaterThan(LocNo))
+        I.setValueUnchecked(DbgValue.decrementLocNosAfterPivot(LocNo));
     }
   }
 
   /// Ensure that all virtual register locations are mapped.
   void mapVirtRegs(LDVImpl *LDV);
 
   /// Add a definition point to this user value.
-  void addDef(SlotIndex Idx, const MachineOperand &LocMO, bool IsIndirect,
-              const DIExpression &Expr) {
-    DbgVariableValue DbgValue(getLocationNo(LocMO), IsIndirect, Expr);
+  void addDef(SlotIndex Idx, ArrayRef<MachineOperand> LocMOs, bool IsIndirect,
+              bool IsList, const DIExpression &Expr) {
+    SmallVector<unsigned> Locs;
+    for (MachineOperand Op : LocMOs)
+      Locs.push_back(getLocationNo(Op));
+    DbgVariableValue DbgValue(Locs, IsIndirect, IsList, Expr);
     // Add a singular (Idx,Idx) -> value mapping.
     LocMap::iterator I = locInts.find(Idx);
     if (!I.valid() || I.start() != Idx)
-      I.insert(Idx, Idx.getNextSlot(), DbgValue);
+      I.insert(Idx, Idx.getNextSlot(), std::move(DbgValue));
     else
       // A later DBG_VALUE at the same SlotIndex overrides the old location.
-      I.setValue(DbgValue);
+      I.setValue(std::move(DbgValue));
   }
 
   /// Extend the current definition as far as possible down.
   ///
   /// Stop when meeting an existing def or when leaving the live
   /// range of VNI. End points where VNI is no longer live are added to Kills.
   ///
   /// We only propagate DBG_VALUES locally here. LiveDebugValues performs a
   /// data-flow analysis to propagate them beyond basic block boundaries.
   ///
   /// \param Idx Starting point for the definition.
   /// \param DbgValue value to propagate.
-  /// \param LR Restrict liveness to where LR has the value VNI. May be null.
-  /// \param VNI When LR is not null, this is the value to restrict to.
+  /// \param LiveIntervalInfo For each location number key in this map,
+  /// restricts liveness to where the LiveRange has the value equal to the\
+  /// VNInfo.
   /// \param [out] Kills Append end points of VNI's live range to Kills.
   /// \param LIS Live intervals analysis.
-  void extendDef(SlotIndex Idx, DbgVariableValue DbgValue, LiveRange *LR,
-                 const VNInfo *VNI, SmallVectorImpl<SlotIndex> *Kills,
+  void extendDef(SlotIndex Idx, DbgVariableValue DbgValue,
+                 SmallDenseMap<unsigned, std::pair<LiveRange *, const VNInfo *>>
+                     &LiveIntervalInfo,
+                 Optional<std::pair<SlotIndex, SmallVector<unsigned>>> &Kills,
                  LiveIntervals &LIS);
 
   /// The value in LI may be copies to other registers. Determine if
   /// any of the copies are available at the kill points, and add defs if
   /// possible.
   ///
-  /// \param LI Scan for copies of the value in LI->reg.
   /// \param DbgValue Location number of LI->reg, and DIExpression.
-  /// \param Kills Points where the range of DbgValue could be extended.
+  /// \param LocIntervals Scan for copies of the value for each location in the
+  /// corresponding LiveInterval->reg.
+  /// \param KilledAt The point where the range of DbgValue could be extended.
   /// \param [in,out] NewDefs Append (Idx, DbgValue) of inserted defs here.
   void addDefsFromCopies(
-      LiveInterval *LI, DbgVariableValue DbgValue,
-      const SmallVectorImpl<SlotIndex> &Kills,
+      DbgVariableValue DbgValue,
+      SmallVectorImpl<std::pair<unsigned, LiveInterval *>> &LocIntervals,
+      SlotIndex KilledAt,
       SmallVectorImpl<std::pair<SlotIndex, DbgVariableValue>> &NewDefs,
       MachineRegisterInfo &MRI, LiveIntervals &LIS);
 
   /// Compute the live intervals of all locations after collecting all their
   /// def points.
   void computeIntervals(MachineRegisterInfo &MRI, const TargetRegisterInfo &TRI,
                         LiveIntervals &LIS, LexicalScopes &LS);
 

 void UserValue::print(raw_ostream &OS, const TargetRegisterInfo *TRI) {
   OS << "!\"";
   printExtendedName(OS, Variable, dl);
 
   OS << "\"\t";
   for (LocMap::const_iterator I = locInts.begin(); I.valid(); ++I) {
     OS << " [" << I.start() << ';' << I.stop() << "):";
     if (I.value().isUndef())
-      OS << "undef";
+      OS << " undef";
     else {
-      OS << I.value().getLocNo();
+      I.value().printLocNos(OS);
       if (I.value().getWasIndirect())
         OS << " ind";
+      else if (I.value().getWasList())
+        OS << " list";
     }
   }
   for (unsigned i = 0, e = locations.size(); i != e; ++i) {
     OS << " Loc" << i << '=';
     locations[i].print(OS, TRI);
   }
   OS << '\n';
 }

 
 UserValue *LDVImpl::lookupVirtReg(Register VirtReg) {
   if (UserValue *UV = virtRegToEqClass.lookup(VirtReg))
     return UV->getLeader();
   return nullptr;
 }
 
 bool LDVImpl::handleDebugValue(MachineInstr &MI, SlotIndex Idx) {
-  // DBG_VALUE loc, offset, variable
-  if (MI.getNumOperands() != 4 ||
-      !(MI.getDebugOffset().isReg() || MI.getDebugOffset().isImm()) ||
-      !MI.getDebugVariableOp().isMetadata()) {
-    LLVM_DEBUG(dbgs() << "Can't handle " << MI);
+  // DBG_VALUE loc, offset, variable, expr
+  // DBG_VALUE_LIST variable, expr, locs...
+  if (!MI.isDebugValue()) {
+    LLVM_DEBUG(dbgs() << "Can't handle non-DBG_VALUE*: " << MI);
+    return false;
+  }
+  if (!MI.getDebugVariableOp().isMetadata()) {
+    LLVM_DEBUG(dbgs() << "Can't handle DBG_VALUE* with invalid variable: "
+                      << MI);
+    return false;
+  }
+  if (MI.isNonListDebugValue() &&
+      (MI.getNumOperands() != 4 ||
+       !(MI.getDebugOffset().isImm() || MI.getDebugOffset().isReg()))) {
+    LLVM_DEBUG(dbgs() << "Can't handle malformed DBG_VALUE: " << MI);
     return false;
   }
 
   // Detect invalid DBG_VALUE instructions, with a debug-use of a virtual
   // register that hasn't been defined yet. If we do not remove those here, then
   // the re-insertion of the DBG_VALUE instruction after register allocation
   // will be incorrect.
   // TODO: If earlier passes are corrected to generate sane debug information
   // (and if the machine verifier is improved to catch this), then these checks
   // could be removed or replaced by asserts.
   bool Discard = false;
-  if (MI.getDebugOperand(0).isReg() &&
-      Register::isVirtualRegister(MI.getDebugOperand(0).getReg())) {
-    const Register Reg = MI.getDebugOperand(0).getReg();
+  for (const MachineOperand &Op : MI.debug_operands()) {
+    if (Op.isReg() && Register::isVirtualRegister(Op.getReg())) {
+      const Register Reg = Op.getReg();
       if (!LIS->hasInterval(Reg)) {
         // The DBG_VALUE is described by a virtual register that does not have a
         // live interval. Discard the DBG_VALUE.
         Discard = true;
         LLVM_DEBUG(dbgs() << "Discarding debug info (no LIS interval): " << Idx
                           << " " << MI);
       } else {
-      // The DBG_VALUE is only valid if either Reg is live out from Idx, or Reg
-      // is defined dead at Idx (where Idx is the slot index for the instruction
-      // preceding the DBG_VALUE).
+        // The DBG_VALUE is only valid if either Reg is live out from Idx, or
+        // Reg is defined dead at Idx (where Idx is the slot index for the
+        // instruction preceding the DBG_VALUE).
         const LiveInterval &LI = LIS->getInterval(Reg);
         LiveQueryResult LRQ = LI.Query(Idx);
         if (!LRQ.valueOutOrDead()) {
           // We have found a DBG_VALUE with the value in a virtual register that
           // is not live. Discard the DBG_VALUE.
           Discard = true;
           LLVM_DEBUG(dbgs() << "Discarding debug info (reg not live): " << Idx
                             << " " << MI);
         }
       }
     }
+  }
 
   // Get or create the UserValue for (variable,offset) here.
   bool IsIndirect = MI.isDebugOffsetImm();
   if (IsIndirect)
     assert(MI.getDebugOffset().getImm() == 0 &&
            "DBG_VALUE with nonzero offset");
+  bool IsList = MI.isDebugValueList();
   const DILocalVariable *Var = MI.getDebugVariable();
   const DIExpression *Expr = MI.getDebugExpression();
   UserValue *UV = getUserValue(Var, Expr->getFragmentInfo(), MI.getDebugLoc());
   if (!Discard)
-    UV->addDef(Idx, MI.getDebugOperand(0), IsIndirect, *Expr);
+    UV->addDef(Idx,
+               ArrayRef<MachineOperand>(MI.debug_operands().begin(),
+                                        MI.debug_operands().end()),
+               IsIndirect, IsList, *Expr);
   else {
     MachineOperand MO = MachineOperand::CreateReg(0U, false);
     MO.setIsDebug();
-    UV->addDef(Idx, MO, false, *Expr);
+    // We should still pass a list the same size as MI.debug_operands() even if
+    // all MOs are undef, so that DbgVariableValue can correctly adjust the
+    // expression while removing the duplicated undefs.
+    SmallVector<MachineOperand, 4> UndefMOs(MI.getNumDebugOperands(), MO);
+    UV->addDef(Idx, UndefMOs, false, IsList, *Expr);
   }
   return true;
 }
 
 bool LDVImpl::handleDebugInstrRef(MachineInstr &MI, SlotIndex Idx) {
   assert(MI.isDebugRef());
   unsigned InstrNum = MI.getOperand(0).getImm();
   unsigned OperandNum = MI.getOperand(1).getImm();
for (MachineBasicBlock::iterator MBBI = MBB.begin(), MBBE = MBB.end();
         } else
           ++MBBI;
       } while (MBBI != MBBE && MBBI->isDebugInstr());
     }
   }
   return Changed;
 }
 
-void UserValue::extendDef(SlotIndex Idx, DbgVariableValue DbgValue, LiveRange *LR,
-                          const VNInfo *VNI, SmallVectorImpl<SlotIndex> *Kills,
-                          LiveIntervals &LIS) {
+void UserValue::extendDef(
+    SlotIndex Idx, DbgVariableValue DbgValue,
+    SmallDenseMap<unsigned, std::pair<LiveRange *, const VNInfo *>>
+        &LiveIntervalInfo,
+    Optional<std::pair<SlotIndex, SmallVector<unsigned>>> &Kills,
   SlotIndex Start = Idx;
   MachineBasicBlock *MBB = LIS.getMBBFromIndex(Start);
   SlotIndex Stop = LIS.getMBBEndIdx(MBB);
   LocMap::iterator I = locInts.find(Start);
 
-  // Limit to VNI's live range.
-  bool ToEnd = true;
-  if (LR && VNI) {
+  // Limit to the intersection of the VNIs' live ranges.
+  for (auto &LII : LiveIntervalInfo) {
+    LiveRange *LR = LII.second.first;
+    assert(LR && LII.second.second && "Missing range info for Idx.");
     LiveInterval::Segment *Segment = LR->getSegmentContaining(Start);
-    if (!Segment || Segment->valno != VNI) {
-      if (Kills)
-        Kills->push_back(Start);
-      return;
-    }
+    assert(Segment && Segment->valno == LII.second.second &&
+           "Invalid VNInfo for Idx given?");
     if (Segment->end < Stop) {
       Stop = Segment->end;
-      ToEnd = false;
+      Kills = {Stop, {LII.first}};
+    } else if (Segment->end == Stop && Kills.hasValue()) {
+      // If multiple locations end at the same place, track all of them in
+      // Kills.
+      Kills->second.push_back(LII.first);
     }
   }
 
   // There could already be a short def at Start.
   if (I.valid() && I.start() <= Start) {
     // Stop when meeting a different location or an already extended interval.
     Start = Start.getNextSlot();
-    if (I.value() != DbgValue || I.stop() != Start)
+    if (I.value() != DbgValue || I.stop() != Start) {
+      // Clear `Kills`, as we have a new def available.
+      Kills = None;
       return;
+    }
     // This is a one-slot placeholder. Just skip it.
     ++I;
   }
 
   // Limited by the next def.
-  if (I.valid() && I.start() < Stop)
+  if (I.valid() && I.start() < Stop) {
     Stop = I.start();
-  // Limited by VNI's live range.
-  else if (!ToEnd && Kills)
-    Kills->push_back(Stop);
+    // Clear `Kills`, as we have a new def available.
+    Kills = None;
+  }
 
-  if (Start < Stop)
-    I.insert(Start, Stop, DbgValue);
+  if (Start < Stop) {
+    DbgVariableValue ExtDbgValue(DbgValue);
+    I.insert(Start, Stop, std::move(ExtDbgValue));
+  }
 }
 
 void UserValue::addDefsFromCopies(
-    LiveInterval *LI, DbgVariableValue DbgValue,
-    const SmallVectorImpl<SlotIndex> &Kills,
+    DbgVariableValue DbgValue,
+    SmallVectorImpl<std::pair<unsigned, LiveInterval *>> &LocIntervals,
+    SlotIndex KilledAt,
     SmallVectorImpl<std::pair<SlotIndex, DbgVariableValue>> &NewDefs,
     MachineRegisterInfo &MRI, LiveIntervals &LIS) {
-  if (Kills.empty())
-    return;
   // Don't track copies from physregs, there are too many uses.
-  if (!Register::isVirtualRegister(LI->reg()))
+  if (any_of(LocIntervals, [](auto LocI) {
+        return !Register::isVirtualRegister(LocI.second->reg());
+      }))
     return;
 
   // Collect all the (vreg, valno) pairs that are copies of LI.
-  SmallVector<std::pair<LiveInterval*, const VNInfo*>, 8> CopyValues;
+  SmallDenseMap<unsigned,
+                SmallVector<std::pair<LiveInterval *, const VNInfo *>, 4>>
+      CopyValues;
+  for (auto &LocInterval : LocIntervals) {
+    unsigned LocNo = LocInterval.first;
+    LiveInterval *LI = LocInterval.second;
     for (MachineOperand &MO : MRI.use_nodbg_operands(LI->reg())) {
       MachineInstr *MI = MO.getParent();
       // Copies of the full value.
       if (MO.getSubReg() || !MI->isCopy())
         continue;
       Register DstReg = MI->getOperand(0).getReg();
 
       // Don't follow copies to physregs. These are usually setting up call
       // arguments, and the argument registers are always call clobbered. We are
-    // better off in the source register which could be a callee-saved register,
-    // or it could be spilled.
+      // better off in the source register which could be a callee-saved
+      // register, or it could be spilled.
       if (!Register::isVirtualRegister(DstReg))
         continue;
 
       // Is the value extended to reach this copy? If not, another def may be
       // blocking it, or we are looking at a wrong value of LI.
       SlotIndex Idx = LIS.getInstructionIndex(*MI);
       LocMap::iterator I = locInts.find(Idx.getRegSlot(true));
       if (!I.valid() || I.value() != DbgValue)
         continue;
 
       if (!LIS.hasInterval(DstReg))
         continue;
       LiveInterval *DstLI = &LIS.getInterval(DstReg);
       const VNInfo *DstVNI = DstLI->getVNInfoAt(Idx.getRegSlot());
       assert(DstVNI && DstVNI->def == Idx.getRegSlot() && "Bad copy value");
-    CopyValues.push_back(std::make_pair(DstLI, DstVNI));
+      CopyValues[LocNo].push_back(std::make_pair(DstLI, DstVNI));
+    }
   }
 
   if (CopyValues.empty())
     return;
 
-  LLVM_DEBUG(dbgs() << "Got " << CopyValues.size() << " copies of " << *LI
-                    << '\n');
+#if !defined(NDEBUG)
+  for (auto &LocInterval : LocIntervals)
+    LLVM_DEBUG(dbgs() << "Got " << CopyValues[LocInterval.first].size()
+                      << " copies of " << *LocInterval.second << '\n');
+#endif
 
-  // Try to add defs of the copied values for each kill point.
-  for (unsigned i = 0, e = Kills.size(); i != e; ++i) {
-    SlotIndex Idx = Kills[i];
-    for (unsigned j = 0, e = CopyValues.size(); j != e; ++j) {
-      LiveInterval *DstLI = CopyValues[j].first;
-      const VNInfo *DstVNI = CopyValues[j].second;
-      if (DstLI->getVNInfoAt(Idx) != DstVNI)
-        continue;
-      // Check that there isn't already a def at Idx
-      LocMap::iterator I = locInts.find(Idx);
-      if (I.valid() && I.start() <= Idx)
+  // Try to add defs of the copied values for the kill point. Check that there
+  // isn't already a def at Idx.
+  LocMap::iterator I = locInts.find(KilledAt);
+  if (I.valid() && I.start() <= KilledAt)
+    return;
+  DbgVariableValue NewValue(DbgValue);
+  for (auto &LocInterval : LocIntervals) {
+    unsigned LocNo = LocInterval.first;
+    bool FoundCopy = false;
+    for (auto &LIAndVNI : CopyValues[LocNo]) {
+      LiveInterval *DstLI = LIAndVNI.first;
+      const VNInfo *DstVNI = LIAndVNI.second;
+      if (DstLI->getVNInfoAt(KilledAt) != DstVNI)
         continue;
-      LLVM_DEBUG(dbgs() << "Kill at " << Idx << " covered by valno #"
+      LLVM_DEBUG(dbgs() << "Kill at " << KilledAt << " covered by valno #"
                         << DstVNI->id << " in " << *DstLI << '\n');
       MachineInstr *CopyMI = LIS.getInstructionFromIndex(DstVNI->def);
       assert(CopyMI && CopyMI->isCopy() && "Bad copy value");
-      unsigned LocNo = getLocationNo(CopyMI->getOperand(0));
-      DbgVariableValue NewValue = DbgValue.changeLocNo(LocNo);
-      I.insert(Idx, Idx.getNextSlot(), NewValue);
-      NewDefs.push_back(std::make_pair(Idx, NewValue));
+      unsigned NewLocNo = getLocationNo(CopyMI->getOperand(0));
+      NewValue = NewValue.changeLocNo(LocNo, NewLocNo);
+      FoundCopy = true;
       break;
     }
+    // If there are any killed locations we can't find a copy for, we can't
+    // extend the variable value.
+    if (!FoundCopy)
+      return;
   }
+  I.insert(KilledAt, KilledAt.getNextSlot(), NewValue);
+  NewDefs.push_back(std::make_pair(KilledAt, NewValue));
 }
 
 void UserValue::computeIntervals(MachineRegisterInfo &MRI,
                                  const TargetRegisterInfo &TRI,
                                  LiveIntervals &LIS, LexicalScopes &LS) {
   SmallVector<std::pair<SlotIndex, DbgVariableValue>, 16> Defs;
 
   // Collect all defs to be extended (Skipping undefs).
   for (LocMap::const_iterator I = locInts.begin(); I.valid(); ++I)
     if (!I.value().isUndef())
       Defs.push_back(std::make_pair(I.start(), I.value()));
 
   // Extend all defs, and possibly add new ones along the way.
   for (unsigned i = 0; i != Defs.size(); ++i) {
     SlotIndex Idx = Defs[i].first;
     DbgVariableValue DbgValue = Defs[i].second;
-    const MachineOperand &LocMO = locations[DbgValue.getLocNo()];
-
-    if (!LocMO.isReg()) {
-      extendDef(Idx, DbgValue, nullptr, nullptr, nullptr, LIS);
+    SmallDenseMap<unsigned, std::pair<LiveRange *, const VNInfo *>> LIs;
+    SmallVector<const VNInfo *, 4> VNIs;
+    bool ShouldExtendDef = false;
+    for (unsigned LocNo : DbgValue.loc_nos()) {
+      const MachineOperand &LocMO = locations[LocNo];
+      if (!LocMO.isReg() || !Register::isVirtualRegister(LocMO.getReg())) {
+        ShouldExtendDef |= !LocMO.isReg();
         continue;
       }
-
-    // Register locations are constrained to where the register value is live.
-    if (Register::isVirtualRegister(LocMO.getReg())) {
+      ShouldExtendDef = true;
       LiveInterval *LI = nullptr;
       const VNInfo *VNI = nullptr;
       if (LIS.hasInterval(LocMO.getReg())) {
         LI = &LIS.getInterval(LocMO.getReg());
         VNI = LI->getVNInfoAt(Idx);
       }
-      SmallVector<SlotIndex, 16> Kills;
-      extendDef(Idx, DbgValue, LI, VNI, &Kills, LIS);
+      if (LI && VNI)
+        LIs[LocNo] = {LI, VNI};
+    }
+    if (ShouldExtendDef) {
+      Optional<std::pair<SlotIndex, SmallVector<unsigned>>> Kills;
+      extendDef(Idx, DbgValue, LIs, Kills, LIS);
+
+      if (Kills) {
+        SmallVector<std::pair<unsigned, LiveInterval *>, 2> KilledLocIntervals;
+        bool AnySubreg = false;
+        for (unsigned LocNo : Kills->second) {
+          const MachineOperand &LocMO = this->locations[LocNo];
+          if (LocMO.getSubReg()) {
+            AnySubreg = true;
+            break;
+          }
+          LiveInterval *LI = &LIS.getInterval(LocMO.getReg());
+          KilledLocIntervals.push_back({LocNo, LI});
+        }
+
         // FIXME: Handle sub-registers in addDefsFromCopies. The problem is that
         // if the original location for example is %vreg0:sub_hi, and we find a
-      // full register copy in addDefsFromCopies (at the moment it only handles
-      // full register copies), then we must add the sub1 sub-register index to
-      // the new location. However, that is only possible if the new virtual
-      // register is of the same regclass (or if there is an equivalent
-      // sub-register in that regclass). For now, simply skip handling copies if
-      // a sub-register is involved.
-      if (LI && !LocMO.getSubReg())
-        addDefsFromCopies(LI, DbgValue, Kills, Defs, MRI, LIS);
-      continue;
+        // full register copy in addDefsFromCopies (at the moment it only
+        // handles full register copies), then we must add the sub1 sub-register
+        // index to the new location. However, that is only possible if the new
+        // virtual register is of the same regclass (or if there is an
+        // equivalent sub-register in that regclass). For now, simply skip
+        // handling copies if a sub-register is involved.
+        if (!AnySubreg)
+          addDefsFromCopies(DbgValue, KilledLocIntervals, Kills->first, Defs,
+                            MRI, LIS);
+      }
     }
 
     // For physregs, we only mark the start slot idx. DwarfDebug will see it
     // as if the DBG_VALUE is valid up until the end of the basic block, or
     // the next def of the physical register. So we do not need to extend the
     // range. It might actually happen that the DBG_VALUE is the last use of
     // the physical register (e.g. if this is an unused input argument to a
     // function).
for (unsigned i = 0; i != NewRegs.size(); ++i) {
     LiveInterval::iterator LIE = LI->end();
     while (LocMapI.valid() && LII != LIE) {
       // At this point, we know that LocMapI.stop() > LII->start.
       LII = LI->advanceTo(LII, LocMapI.start());
       if (LII == LIE)
         break;
 
       // Now LII->end > LocMapI.start(). Do we have an overlap?
-      if (LocMapI.value().getLocNo() == OldLocNo &&
+      if (LocMapI.value().containsLocNo(OldLocNo) &&
           LII->start < LocMapI.stop()) {
         // Overlapping correct location. Allocate NewLocNo now.
         if (NewLocNo == UndefLocNo) {
           MachineOperand MO = MachineOperand::CreateReg(LI->reg(), false);
           MO.setSubReg(locations[OldLocNo].getSubReg());
           NewLocNo = getLocationNo(MO);
           DidChange = true;
         }
 
         SlotIndex LStart = LocMapI.start();
         SlotIndex LStop = LocMapI.stop();
         DbgVariableValue OldDbgValue = LocMapI.value();
 
         // Trim LocMapI down to the LII overlap.
         if (LStart < LII->start)
           LocMapI.setStartUnchecked(LII->start);
         if (LStop > LII->end)
           LocMapI.setStopUnchecked(LII->end);
 
         // Change the value in the overlap. This may trigger coalescing.
-        LocMapI.setValue(OldDbgValue.changeLocNo(NewLocNo));
+        LocMapI.setValue(OldDbgValue.changeLocNo(OldLocNo, NewLocNo));
 
         // Re-insert any removed OldDbgValue ranges.
         if (LStart < LocMapI.start()) {
           LocMapI.insert(LStart, LocMapI.start(), OldDbgValue);
           ++LocMapI;
           assert(LocMapI.valid() && "Unexpected coalescing");
         }
         if (LStop > LocMapI.stop()) {
for (auto &Pair : NewLocations) {
     }
   }
 
   // Update the interval map, but only coalesce left, since intervals to the
   // right use the old location numbers. This should merge two contiguous
   // DBG_VALUE intervals with different vregs that were allocated to the same
   // physical register.
   for (LocMap::iterator I = locInts.begin(); I.valid(); ++I) {
-    DbgVariableValue DbgValue = I.value();
-    // Undef values don't exist in locations (and thus not in LocNoMap either)
-    // so skip over them. See getLocationNo().
-    if (DbgValue.isUndef())
-      continue;
-    unsigned NewLocNo = LocNoMap[DbgValue.getLocNo()];
-    I.setValueUnchecked(DbgValue.changeLocNo(NewLocNo));
+    I.setValueUnchecked(I.value().remapLocNos(LocNoMap));
     I.setStart(I.start());
   }
 }
 
 /// Find an iterator for inserting a DBG_VALUE instruction.
 static MachineBasicBlock::iterator
 findInsertLocation(MachineBasicBlock *MBB, SlotIndex Idx, LiveIntervals &LIS,
                    BlockSkipInstsMap &BBSkipInstsMap) {
findInsertLocation(MachineBasicBlock *MBB, SlotIndex Idx, LiveIntervals &LIS,
   // Don't insert anything after the first terminator, though.
   return MI->isTerminator() ? MBB->getFirstTerminator() :
                               std::next(MachineBasicBlock::iterator(MI));
 }
 
 /// Find an iterator for inserting the next DBG_VALUE instruction
 /// (or end if no more insert locations found).
 static MachineBasicBlock::iterator
-findNextInsertLocation(MachineBasicBlock *MBB,
-                       MachineBasicBlock::iterator I,
-                       SlotIndex StopIdx, MachineOperand &LocMO,
-                       LiveIntervals &LIS,
-                       const TargetRegisterInfo &TRI) {
-  if (!LocMO.isReg())
+findNextInsertLocation(MachineBasicBlock *MBB, MachineBasicBlock::iterator I,
+                       SlotIndex StopIdx, ArrayRef<MachineOperand> LocMOs,
+                       LiveIntervals &LIS, const TargetRegisterInfo &TRI) {
+  SmallVector<Register, 4> Regs;
+  for (const MachineOperand &LocMO : LocMOs)
+    if (LocMO.isReg())
+      Regs.push_back(LocMO.getReg());
+  if (Regs.empty())
     return MBB->instr_end();
-  Register Reg = LocMO.getReg();
 
   // Find the next instruction in the MBB that define the register Reg.
   while (I != MBB->end() && !I->isTerminator()) {
     if (!LIS.isNotInMIMap(*I) &&
         SlotIndex::isEarlierEqualInstr(StopIdx, LIS.getInstructionIndex(*I)))
       break;
-    if (I->definesRegister(Reg, &TRI))
+    if (any_of(Regs, [&I, &TRI](Register &Reg) {
+          return I->definesRegister(Reg, &TRI);
+        }))
       // The insert location is directly after the instruction/bundle.
       return std::next(I);
     ++I;
   }
   return MBB->end();
 }
 
 void UserValue::insertDebugValue(MachineBasicBlock *MBB, SlotIndex StartIdx,
                                  SlotIndex StopIdx, DbgVariableValue DbgValue,
-                                 bool Spilled, unsigned SpillOffset,
+                                 ArrayRef<bool> LocSpills,
+                                 ArrayRef<unsigned> SpillOffsets,
                                  LiveIntervals &LIS, const TargetInstrInfo &TII,
                                  const TargetRegisterInfo &TRI,
                                  BlockSkipInstsMap &BBSkipInstsMap) {
   SlotIndex MBBEndIdx = LIS.getMBBEndIdx(&*MBB);
   // Only search within the current MBB.
   StopIdx = (MBBEndIdx < StopIdx) ? MBBEndIdx : StopIdx;
   MachineBasicBlock::iterator I =
       findInsertLocation(MBB, StartIdx, LIS, BBSkipInstsMap);
   // Undef values don't exist in locations so create new "noreg" register MOs
   // for them. See getLocationNo().
-  MachineOperand MO =
-      !DbgValue.isUndef()
-          ? locations[DbgValue.getLocNo()]
-          : MachineOperand::CreateReg(
+  SmallVector<MachineOperand, 8> MOs;
+  if (DbgValue.isUndef()) {
+    MOs.assign(DbgValue.loc_nos().size(),
+               MachineOperand::CreateReg(
                    /* Reg */ 0, /* isDef */ false, /* isImp */ false,
                    /* isKill */ false, /* isDead */ false,
                    /* isUndef */ false, /* isEarlyClobber */ false,
-                /* SubReg */ 0, /* isDebug */ true);
+                   /* SubReg */ 0, /* isDebug */ true));
+  } else {
+    for (unsigned LocNo : DbgValue.loc_nos())
+      MOs.push_back(locations[LocNo]);
+  }
 
   ++NumInsertedDebugValues;
 
   assert(cast<DILocalVariable>(Variable)
              ->isValidLocationForIntrinsic(getDebugLoc()) &&
          "Expected inlined-at fields to agree");
 
   // If the location was spilled, the new DBG_VALUE will be indirect. If the
   // original DBG_VALUE was indirect, we need to add DW_OP_deref to indicate
   // that the original virtual register was a pointer. Also, add the stack slot
   // offset for the spilled register to the expression.
   const DIExpression *Expr = DbgValue.getExpression();
-  uint8_t DIExprFlags = DIExpression::ApplyOffset;
   bool IsIndirect = DbgValue.getWasIndirect();
-  if (Spilled) {
+  bool IsList = DbgValue.getWasList();
+  for (unsigned I = 0, E = LocSpills.size(); I != E; ++I) {
+    if (LocSpills[I]) {
+      if (!IsList) {
+        uint8_t DIExprFlags = DIExpression::ApplyOffset;
         if (IsIndirect)
           DIExprFlags |= DIExpression::DerefAfter;
-    Expr =
-        DIExpression::prepend(Expr, DIExprFlags, SpillOffset);
+        Expr = DIExpression::prepend(Expr, DIExprFlags, SpillOffsets[I]);
         IsIndirect = true;
+      } else {
+        SmallVector<uint64_t, 4> Ops;
+        DIExpression::appendOffset(Ops, SpillOffsets[I]);
+        Ops.push_back(dwarf::DW_OP_deref);
+        Expr = DIExpression::appendOpsToArg(Expr, Ops, I);
+      }
     }
 
-  assert((!Spilled || MO.isFI()) && "a spilled location must be a frame index");
+    assert((!LocSpills[I] || MOs[I].isFI()) &&
+           "a spilled location must be a frame index");
+  }
 
+  unsigned DbgValueOpcode =
+      IsList ? TargetOpcode::DBG_VALUE_LIST : TargetOpcode::DBG_VALUE;
   do {
-    BuildMI(*MBB, I, getDebugLoc(), TII.get(TargetOpcode::DBG_VALUE),
-            IsIndirect, MO, Variable, Expr);
+    BuildMI(*MBB, I, getDebugLoc(), TII.get(DbgValueOpcode), IsIndirect, MOs,
+            Variable, Expr);
 
-    // Continue and insert DBG_VALUES after every redefinition of register
+    // Continue and insert DBG_VALUES after every redefinition of a register
     // associated with the debug value within the range
-    I = findNextInsertLocation(MBB, I, StopIdx, MO, LIS, TRI);
+    I = findNextInsertLocation(MBB, I, StopIdx, MOs, LIS, TRI);
   } while (I != MBB->end());
 }
 
 void UserLabel::insertDebugLabel(MachineBasicBlock *MBB, SlotIndex Idx,
                                  LiveIntervals &LIS, const TargetInstrInfo &TII,
                                  BlockSkipInstsMap &BBSkipInstsMap) {
   MachineBasicBlock::iterator I =
       findInsertLocation(MBB, Idx, LIS, BBSkipInstsMap);
   ++NumInsertedDebugLabels;
   BuildMI(*MBB, I, getDebugLoc(), TII.get(TargetOpcode::DBG_LABEL))
       .addMetadata(Label);
 }
 
 void UserValue::emitDebugValues(VirtRegMap *VRM, LiveIntervals &LIS,
                                 const TargetInstrInfo &TII,
                                 const TargetRegisterInfo &TRI,
                                 const SpillOffsetMap &SpillOffsets,
                                 BlockSkipInstsMap &BBSkipInstsMap) {
   MachineFunction::iterator MFEnd = VRM->getMachineFunction().end();
 
   for (LocMap::const_iterator I = locInts.begin(); I.valid();) {
     SlotIndex Start = I.start();
     SlotIndex Stop = I.stop();
     DbgVariableValue DbgValue = I.value();
-    auto SpillIt = !DbgValue.isUndef() ? SpillOffsets.find(DbgValue.getLocNo())
-                                       : SpillOffsets.end();
+
+    SmallVector<bool> SpilledLocs;
+    SmallVector<unsigned> LocSpillOffsets;
+    for (unsigned LocNo : DbgValue.loc_nos()) {
+      auto SpillIt =
+          !DbgValue.isUndef() ? SpillOffsets.find(LocNo) : SpillOffsets.end();
       bool Spilled = SpillIt != SpillOffsets.end();
-    unsigned SpillOffset = Spilled ? SpillIt->second : 0;
+      SpilledLocs.push_back(Spilled);
+      LocSpillOffsets.push_back(Spilled ? SpillIt->second : 0);
+    }
 
     // If the interval start was trimmed to the lexical scope insert the
     // DBG_VALUE at the previous index (otherwise it appears after the
     // first instruction in the range).
     if (trimmedDefs.count(Start))
       Start = Start.getPrevIndex();
 
-    LLVM_DEBUG(dbgs() << "\t[" << Start << ';' << Stop
-                      << "):" << DbgValue.getLocNo());
+    LLVM_DEBUG(auto &dbg = dbgs(); dbg << "\t[" << Start << ';' << Stop << "):";
+               DbgValue.printLocNos(dbg));
     MachineFunction::iterator MBB = LIS.getMBBFromIndex(Start)->getIterator();
     SlotIndex MBBEnd = LIS.getMBBEndIdx(&*MBB);
 
     LLVM_DEBUG(dbgs() << ' ' << printMBBReference(*MBB) << '-' << MBBEnd);
-    insertDebugValue(&*MBB, Start, Stop, DbgValue, Spilled, SpillOffset, LIS,
-                     TII, TRI, BBSkipInstsMap);
+    insertDebugValue(&*MBB, Start, Stop, DbgValue, SpilledLocs, LocSpillOffsets,
+                     LIS, TII, TRI, BBSkipInstsMap);
     // This interval may span multiple basic blocks.
     // Insert a DBG_VALUE into each one.
     while (Stop > MBBEnd) {
       // Move to the next block.
       Start = MBBEnd;
       if (++MBB == MFEnd)
         break;
       MBBEnd = LIS.getMBBEndIdx(&*MBB);
       LLVM_DEBUG(dbgs() << ' ' << printMBBReference(*MBB) << '-' << MBBEnd);
-      insertDebugValue(&*MBB, Start, Stop, DbgValue, Spilled, SpillOffset, LIS,
-                       TII, TRI, BBSkipInstsMap);
+      insertDebugValue(&*MBB, Start, Stop, DbgValue, SpilledLocs,
+                       LocSpillOffsets, LIS, TII, TRI, BBSkipInstsMap);
     }
     LLVM_DEBUG(dbgs() << '\n');
     if (MBB == MFEnd)
       break;
 
     ++I;
   }
 }

llvm/test/DebugInfo/MIR/X86/dvl-livedebugvars-movements.mir

+# RUN: llc -start-after=phi-node-elimination -stop-after=virtregrewriter %s -mtriple=x86_64-unknown-unknown -o - | FileCheck %s
+#
+# Test that when a livedebugvars interval is split, DBG_VALUE_LISTs are created
+# with the correct operands and exprs. Force values to be moved around between
+# registers and stack through inline asm blocks that clobber things.
+#
+# CHECK-LABEL: bb.0.entry:
+# CHECK:       $rbx = COPY $rsi
+# CHECK-NEXT:  MOV64mr %stack.0, 1, $noreg, 0, $noreg, $rdi
+# CHECK-NEXT:  DBG_VALUE_LIST !14, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_deref, DW_OP_LLVM_arg, 1, DW_OP_plus), %stack.0, $rbx,
+# CHECK-NEXT:  INLINEASM
+# CHECK-NEXT:  $rax = MOV64rm %stack.0,
+# CHECK-NEXT:  DBG_VALUE_LIST !14, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus), $rax, $rbx,
+# CHECK-NEXT:  CALL64pcrel32 @foo
+# CHECK-NEXT:  DBG_VALUE_LIST !14, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_deref, DW_OP_LLVM_arg, 1, DW_OP_plus), %stack.0, $rbx,
+# CHECK-NEXT:  $rcx = COPY killed renamable $rbx
+# CHECK-NEXT:  DBG_VALUE_LIST !14, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_deref, DW_OP_LLVM_arg, 1, DW_OP_plus), %stack.0, $rcx,
+# CHECK-NEXT:  INLINEASM
+# CHECK-NEXT:  $rax = MOV64rm %stack.0
+# CHECK-NEXT:  DBG_VALUE_LIST !14, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus), $rax, $rcx,
+
+--- |
+  ; ModuleID = 'tmp.ll'
+  source_filename = "tmp.ll"
+  target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+  
+  %struct.a = type { i32 }
+  
+  ; Function Attrs: nounwind ssp
+  define i32 @bar(%struct.a* nocapture %b, i32 %shoes) !dbg !4 {
+  entry:
+    tail call void @llvm.dbg.value(metadata i32 %shoes, metadata !9, metadata !DIExpression()), !dbg !16
+    %tmp1 = getelementptr inbounds %struct.a, %struct.a* %b, i64 0, i32 0, !dbg !17
+    br label %bb3
+  
+  bb1:                                              ; preds = %bb2
+    tail call void @llvm.dbg.value(metadata i32 %shoes, metadata !9, metadata !DIExpression()), !dbg !16
+    %add = add nsw i32 %tmp2, 1, !dbg !18
+    br label %exit
+  
+  bb2:                                              ; preds = %bb3
+    tail call void @llvm.dbg.value(metadata i32 %tmp2, metadata !14, metadata !DIExpression()), !dbg !17
+    %call = tail call i32 (...) @foo(i32 %tmp2), !dbg !19
+    br label %bb1
+  
+  bb3:                                              ; preds = %entry
+    %tmp2 = load i32, i32* %tmp1, align 4, !dbg !17
+    br label %bb2
+  
+  exit:                                             ; preds = %bb1
+    ret i32 %shoes, !dbg !18
+  }
+  
+  declare i32 @foo(...)
+  
+  ; Function Attrs: nounwind readnone speculatable willreturn
+  declare void @llvm.dbg.value(metadata, metadata, metadata)
+  
+  !llvm.dbg.cu = !{!0}
+  !llvm.module.flags = !{!3}
+  
+  !0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 2.9 (trunk 122997)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !2)
+  !1 = !DIFile(filename: "bar.c", directory: "/private/tmp")
+  !2 = !{}
+  !3 = !{i32 1, !"Debug Info Version", i32 3}
+  !4 = distinct !DISubprogram(name: "bar", scope: !1, file: !1, line: 5, type: !5, virtualIndex: 6, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !8)
+  !5 = !DISubroutineType(types: !6)
+  !6 = !{!7}
+  !7 = !DIBasicType(name: "int", size: 32, align: 32, encoding: DW_ATE_signed)
+  !8 = !{!9, !14}
+  !9 = !DILocalVariable(name: "b", arg: 1, scope: !4, file: !1, line: 5, type: !10)
+  !10 = !DIDerivedType(tag: DW_TAG_pointer_type, scope: !0, baseType: !11, size: 64, align: 64)
+  !11 = !DICompositeType(tag: DW_TAG_structure_type, name: "a", scope: !0, file: !1, line: 1, size: 32, align: 32, elements: !12)
+  !12 = !{!13}
+  !13 = !DIDerivedType(tag: DW_TAG_member, name: "c", scope: !1, file: !1, line: 2, baseType: !7, size: 32, align: 32)
+  !14 = !DILocalVariable(name: "x", scope: !15, file: !1, line: 6, type: !7)
+  !15 = distinct !DILexicalBlock(scope: !4, file: !1, line: 5, column: 22)
+  !16 = !DILocation(line: 5, column: 19, scope: !4)
+  !17 = !DILocation(line: 6, column: 14, scope: !15)
+  !18 = !DILocation(line: 8, column: 2, scope: !15)
+  !19 = !DILocation(line: 7, column: 2, scope: !15)
+
+...
+---
+name:            bar
+tracksRegLiveness: true
+body:             |
+  bb.0.entry:
+    liveins: $rdi, $rsi
+  
+    %4:gr64= COPY $rsi
+    %2:gr64 = COPY $rdi
+    DBG_VALUE_LIST !14, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus), %2, %4, debug-location !17
+    %3:gr64 = COPY killed %2
+    %5:gr64 = COPY killed %4
+
+    ; Force allocation into $rax and $rbx
+    INLINEASM &"", 1, 12, implicit-def dead $rcx, 12, implicit-def dead $rdx, 12, implicit-def dead $rsi, 12, implicit-def dead $rdi, 12, implicit-def $rbp, 12, implicit-def dead $r8, 12, implicit-def dead $r9, 12, implicit-def dead $r10, 12, implicit-def dead $r11, 12, implicit-def dead $r12, 12, implicit-def dead $r13, 12, implicit-def dead $r14, 12, implicit-def dead $r15, 12, !18, debug-location !17
+
+    ; Force a use of these two registers.
+    CALL64pcrel32 @foo, csr_64, implicit $rsp, implicit $ssp, implicit %3, implicit %5
+
+    ; Now make the register allocator move them to rcx and rdx!
+    INLINEASM &"", 1, 12, implicit-def dead $rax, 12, implicit-def dead $rbx, 12, implicit-def dead $rsi, 12, implicit-def dead $rdi, 12, implicit-def $rbp, 12, implicit-def dead $r8, 12, implicit-def dead $r9, 12, implicit-def dead $r10, 12, implicit-def dead $r11, 12, implicit-def dead $r12, 12, implicit-def dead $r13, 12, implicit-def dead $r14, 12, implicit-def dead $r15, 12, !18, debug-location !17
+
+    CALL64pcrel32 @foo, csr_64, implicit $rsp, implicit $ssp, implicit %3, implicit %5
+
+    RETQ
+...

llvm/test/DebugInfo/MIR/X86/dvl-livedebugvars-stackptr.mir

+# RUN: llc -start-after=phi-node-elimination -stop-after=virtregrewriter %s -mtriple=x86_64-unknown-unknown -o - | FileCheck %s
+#
+# This is a copy of the adjacent "-movements.mir" file, but where one of the
+# operands to DBG_VALUE_LIST is a stack _pointer_ rather than a spilt value.
+# The expression should grow no additional derefs for the stack pointer.
+#
+# CHECK-LABEL: bb.0.entry:
+# CHECK:       $rbx = COPY $rsi
+# CHECK-NEXT:  MOV64mr %stack.1, 1, $noreg, 0, $noreg, $rdi
+# CHECK-NEXT:  DBG_VALUE_LIST !14, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_deref, DW_OP_LLVM_arg, 1, DW_OP_plus), %stack.1, %stack.0.local1
+# CHECK-NEXT:  INLINEASM
+# CHECK-NEXT:  $rax = MOV64rm %stack.1,
+# CHECK-NEXT:  DBG_VALUE_LIST !14, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus), $rax, %stack.0.local1,
+# CHECK-NEXT:  CALL64pcrel32 @foo
+# CHECK-NEXT:  DBG_VALUE_LIST !14, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_deref, DW_OP_LLVM_arg, 1, DW_OP_plus), %stack.1, %stack.0.local1,
+# CHECK-NEXT:  $rcx = COPY killed renamable $rbx
+# CHECK-NEXT:  INLINEASM
+# CHECK-NEXT:  $rax = MOV64rm %stack.1
+# CHECK-NEXT:  DBG_VALUE_LIST !14, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus), $rax, %stack.0.local1,
+
+--- |
+  ; ModuleID = 'tmp.ll'
+  source_filename = "tmp.ll"
+  target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+  
+  %struct.a = type { i32 }
+  
+  ; Function Attrs: nounwind ssp
+  define i32 @bar(%struct.a* nocapture %b, i32 %shoes) !dbg !4 {
+  entry:
+    %local1 = alloca i64
+    tail call void @llvm.dbg.value(metadata i32 %shoes, metadata !9, metadata !DIExpression()), !dbg !16
+    %tmp1 = getelementptr inbounds %struct.a, %struct.a* %b, i64 0, i32 0, !dbg !17
+    br label %bb3
+  
+  bb1:                                              ; preds = %bb2
+    tail call void @llvm.dbg.value(metadata i32 %shoes, metadata !9, metadata !DIExpression()), !dbg !16
+    %add = add nsw i32 %tmp2, 1, !dbg !18
+    br label %exit
+  
+  bb2:                                              ; preds = %bb3
+    tail call void @llvm.dbg.value(metadata i32 %tmp2, metadata !14, metadata !DIExpression()), !dbg !17
+    %call = tail call i32 (...) @foo(i32 %tmp2), !dbg !19
+    br label %bb1
+  
+  bb3:                                              ; preds = %entry
+    %tmp2 = load i32, i32* %tmp1, align 4, !dbg !17
+    br label %bb2
+  
+  exit:                                             ; preds = %bb1
+    ret i32 %shoes, !dbg !18
+  }
+  
+  declare i32 @foo(...)
+  
+  ; Function Attrs: nounwind readnone speculatable willreturn
+  declare void @llvm.dbg.value(metadata, metadata, metadata)
+  
+  !llvm.dbg.cu = !{!0}
+  !llvm.module.flags = !{!3}
+  
+  !0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 2.9 (trunk 122997)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !2)
+  !1 = !DIFile(filename: "bar.c", directory: "/private/tmp")
+  !2 = !{}
+  !3 = !{i32 1, !"Debug Info Version", i32 3}
+  !4 = distinct !DISubprogram(name: "bar", scope: !1, file: !1, line: 5, type: !5, virtualIndex: 6, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !8)
+  !5 = !DISubroutineType(types: !6)
+  !6 = !{!7}
+  !7 = !DIBasicType(name: "int", size: 32, align: 32, encoding: DW_ATE_signed)
+  !8 = !{!9, !14}
+  !9 = !DILocalVariable(name: "b", arg: 1, scope: !4, file: !1, line: 5, type: !10)
+  !10 = !DIDerivedType(tag: DW_TAG_pointer_type, scope: !0, baseType: !11, size: 64, align: 64)
+  !11 = !DICompositeType(tag: DW_TAG_structure_type, name: "a", scope: !0, file: !1, line: 1, size: 32, align: 32, elements: !12)
+  !12 = !{!13}
+  !13 = !DIDerivedType(tag: DW_TAG_member, name: "c", scope: !1, file: !1, line: 2, baseType: !7, size: 32, align: 32)
+  !14 = !DILocalVariable(name: "x", scope: !15, file: !1, line: 6, type: !7)
+  !15 = distinct !DILexicalBlock(scope: !4, file: !1, line: 5, column: 22)
+  !16 = !DILocation(line: 5, column: 19, scope: !4)
+  !17 = !DILocation(line: 6, column: 14, scope: !15)
+  !18 = !DILocation(line: 8, column: 2, scope: !15)
+  !19 = !DILocation(line: 7, column: 2, scope: !15)
+
+...
+---
+name:            bar
+tracksRegLiveness: true
+stack:
+  - { id: 0, name: local1, type: default, offset: -24, size: 8, alignment: 8,
+      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+body:             |
+  bb.0.entry:
+    liveins: $rdi, $rsi
+  
+    %4:gr64= COPY $rsi
+    %2:gr64 = COPY $rdi
+    DBG_VALUE_LIST !14, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_arg, 1, DW_OP_plus), %2, %stack.0, debug-location !17
+    %3:gr64 = COPY killed %2
+    %5:gr64 = COPY killed %4
+
+    ; Force allocation into $rax and $rbx
+    INLINEASM &"", 1, 12, implicit-def dead $rcx, 12, implicit-def dead $rdx, 12, implicit-def dead $rsi, 12, implicit-def dead $rdi, 12, implicit-def $rbp, 12, implicit-def dead $r8, 12, implicit-def dead $r9, 12, implicit-def dead $r10, 12, implicit-def dead $r11, 12, implicit-def dead $r12, 12, implicit-def dead $r13, 12, implicit-def dead $r14, 12, implicit-def dead $r15, 12, !18, debug-location !17
+
+    ; Force a use of these two registers.
+    CALL64pcrel32 @foo, csr_64, implicit $rsp, implicit $ssp, implicit %3, implicit %5
+
+    ; Now make the register allocator move them to rcx and rdx!
+    INLINEASM &"", 1, 12, implicit-def dead $rax, 12, implicit-def dead $rbx, 12, implicit-def dead $rsi, 12, implicit-def dead $rdi, 12, implicit-def $rbp, 12, implicit-def dead $r8, 12, implicit-def dead $r9, 12, implicit-def dead $r10, 12, implicit-def dead $r11, 12, implicit-def dead $r12, 12, implicit-def dead $r13, 12, implicit-def dead $r14, 12, implicit-def dead $r15, 12, !18, debug-location !17
+
+    CALL64pcrel32 @foo, csr_64, implicit $rsp, implicit $ssp, implicit %3, implicit %5
+
+    RETQ
+...

llvm/test/DebugInfo/MIR/X86/live-debug-vars-unused-arg-debugonly.mir

 
 # Let's verify that the slot index ranges for the unused variables argc/argv,
 # connected to physical regs $edi and $rsi, does not overlap with the ranges
 # for %2 and %3. The register allocator is actually allocating the
 # virtual registers # to $edi and $esi, so the ranges for argc/argv should
 # not cover the whole BB.
 #
 # CHECKDBG-LABEL: ********** EMITTING LIVE DEBUG VARIABLES **********
-# CHECKDBG-NEXT: !"argc,5"        [0B;0e):0 Loc0=$edi
-# CHECKDBG-NEXT:         [0B;0e):0 %bb.0-160B
-# CHECKDBG-NEXT: !"argv,5"        [0B;0e):0 Loc0=$rsi
-# CHECKDBG-NEXT:         [0B;0e):0 %bb.0-160B
-# CHECKDBG-NEXT: !"a0,7"  [16r;64r):0 Loc0=%2
-# CHECKDBG-NEXT:         [16r;64r):0 %bb.0-160B
-# CHECKDBG-NEXT: !"a1,8"  [32r;80r):0 Loc0=%3
-# CHECKDBG-NEXT:         [32r;80r):0 %bb.0-160B
+# CHECKDBG-NEXT: !"argc,5"        [0B;0e): 0 Loc0=$edi
+# CHECKDBG-NEXT:         [0B;0e): 0 %bb.0-160B
+# CHECKDBG-NEXT: !"argv,5"        [0B;0e): 0 Loc0=$rsi
+# CHECKDBG-NEXT:         [0B;0e): 0 %bb.0-160B
+# CHECKDBG-NEXT: !"a0,7"  [16r;64r): 0 Loc0=%2
+# CHECKDBG-NEXT:         [16r;64r): 0 %bb.0-160B
+# CHECKDBG-NEXT: !"a1,8"  [32r;80r): 0 Loc0=%3
+# CHECKDBG-NEXT:         [32r;80r): 0 %bb.0-160B