[LiveDebugValues][InstrRef][2/2] Emit entry value variable locations

[jmorse]

Summary

Hi,

This patch adds support to the instruction-referencing LiveDebugValues implementation for emitting entry values. It depends on the parent patch to detect when variable locations are clobbered within blocks. Here are the headline figures from the 'PC Ranges covered' field of llvm-locstats:

	VarLocBasedLDV	InstrRefBasedLDV
No entry values	54%	56%
Entry values enabled	56%	61%

Which, assuming that I've followed the preconditions for emitting entry value locations, is an improvement. Full output of llvm-locstats can be found here [0]. ("Total availability" goes down a little because instr-ref-ldv is a little more aggressive about not propagating variable locations out of their scopes).

The instruction referencing implementations tracking by value rather than location means that we can get around two of the issues with the VarLoc implementation. DBG_VALUE instructions that re-assign the same value to a variable are no longer a problem, because we can "see through" to the value being assigned rather than just the location. We also don't need to do anything special during the dataflow stages: the "variable value problem" doesn't need to know whether a value is available most of the time, and the times it does needs to know are always when entry values need to be terminated (i.e., when differing variable values merge).

The patch modifies the "TransferTracker" class, which is responsible for turning maps of variable => values, and values => locations, into DBG_VALUE instructions within blocks. Two helper functions identify when a variable is an entry value candidate; and when a machine value is an entry value. recoverAsEntryValue tests these two things and emits a DBG_VALUE with an entry value expression if they're true. recoverAsEntryValue is called on two occasions: when a register (or spill slot) is clobbered and we can't find a replacement location for the value it contained; and on entry to a block, if the value isn't live at that point.

This can be tested by using the flag added in the parent patch: -force-instr-ref-livedebugvalues=1. I've added additional RUN lines to a bunch of entry value tests to ensure they work with InstRefBasedLDV: I found these by "git grep"'ing for 'entry_value' and ignoring anything that appeared to be testing call sites. Note that there's one serious modification to a test, kill-entry-value-after-diamond-bbs.mir. My reading of the test is that VarLocBasedLDV has to drop the entry value location because it doesn't know if the DBG_VALUEs in the diamond structure are new value assignments; wheras InstrRefBasedLDV can "see" that the DBG_VALUEs aren't assigning a new value. Advice most welcome: it might be best to update the test to ensure the DBG_VALUEs refer to a different value.

[0] https://gist.github.com/jmorse/a14763d935ef5d20e6e5dc41672c734a

[jmorse]

...additionally there are a few lines modified to better support instruction bundles (DBG_VALUEs were being inserted inside them on Sparc). I can split that out into a different patch if it gets int the way.

[djtodoro]

Hi @jmorse, cool, nice improvement! It turns out that this value-based value tracking pass makes it simple as well as effective!

My overall impression regarding the instr-based-ldv is very positive, but I haven't had much time to take a look into all the details regarding the implementation (but I have started looking into it). So I have a problem reading of some of these statements, e.g.:

DBG_VALUE instructions that re-assign the same value to a variable are no longer a problem, because we can "see through" to the value being assigned rather than just the location.

Can you please explain what part of the implementation of the instr-based-ldv gives us ability to "see through" to the value assignments?

llvm/test/DebugInfo/MIR/X86/kill-entry-value-after-diamond-bbs.mir

26

if we can "see through", it does seem reasonable to me.

[jmorse]

Djordje wrote:

Can you please explain what part of the implementation of the instr-based-ldv gives us ability to "see through" to the value assignments?

Good question, and it's probably time for a worked example of the whole "system", using kill-entry-value-after-diamond-bbs.mir. It's ideal because it doesn't contains loops, and loops are an entirely different kettle of fish. The overall summary is: we effectively compute a map for every instruction in the function recording which value is in each register, so we can "see" that the DBG_VALUEs assigns the same value that was already assigned to the variable. I'll be overly verbose below and use some tables to illustrate what's going on

Let's also pretend that I've applied D85760 to the tree -- it doesn't change the overall algorithm, but makes it easier to understand. Without it, steps one and three (below) are merged. Haven't gotten around to commiting D85760 yet :|

The kill-entry-value-after-diamond-bbs.mir test has the diamond shape:

   a
  / \
 /   \
b     c
 \   /
  \ /
   d

Each value in the function is represented by a unique ValueIDNum, which I'm representing below as "Value{a, b, c}", where 'a' is the block number, 'b' is the defining instructions number in the block or 'live-in', and 'c' identifies the register where the value is defined.

There are roughly five steps in the "ExtendRanges" method in InstrRefBasedLDV that deal with everything:

1. Produce machine value transfer function
2. Work out the live-in machine values for each block
3. Produce variable value transfer function
4. Work out the live-in variable values for each block
5. Produce variable locations for each variable in each block.

Step one: we produce a transfer function for each block. This is the same ``transfer function'' as defined by dataflow literature, it defines what values each block places in each location. Here's an example for the entry block in the example test:

$rsp = Value{0, 8, $rsp}
$edi = Value{0, 18, $edi}
$ebp = Value{0, live-in, $edx}
$edx = Value{0, 16, $edx}

It's a summary of how values move around in the block. All of $rsp, $edx and $edi are assigned new values in the block -- but $ebp is assigned a copy of $edx in the entry block. Whatever value is live-in to the block in $edx is copied into $ebp during the block. All registers that aren't defined in a block are live-through, whatever value comes in goes out in the same location. The transfer functions for the two diamond branches, block 1:

$ebx = Value{1, 4, $ebx}
$esi = Value{1, 3, $esi}

Block 2:

$ebp = Value{2, 4, $ebp}
$esi = Value{2, 3, $esi}

Which represents the registers defined by the MOV32ri instructions.

Step two: we use the "normal" dataflow process to work out what values are live-in to each block. The lattice used is a bit crazy, but ignoring loops it works like this: the initial live-ins to the entry block are these values (lets ignore rsp):

$edi = Value{0, live-in, $edi}
$esi = Value{0, live-in, $esi}
$ebx = Value{0, live-in, $ebx}
$ebp = Value{0, live-in, $ebp}
$edx = Value{0, live-in, $edx}

We can apply the transfer function above, and find the live-outs from the entry block to be:

$edi = Value{0, 18,  $edi}
$esi = Value{0, live-in, $esi}
$ebx = Value{0, live-in, $ebx}
$ebp = Value{0, live-in, $edx}
$edx = Value{0, 16, $edx}

Note that $edi and $edx have been assigned values; $ebp has been assigned whatever was in $edx on entry, and both $esi and $ebx are "live-through", so unchanged. Because blocks 1 and 2 only have the entry block as their predecessor, we can just copy the live-outs from the entry block to be the live-ins for blocks 1 and 2. We apply the transfer function from each block again to work out what their live-outs are: in block 1 $ebx and $esi are assigned values, while in block 2 $ebp and $esi are assigned values. Given the live-out values for blocks 1 and 2, we can compute the live-ins to block 3: but we have to resolve the different values being merged from the two predecessors: {$ebx, $esi, $ebp} have different values in each block. These are recorded as being PHI values, like this:

$edi = Value{0, 18,  $edi}
$esi = Value{3, live-in, $esi}
$ebx = Value{3, live-in, $ebx}
$ebp = Value{3, live-in, $ebx}
$edx = Value{0, 16, $edx}

$edi and $edx get the same Value because the predecessors agree on the value; wheras the other registers get new Values invented indicating that a PHI occurred on entry to block 3.

Key observation about this: once we have a set of live-in values for each block, we can work out a mapping that identifies what value each register has for every instruction. That brings us on to,

Step three (assume D85760 is applied, see the code immediately after mlocDataflow is called). For each block, we:

• Take its live-in values as calculated above (they get loaded into the MLocTracker class),
• Step through each instruction in the block again,
• For every DBG_VALUE instruction read the value in the specified register, and record it as a variable assignment.

How we "see through" those DBG_VALUE assignments should now be clear(er): We know the live-in value in $esi for blocks 1 and 2 is Value{0, live-in, $esi}, and we can track the copy to $ebx or $ebp respectively. When the DBG_VALUE instructions read either $ebx or $ebp, they will read Value{0, live-in, $esi}.

We then get the following variable value transfer functions for each block (looking only at !17):

entry: !17 = Value{0, live-in, $esi}
block1: !17 = Value{0, live-in, $esi}
block2: !17 = Value{0, live-in, $esi}

And can then perform:

Step four: another dataflow process with the variable transfer functions above, which calculates the live-in values for each variable. It's very simple in this example: the variable is assigned the same value in every block. It becomes much more painful with loops, but let's skip over that, and move to:

Step five: We now have a set of live-in values for each register for each block; and a set of live-in values for each variable in each block. For each block, we:

• Create a map of each Value to a location (register)
• For each live-in variable value, look up whether that value is in a register somewhere, and if so emit a DBG_VALUE,
• Also for each live-in variable, record what location we placed it in,
• Step through each instruction, and
  • If it's a "normal" instruction, record any definitions / copies of values that happen,
  • If it's a DBG_VALUE, update the value and location that the variable has,
  • If a "normal" instruction clobbers any variable locations, try to find a backup location and emit a DBG_VALUE.

Entry values can be emitted when the variable value is an entry value, i.e. it was the value live-in to the entry block Value{0, live-in, $somereg}, and we can't find a location when entering a block or when a register is clobbered.

~

So to repeat the summary from above: we have a map for every instruction recording what value is in each register. The difference that those DBG_VALUEs in blocks 1 and 2 make is that the variable value transfer function is:

entry: !17 = Value{0, live-in, $esi}
block1: !17 = Value{0, live-in, $esi}
block2: !17 = Value{0, live-in, $esi}

Wheras without them it would be:

entry: !17 = Value{0, live-in, $esi}

Where the value in entry would propagate into the other blocks anyway.

~

I'm still struggling to find a good way to describe how these things (are supposed to) work; hopefully spamming the information above will help a bit.

[djtodoro]

Thanks a lot for such a nice explanation! I've got it (at least for this example)!

The entry value work should be "simple" as well as more effective with this approach. Some initial comments inline.

llvm/lib/CodeGen/LiveDebugValues/InstrRefBasedImpl.cpp

3438

Should this be a separate patch? Does this affect/apply to the "copies" as backups?

llvm/test/DebugInfo/MIR/X86/livedebugvalues_load_in_loop.mir

83

Why including this here?

[jmorse]

llvm/lib/CodeGen/LiveDebugValues/InstrRefBasedImpl.cpp

3438

Shouldn't affect variable locations; I'll split it into a separate patch though and have a dedicated test case. The quick summary is that on X86 tailcalls are written:

TAILCALL @foo, implicit $rdi, $implicit-def $rax

Or similar, and are the last instruction in the block. InstrRefBasedLDV sees the implicit-def of $rax, and tries to recover variable locations afterwards, that leads to DBG_VALUEs being inserted after the terminator; which the MachineVerifier complains about.

There may be some exotic architectures out there that can have multiple terminators in their block, and def registers on the first terminator, which would be tricker to handle. But I think that bridge can be crossed when we come to it.

llvm/test/DebugInfo/MIR/X86/livedebugvalues_load_in_loop.mir

83

/me squints; I think I moved this because otherwise the the output looks like this:

DBG_VALUE $rdi
$rbp = MOV64rr $rdi
$rcx = MOV64ri 0
CALL64pcrel32 @bees
DBG_VALUE $rbp

i.e., the location is recovered to $rbp after the call clobbers $rdi. Wheras with this change, LiveDebugValues does not have to recover any clobbers.

My thinking was that, because this test is targeted at one thing VarLocBasedLDV does poorly, it's best to reduce the number of features being tested in the test, so that clobber being recovered from should be avoided.

[djtodoro]

llvm/lib/CodeGen/LiveDebugValues/InstrRefBasedImpl.cpp

3438

OK. That is fine with me. We avoid such entry-values within old-(varloc-based)-ldv as well.

There may be some exotic architectures out there that can have multiple terminators in their block, and def registers on the first terminator, which would be tricker to handle. But I think that bridge can be crossed when we come to it.

I think that I have faced such case on the PPC64, but as long as we have the MachineVerifier checking that, we are safe.

llvm/test/DebugInfo/MIR/X86/livedebugvalues_load_in_loop.mir

83

OK then :)

Does moving it into the previous change(D88405) make sense to you?

[jmorse]

Rebase to account for me gluing patch series together; one or two new code paths crop up where one has to decide whether to try and use an entry value OR register a debug use-before-def.

llvm/test/DebugInfo/MIR/X86/livedebugvalues_load_in_loop.mir

83

Yeah, I've got an intermediate patch to stick between the two that sorts these things out, I'm doing a little patch gardening first though., should appear shortly.

[djtodoro]

llvm/test/DebugInfo/ARM/entry-value-multi-byte-expr.ll

2

is the -force-instr-ref-livedebugvalues=1 equivalent to the -experimental-debug-variable-locations?

[jmorse]

llvm/test/DebugInfo/ARM/entry-value-multi-byte-expr.ll

2

Not quite, the truth table is:

	Produce DBG_INSTR_REF in isel?	Use InstrRefBasedLDV?
no options	False	False
-force-isntr-ref-livedebugvalues=1	False	True
-experimental-debug-variable-locations	True	True

Thus, this option lets you can explore how InstrRefBasedLDV behaves compared to VarLocBasedLDV without using all the other instruction referencing work. The two implementations should produce the same output at all times, aside from a few limitations, see the "Testing and Validation" section in:

http://lists.llvm.org/pipermail/llvm-dev/2020-June/142368.html

(There are still a couple of minor bugs hanging around InstrRefBasedLDV, and this is helping me hunt them).

[djtodoro]

llvm/test/DebugInfo/ARM/entry-value-multi-byte-expr.ll

2

Oh, I see.. It will be in after D88405 is landed.

Thanks!

[djtodoro]

LGTM; Please remove "Entry values" from the top-level comment TODO list.

llvm/lib/CodeGen/LiveDebugValues/InstrRefBasedImpl.cpp

 #include "llvm/ADT/UniqueVector.h"
 #include "llvm/CodeGen/LexicalScopes.h"
 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
+#include "llvm/CodeGen/MachineInstrBundle.h"
 #include "llvm/CodeGen/MachineMemOperand.h"
 #include "llvm/CodeGen/MachineOperand.h"
 #include "llvm/CodeGen/PseudoSourceValue.h"
 #include "llvm/CodeGen/RegisterScavenging.h"
 #include "llvm/CodeGen/TargetFrameLowering.h"
 #include "llvm/CodeGen/TargetInstrInfo.h"
 #include "llvm/CodeGen/TargetLowering.h"
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/IR/DIBuilder.h"
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/DebugLoc.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/Module.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/MC/MCRegisterInfo.h"
 #include "llvm/Pass.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/TypeSize.h"
 #include "llvm/Support/raw_ostream.h"
+#include "llvm/Target/TargetMachine.h"
 #include "llvm/Transforms/Utils/SSAUpdaterImpl.h"
 #include <algorithm>
 #include <cassert>
 #include <cstdint>
 #include <functional>
 #include <queue>
 #include <tuple>
 #include <utility>

 ///
 /// This is where debug use-before-defs would be resolved: a variable with an
 /// unavailable value could materialize in the middle of a block, when the
 /// value becomes available. Or, we could detect clobbers and re-specify the
 /// variable in a backup location. (XXX these are unimplemented).
 class TransferTracker {
 public:
   const TargetInstrInfo *TII;
+  const TargetLowering *TLI;
   /// This machine location tracker is assumed to always contain the up-to-date
   /// value mapping for all machine locations. TransferTracker only reads
   /// information from it. (XXX make it const?)
   MLocTracker *MTracker;
   MachineFunction &MF;
+  bool ShouldEmitDebugEntryValues;
 
   /// Record of all changes in variable locations at a block position. Awkwardly
   /// we allow inserting either before or after the point: MBB != nullptr
   /// indicates it's before, otherwise after.
   struct Transfer {
-    MachineBasicBlock::iterator Pos; /// Position to insert DBG_VALUes
-    MachineBasicBlock *MBB;          /// non-null if we should insert after.
+    MachineBasicBlock::instr_iterator Pos; /// Position to insert DBG_VALUes
+    MachineBasicBlock *MBB; /// non-null if we should insert after.
     SmallVector<MachineInstr *, 4> Insts; /// Vector of DBG_VALUEs to insert.
   };
 
   struct LocAndProperties {
     LocIdx Loc;
     DbgValueProperties Properties;
   };
 
public:
   /// collection prevents the use-before-def materializing.
   DenseSet<DebugVariable> UseBeforeDefVariables;
 
   const TargetRegisterInfo &TRI;
   const BitVector &CalleeSavedRegs;
 
   TransferTracker(const TargetInstrInfo *TII, MLocTracker *MTracker,
                   MachineFunction &MF, const TargetRegisterInfo &TRI,
-                  const BitVector &CalleeSavedRegs)
+                  const BitVector &CalleeSavedRegs, const TargetPassConfig &TPC)
       : TII(TII), MTracker(MTracker), MF(MF), TRI(TRI),
-        CalleeSavedRegs(CalleeSavedRegs) {}
+        CalleeSavedRegs(CalleeSavedRegs) {
+    TLI = MF.getSubtarget().getTargetLowering();
+    auto &TM = TPC.getTM<TargetMachine>();
+    ShouldEmitDebugEntryValues = TM.Options.ShouldEmitDebugEntryValues();
+  }
 
   /// Load object with live-in variable values. \p mlocs contains the live-in
   /// values in each machine location, while \p vlocs the live-in variable
   /// values. This method picks variable locations for the live-in variables,
   /// creates DBG_VALUEs and puts them in #Transfers, then prepares the other
   /// object fields to track variable locations as we step through the block.
   /// FIXME: could just examine mloctracker instead of passing in \p mlocs?
   void loadInlocs(MachineBasicBlock &MBB, ValueIDNum *MLocs,
for (auto Var : VLocs) {
       // If the value has no location, we can't make a variable location.
       const ValueIDNum &Num = Var.second.ID;
       auto ValuesPreferredLoc = ValueToLoc.find(Num);
       if (ValuesPreferredLoc == ValueToLoc.end()) {
         // If it's a def that occurs in this block, register it as a
         // use-before-def to be resolved as we step through the block.
         if (Num.getBlock() == (unsigned)MBB.getNumber() && !Num.isPHI())
           addUseBeforeDef(Var.first, Var.second.Properties, Num);
+        else
+          recoverAsEntryValue(Var.first, Var.second.Properties, Num);
         continue;
       }
 
       LocIdx M = ValuesPreferredLoc->second;
       auto NewValue = LocAndProperties{M, Var.second.Properties};
       auto Result = ActiveVLocs.insert(std::make_pair(Var.first, NewValue));
       if (!Result.second)
         Result.first->second = NewValue;
for (auto &Use : MIt->second) {
 
       PendingDbgValues.push_back(MTracker->emitLoc(L, Use.Var, Use.Properties));
     }
     flushDbgValues(pos, nullptr);
   }
 
   /// Helper to move created DBG_VALUEs into Transfers collection.
   void flushDbgValues(MachineBasicBlock::iterator Pos, MachineBasicBlock *MBB) {
-    if (PendingDbgValues.size() > 0) {
-      Transfers.push_back({Pos, MBB, PendingDbgValues});
-      PendingDbgValues.clear();
-    }
+    if (PendingDbgValues.size() == 0)
+      return;
+
+    // Pick out the instruction start position.
+    MachineBasicBlock::instr_iterator BundleStart;
+    if (MBB && Pos == MBB->begin())
+      BundleStart = MBB->instr_begin();
+    else
+      BundleStart = getBundleStart(Pos->getIterator());
+
+    Transfers.push_back({BundleStart, MBB, PendingDbgValues});
+
+  bool isEntryValueVariable(const DebugVariable &Var,
+                            const DIExpression *Expr) const {
+    if (!Var.getVariable()->isParameter())
+      return false;
+
+    if (Var.getInlinedAt())
+      return false;
+
+    if (Expr->getNumElements() > 0)
+      return false;
+
+    return true;
+  }
+
+  bool isEntryValueValue(const ValueIDNum &Val) const {
+    // Must be in entry block (block number zero), and be a PHI / live-in value.
+    if (Val.getBlock() || !Val.isPHI())
+      return false;
+
+    // Entry values must enter in a register.
+    if (MTracker->isSpill(Val.getLoc()))
+      return false;
+
+    Register SP = TLI->getStackPointerRegisterToSaveRestore();
+    Register FP = TRI.getFrameRegister(MF);
+    Register Reg = MTracker->LocIdxToLocID[Val.getLoc()];
+    return Reg != SP && Reg != FP;
+  }
+
+  bool recoverAsEntryValue(const DebugVariable &Var, DbgValueProperties &Prop,
+                           const ValueIDNum &Num) {
+    // Is this variable location a candidate to be an entry value. First,
+    // should we be trying this at all?
+    if (!ShouldEmitDebugEntryValues)
+      return false;
+
+    // Is the variable appropriate for entry values (i.e., is a parameter).
+    if (!isEntryValueVariable(Var, Prop.DIExpr))
+      return false;
+
+    // Is the value assigned to this variable still the entry value?
+    if (!isEntryValueValue(Num))
+      return false;
+
+    // Emit a variable location using an entry value expression.
+    DIExpression *NewExpr =
+        DIExpression::prepend(Prop.DIExpr, DIExpression::EntryValue);
+    Register Reg = MTracker->LocIdxToLocID[Num.getLoc()];
+    MachineOperand MO = MachineOperand::CreateReg(Reg, false);
+    MO.setIsDebug(true);
+
+    PendingDbgValues.push_back(emitMOLoc(MO, Var, {NewExpr, Prop.Indirect}));
+    return true;
   }
 
   /// Change a variable value after encountering a DBG_VALUE inside a block.
   void redefVar(const MachineInstr &MI) {
     DebugVariable Var(MI.getDebugVariable(), MI.getDebugExpression(),
                       MI.getDebugLoc()->getInlinedAt());
     DbgValueProperties Properties(MI);
 
void clobberMloc(LocIdx MLoc, MachineBasicBlock::iterator Pos,
     // again, we can recover the location.
     Optional<LocIdx> NewLoc = None;
     for (auto Loc : MTracker->locations())
       if (Loc.Value == OldValue)
         NewLoc = Loc.Idx;
 
     // If there is no location, and we weren't asked to make the variable
     // explicitly undef, then stop here.
-    if (!NewLoc && !MakeUndef)
+    if (!NewLoc && !MakeUndef) {
+      // Try and recover a few more locations with entry values.
+      for (auto &Var : ActiveMLocIt->second) {
+        auto &Prop = ActiveVLocs.find(Var)->second.Properties;
+        recoverAsEntryValue(Var, Prop, OldValue);
+      }
+      flushDbgValues(Pos, nullptr);
       return;
+    }
 
     // Examine all the variables based on this location.
     DenseSet<DebugVariable> NewMLocs;
     for (auto &Var : ActiveMLocIt->second) {
       auto ActiveVLocIt = ActiveVLocs.find(Var);
       // Re-state the variable location: if there's no replacement then NewLoc
       // is None and a $noreg DBG_VALUE will be created. Otherwise, a DBG_VALUE
       // identifying the alternative location will be emitted.
private:
   /// in \p MInLocs and live-in variable values in \p SavedLiveIns, runs the
   /// TransferTracker class over the function to produce live-in and transfer
   /// DBG_VALUEs, then inserts them. Groups of DBG_VALUEs are inserted in the
   /// order given by AllVarsNumbering -- this could be any stable order, but
   /// right now "order of appearence in function, when explored in RPO", so
   /// that we can compare explictly against VarLocBasedImpl.
   void emitLocations(MachineFunction &MF, LiveInsT SavedLiveIns,
                      ValueIDNum **MOutLocs, ValueIDNum **MInLocs,
-                     DenseMap<DebugVariable, unsigned> &AllVarsNumbering);
+                     DenseMap<DebugVariable, unsigned> &AllVarsNumbering,
+                     const TargetPassConfig &TPC);
 
   /// Boilerplate computation of some initial sets, artifical blocks and
   /// RPOT block ordering.
   void initialSetup(MachineFunction &MF);
 
   bool ExtendRanges(MachineFunction &MF, TargetPassConfig *TPC) override;
 
 public:
for (auto &P : mloc_transfer) {
     std::string bar = MTracker->IDAsString(P.second);
     dbgs() << "Loc " << foo << " --> " << bar << "\n";
   }
 }
 #endif
 
 void InstrRefBasedLDV::emitLocations(
     MachineFunction &MF, LiveInsT SavedLiveIns, ValueIDNum **MOutLocs,
-    ValueIDNum **MInLocs, DenseMap<DebugVariable, unsigned> &AllVarsNumbering) {
-  TTracker = new TransferTracker(TII, MTracker, MF, *TRI, CalleeSavedRegs);
+    ValueIDNum **MInLocs, DenseMap<DebugVariable, unsigned> &AllVarsNumbering,
+    const TargetPassConfig &TPC) {
+  TTracker = new TransferTracker(TII, MTracker, MF, *TRI, CalleeSavedRegs, TPC);
   unsigned NumLocs = MTracker->getNumLocs();
 
   // For each block, load in the machine value locations and variable value
   // live-ins, then step through each instruction in the block. New DBG_VALUEs
   // to be inserted will be created along the way.
   for (MachineBasicBlock &MBB : MF) {
     unsigned bbnum = MBB.getNumber();
     MTracker->reset();
for (auto &P : TTracker->Transfers) {
     llvm::sort(P.Insts, OrderDbgValues);
     // Insert either before or after the designated point...
     if (P.MBB) {
       MachineBasicBlock &MBB = *P.MBB;
       for (auto *MI : P.Insts) {
         MBB.insert(P.Pos, MI);
       }
     } else {
+      // Terminators, like tail calls, can clobber things. Don't try and place
+      // transfers after them.
+      if (P.Pos->isTerminator())
+        continue;
+
       MachineBasicBlock &MBB = *P.Pos->getParent();
       for (auto *MI : P.Insts) {
-        MBB.insertAfter(P.Pos, MI);
+        MBB.insertAfterBundle(P.Pos, MI);
       }
     }
   }
 }
 
 void InstrRefBasedLDV::initialSetup(MachineFunction &MF) {
   // Build some useful data structures.
   auto hasNonArtificialLocation = [](const MachineInstr &MI) -> bool {
for (auto &P : ScopeToVars) {
     vlocDataflow(P.first, ScopeToDILocation[P.first], P.second,
                  ScopeToBlocks[P.first], SavedLiveIns, MOutLocs, MInLocs,
                  vlocs);
   }
 
   // Using the computed value locations and variable values for each block,
   // create the DBG_VALUE instructions representing the extended variable
   // locations.
-  emitLocations(MF, SavedLiveIns, MOutLocs, MInLocs, AllVarsNumbering);
+  emitLocations(MF, SavedLiveIns, MOutLocs, MInLocs, AllVarsNumbering, *TPC);
 
   for (int Idx = 0; Idx < MaxNumBlocks; ++Idx) {
     delete[] MOutLocs[Idx];
     delete[] MInLocs[Idx];
   }
   delete[] MOutLocs;
   delete[] MInLocs;
 

llvm/test/DebugInfo/ARM/entry-value-multi-byte-expr.ll

 ; RUN: llc -debug-entry-values -filetype=asm -o - %s | FileCheck %s
+; RUN: llc -force-instr-ref-livedebugvalues=1 -debug-entry-values -filetype=asm -o - %s | FileCheck %s
 
 ; Verify that the size operands of the DW_OP_GNU_entry_value operations are
 ; correct for the multi-byte DW_OP_regx expressions.
 
 ; Based on the following C reproducer:
 ;
 ; extern void clobber();
 ; double global;

llvm/test/DebugInfo/MIR/Hexagon/live-debug-values-bundled-entry-values.mir

 # We do not support the call site info for the target now, so we use the experimental option (-emit-call-site-info -debug-entry-values).
 
 # RUN: llc -emit-call-site-info -debug-entry-values -run-pass=livedebugvalues -o - %s | FileCheck %s
+# RUN: llc -emit-call-site-info -debug-entry-values -force-instr-ref-livedebugvalues=1 -run-pass=livedebugvalues -o - %s | FileCheck %s
 
 # Verify that the entry values for the input parameters are inserted after the
 # bundles which contains the registers' clobbering instructions (the calls to
 # @clobber). Previously the insertions would thrash the bundles, later on
 # triggering a crash in DebugHandlerBase.
 
 --- |
   target datalayout = "e-m:e-p:32:32:32-a:0-n16:32-i64:64:64-i32:32:32-i16:16:16-i1:8:8-f32:32:32-f64:64:64-v32:32:32-v64:64:64-v512:512:512-v1024:1024:1024-v2048:2048:2048"

llvm/test/DebugInfo/MIR/X86/dbginfo-entryvals.mir

 # RUN: llc -run-pass=livedebugvalues -verify-machineinstrs -march=x86-64 -o - %s | FileCheck %s
+# RUN: llc -force-instr-ref-livedebugvalues=1 -run-pass=livedebugvalues -verify-machineinstrs -march=x86-64 -o - %s | FileCheck %s
 #
 #extern void fn2(int);
 #
 #__attribute__((noinline))
 #void
 #fn1 (int x, int y) {
 #  int u = x + y;
 #  if (x > 1)

llvm/test/DebugInfo/MIR/X86/debug-entry-value-operation.mir

 # RUN: llc -start-before=livedebugvalues -mtriple=x86_64-apple-darwin -o %t %s -filetype=obj
 # RUN: llvm-dwarfdump %t | FileCheck %s
 
+# RUN: llc -force-instr-ref-livedebugvalues=1 -start-before=livedebugvalues -mtriple=x86_64-apple-darwin -o %t %s -filetype=obj
+# RUN: llvm-dwarfdump %t | FileCheck %s
+
 # RUN: llc -start-before=livedebugvalues -debugger-tune=sce -mtriple=x86_64-sce-ps4 -o %t1 %s -filetype=obj
 # RUN: llvm-dwarfdump %t1 | FileCheck %s -check-prefix=SCE
 
+# RUN: llc -force-instr-ref-livedebugvalues=1 -start-before=livedebugvalues -debugger-tune=sce -mtriple=x86_64-sce-ps4 -o %t1 %s -filetype=obj
+# RUN: llvm-dwarfdump %t1 | FileCheck %s -check-prefix=SCE
+
 ## Based on:
 ## int global;
 ## int foo(int p, int q, int r) {
 ##   global = p + 1;
 ##   asm __volatile("" : : : "edi", "esi", "edx");
 ##   return 123;
 ## }
 

llvm/test/DebugInfo/MIR/X86/entry-value-of-modified-param.mir

 # RUN: llc -run-pass=livedebugvalues -march=x86-64 -o - %s | FileCheck %s
+# RUN: llc -force-instr-ref-livedebugvalues=1 -run-pass=livedebugvalues -march=x86-64 -o - %s | FileCheck %s
 #
 #extern void fn1 (int, int, int);
 #
 #__attribute__((noinline))
 #int
 #fn2 (int a, int b, int c) {
 #  int q = 2 + a;
 #

llvm/test/DebugInfo/MIR/X86/entry-values-diamond-bbs.mir

 # RUN: llc -run-pass=livedebugvalues -march=x86-64 -o - %s | FileCheck %s
+# RUN: llc -force-instr-ref-livedebugvalues=1 -run-pass=livedebugvalues -march=x86-64 -o - %s | FileCheck %s
 #
 # The test case was artificially adjusted, in order to make proper diamond basic
 # block structure relevant to the debug entry values propagation.
 #
 # CHECK: ![[ARG_B:.*]] = !DILocalVariable(name: "b"
 # CHECK: ![[ARG_C:.*]] = !DILocalVariable(name: "c"
 # CHECK: ![[ARG_Q:.*]] = !DILocalVariable(name: "q"
 # CHECK: bb.0.entry

llvm/test/DebugInfo/MIR/X86/kill-entry-value-after-diamond-bbs.mir

-# RUN: llc -debug-entry-values -run-pass=livedebugvalues -march=x86-64 -o - %s | FileCheck %s
+# RUN: llc -debug-entry-values -run-pass=livedebugvalues -march=x86-64 -o - %s | FileCheck %s --check-prefixes=CHECK,VARLOCLDV
+# RUN: llc -debug-entry-values -force-instr-ref-livedebugvalues=1 -run-pass=livedebugvalues -march=x86-64 -o - %s | FileCheck %s --check-prefixes=CHECK,INSTRREFLDV
 #
 # The test case was artificially adjusted, in order to make proper diamond basic
 # block structure relevant to the debug entry values clobbering.
 #
 # CHECK: ![[ARG_B:.*]] = !DILocalVariable(name: "b"
 # CHECK: bb.0.entry
 # CHECK: DBG_VALUE $esi, $noreg, ![[ARG_B]], !DIExpression()
 # CHECK: bb.1.if.then
 # CHECK: DBG_VALUE $esi, $noreg, ![[ARG_B]], !DIExpression()
 # CHECK: $ebx = MOV32rr $esi
 # CHECK-NEXT: DBG_VALUE $ebx, $noreg, ![[ARG_B]], !DIExpression()
 # CHECK-NEXT: $esi = MOV32ri 5
 # CHECK-NEXT: $ebx = MOV32ri 1
 # CHECK-NEXT: DBG_VALUE $esi, $noreg, ![[ARG_B]], !DIExpression(DW_OP_LLVM_entry_value, 1)
 # CHECK: bb.2.if.else
 # CHECK: DBG_VALUE $esi, $noreg, ![[ARG_B]], !DIExpression()
 # CHECK: $ebp = MOV32rr $esi
 # CHECK: DBG_VALUE $ebp, $noreg, ![[ARG_B]], !DIExpression()
 # CHECK-NEXT: $esi = MOV32ri 1
 # CHECK-NEXT: $ebp = MOV32ri 2
 # CHECK-NEXT: DBG_VALUE $esi, $noreg, ![[ARG_B]], !DIExpression(DW_OP_LLVM_entry_value, 1)
 # CHECK: bb.3.if.end
-# CHECK-NOT: DBG_VALUE $esi, $noreg, ![[ARG_B]], !DIExpression(DW_OP_LLVM_entry_value, 1)
+# VARLOCLDV-NOT: DBG_VALUE $esi, $noreg, ![[ARG_B]], !DIExpression(DW_OP_LLVM_entry_value, 1)
+# INSTRREFLDV: DBG_VALUE $esi, $noreg, ![[ARG_B]], !DIExpression(DW_OP_LLVM_entry_value, 1)
+#
+## Final two lines: VarLoc LiveDebugValues cannot determine that the DBG_VALUEs
+## down either path of the diamond set the variable to be its original value,
+## wheras instruction referencing LiveDebugValues can.
 #
 --- |
   ; ModuleID = 'test.c'
   source_filename = "test.c"
   target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
   target triple = "x86_64-unknown-linux-gnu"
 
   ; Function Attrs: noinline nounwind uwtable

llvm/test/DebugInfo/MIR/X86/live-debug-values-entry-transfer.mir

 # RUN: llc %s -o - -run-pass=livedebugvalues -mtriple=x86_64-unknown-unknown | FileCheck %s
+# RUN: llc %s -o - -force-instr-ref-livedebugvalues=1 -run-pass=livedebugvalues -mtriple=x86_64-unknown-unknown | FileCheck %s
 #
 # In this lightly modified test case, the transfer in the entry block from
 # geti32's return value in $eax to the non-volatile $ebx should be recognized,
 # and propagated throughout the whole function.
 #
 # CHECK-LABEL: bb.0.entry
 # CHECK:       DBG_VALUE $eax
 # CHECK:       DBG_VALUE $ebx

llvm/test/DebugInfo/MIR/X86/livedebugvalues_load_in_loop.mir

 --- |
   ; RUN: llc %s -march=x86-64 -run-pass=livedebugvalues -o - -experimental-debug-variable-locations -emulate-old-livedebugvalues=0 | FileCheck %s -implicit-check-not=DBG_VALUE
 
   ; Sometimes, variables can have multiple locations, and when control flow
   ; merges LiveDebugValues has a hard time picking which one the variable lives
   ; in. Test two of these scenarios that old LiveDebugValues can't handle: when
   ; a value is in two registers, and when a value is both in a register and
   ; on the stack.
 
   ; In a register:
 
   ; CHECK-LABEL: bb.0.entry:
-  ; CHECK:       DBG_VALUE $rdi, $noreg, !16, !DIExpression()
   ; CHECK:       DBG_VALUE $rbp, $noreg, !16, !DIExpression()
   ; CHECK-LABEL: bb.1.bb1:
   ; CHECK:       DBG_VALUE $rbp, $noreg, !16, !DIExpression()
   ; CHECK:       DBG_VALUE $rbp, $noreg, !16, !DIExpression()
   ; CHECK-LABEL: bb.2.bb2:
   ; CHECK:       DBG_VALUE $rbp, $noreg, !16, !DIExpression()
   ; CHECK-LABEL: bb.3.bb3:
   ; CHECK:       DBG_VALUE $rbp, $noreg, !16, !DIExpression()

 stack:
   - { id: 0, name: '', type: spill-slot, offset: -8, size: 8, alignment: 8,
       stack-id: default, callee-saved-register: '', callee-saved-restored: true,
       debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
 body:  |
   bb.0.entry:
   liveins: $rdi
     successors: %bb.1, %bb.2
-    DBG_VALUE $rdi, $noreg, !16, !DIExpression(), debug-location !17
     $rbp = MOV64rr $rdi, debug-location !17
+    DBG_VALUE $rbp, $noreg, !16, !DIExpression(), debug-location !17
     dead $rcx = MOV64ri 0, debug-location !17
     CALL64pcrel32 @bees, csr_64, implicit $rsp, implicit $ssp, implicit $rdi, implicit-def $rsp, implicit-def $ssp, implicit-def $rax, debug-location !17
     CMP64ri8 renamable $rax, 1, implicit-def $eflags, debug-location !17
     JCC_1 %bb.2, 4, implicit killed $eflags, debug-location !17
   bb.1.bb1:
   liveins: $rax, $rbp
     successors: %bb.3
     $rbp = MOV64ri 0, debug-location !17

llvm/test/DebugInfo/MIR/X86/multiple-param-dbg-value-entry.mir

 # RUN: llc -debug-entry-values -run-pass=livedebugvalues -march=x86-64 -o - %s| FileCheck %s
+# RUN: llc -force-instr-ref-livedebugvalues=1 -debug-entry-values -run-pass=livedebugvalues -march=x86-64 -o - %s| FileCheck %s
 ## Test case:
 ## int global;
 ## int foo(int p, int q, int r) {
 ##   global = p + 1;
 ##   asm __volatile("" : : : "edi", "esi", "edx");
 ##   return 123;
 ## }
 ##
 ##  Verify that DW_OP_LLVM_entry_values are generated for parameters with multiple
 ##  DBG_VALUEs at entry block.
 # CHECK: DBG_VALUE $edi, $noreg, !{{.*}}, !DIExpression(DW_OP_LLVM_entry_value, 1), debug-location {{.*}}
-# CHECK: DBG_VALUE $esi, $noreg, !{{.*}}, !DIExpression(DW_OP_LLVM_entry_value, 1), debug-location {{.*}}
-# CHECK: DBG_VALUE $edx, $noreg, !{{.*}}, !DIExpression(DW_OP_LLVM_entry_value, 1), debug-location {{.*}}
+# CHECK: INLINEASM
+# CHECK-DAG: DBG_VALUE $esi, $noreg, !{{.*}}, !DIExpression(DW_OP_LLVM_entry_value, 1), debug-location {{.*}}
+# CHECK-DAG: DBG_VALUE $edx, $noreg, !{{.*}}, !DIExpression(DW_OP_LLVM_entry_value, 1), debug-location {{.*}}
+# CHECK $eax = MOV32ri 123
 
 --- |
   ; ModuleID = 'multiple-param-dbg-value-entry.ll'
   source_filename = "multiple-param-dbg-value-entry.c"
   target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
   @global = common dso_local local_unnamed_addr global i32 0, align 4, !dbg !0
 

llvm/test/DebugInfo/MIR/X86/propagate-entry-value-cross-bbs.mir

 # RUN: llc -run-pass=livedebugvalues -march=x86-64 -o - %s | FileCheck %s
+# RUN: llc -force-instr-ref-livedebugvalues=1 -run-pass=livedebugvalues -march=x86-64 -o - %s | FileCheck %s
 #
 #extern void fn1 (int, int, int);
 #__attribute__((noinline))
 #int
 #fn2 (int a, int b, int c) {
 #  int q = 2 + a;
 #  fn1 (5, 6, q);
 #  if (b < 17) {

llvm/test/DebugInfo/Mips/dw_op_entry_value_32bit.ll

 ;; Test mips32:
 ; RUN: llc -emit-call-site-info -stop-after=livedebugvalues -mtriple=mips-linux-gnu -o - %s | FileCheck %s
+; RUN: llc -force-instr-ref-livedebugvalues=1 -emit-call-site-info -stop-after=livedebugvalues -mtriple=mips-linux-gnu -o - %s | FileCheck %s
 ;; Test mipsel:
 ; RUN: llc -emit-call-site-info -stop-after=livedebugvalues -mtriple=mipsel-linux-gnu -o - %s | FileCheck %s --check-prefix=CHECKel
+; RUN: llc -force-instr-ref-livedebugvalues=1 -emit-call-site-info -stop-after=livedebugvalues -mtriple=mipsel-linux-gnu -o - %s | FileCheck %s --check-prefix=CHECKel
 
 ;; Built from source:
 ;; extern long fn1(long,long,long);
 ;; long fn2(long a, long b, long c) {
 ;;   long local = fn1(a+b, c, b+10);
 ;;   if (local > 10)
 ;;     return local + 10;
 ;;   return b;

llvm/test/DebugInfo/Mips/dw_op_entry_value_64bit.ll

 ;; Test mips64:
 ; RUN: llc -emit-call-site-info -stop-after=livedebugvalues -mtriple=mips64-linux-gnu -o - %s | FileCheck %s --check-prefix=CHECK64
+; RUN: llc -force-instr-ref-livedebugvalues=1 -emit-call-site-info -stop-after=livedebugvalues -mtriple=mips64-linux-gnu -o - %s | FileCheck %s --check-prefix=CHECK64
 ;; Test mips64el:
 ; RUN: llc -emit-call-site-info -stop-after=livedebugvalues -mtriple=mips64el-linux-gnu -o - %s | FileCheck %s --check-prefix=CHECK64el
+; RUN: llc -force-instr-ref-livedebugvalues=1 -emit-call-site-info -stop-after=livedebugvalues -mtriple=mips64el-linux-gnu -o - %s | FileCheck %s --check-prefix=CHECK64el
 
 ;; Built from source:
 ;; extern long fn1(long,long,long);
 ;; long fn2(long a, long b, long c) {
 ;;   long local = fn1(a+b, c, b+10);
 ;;   if (local > 10)
 ;;     return local + 10;
 ;;   return b;

llvm/test/DebugInfo/Sparc/entry-value-complex-reg-expr.ll

 ; RUN: llc -debug-entry-values -filetype=asm -o - %s | FileCheck %s
+; RUN: llc -force-instr-ref-livedebugvalues=1 -debug-entry-values -filetype=asm -o - %s | FileCheck %s
 
 ; The q0 register does not have a DWARF register number, and is instead emitted
 ; as a composite location description with two sub-registers. Previously we
 ; emitted a single DW_OP_entry_value wrapping that whole composite location
 ; description, but that is not valid DWARF; DW_OP_entry_value operations can
 ; only hold DWARF expressions and register location descriptions.
 ;
 ; In the future we may want to emit a composite location description where each

llvm/test/DebugInfo/X86/dbg-value-regmask-clobber.ll

 ; RUN: llc < %s | FileCheck %s --check-prefix=ASM
+; RUN: llc -force-instr-ref-livedebugvalues=1 < %s | FileCheck %s --check-prefix=ASM
 ; RUN: llc < %s -filetype=obj | llvm-dwarfdump - | FileCheck %s --check-prefix=DWARF
+; RUN: llc -force-instr-ref-livedebugvalues=1 < %s -filetype=obj | llvm-dwarfdump - | FileCheck %s --check-prefix=DWARF
 
 ; Values in registers should be clobbered by calls, which use a regmask instead
 ; of individual register def operands.
 
 ; ASM: main: # @main
 ; ASM: #DEBUG_VALUE: main:argc <- $ecx
 ; ASM: movl $1, x(%rip)
 ; ASM: callq clobber

llvm/test/DebugInfo/X86/no-entry-values-with-O0.ll

 ; RUN: llc -O0  -dwarf-version=5 -debugger-tune=lldb -march=x86-64 -filetype=obj < %s \
 ; RUN:  | llvm-dwarfdump - | FileCheck --implicit-check-not=DW_OP_entry_value %s
 ; RUN: llc -O0  -dwarf-version=5 -debugger-tune=gdb -march=x86-64 -filetype=obj < %s \
 ; RUN:  | llvm-dwarfdump - | FileCheck --implicit-check-not=DW_OP_entry_value %s
+; RUN: llc -force-instr-ref-livedebugvalues=1 -O0  -dwarf-version=5 -debugger-tune=lldb -march=x86-64 -filetype=obj < %s \
+; RUN:  | llvm-dwarfdump - | FileCheck --implicit-check-not=DW_OP_entry_value %s
+; RUN: llc -force-instr-ref-livedebugvalues=1 -O0  -dwarf-version=5 -debugger-tune=gdb -march=x86-64 -filetype=obj < %s \
+; RUN:  | llvm-dwarfdump - | FileCheck --implicit-check-not=DW_OP_entry_value %s
 
 ; The call-site-params are created iff corresponding DISubprogram contains
 ; the AllCallsDescribed DIFlag.
 ; CHECK-NOT: DW_TAG_call_site_param
 
 ; Genarated with:
 ; clang -gdwarf-5 -O0 test.c -S -emit-llvm
 ;