MachineScheduler: Allow independent scheduling of sub register defs

[MatzeB]

Summary

If the TrackLaneMasks policy/strategy is enabled the MachineScheduler
will build a schedule graph where definitions of independent
subregisters are no longer serialised.

Implementation comments:

• Without lane mask tracking a sub register def also counts as a use (except for the first one with the read-undef flag set), with lane mask tracking enabled this is no longer the case.
• Pressure Diffs where previously maintained per definition of a vreg with the help of the SSA information contained in the LiveIntervals. With lanemask tracking enabled we cannot do this anymore and instead change the pressure diffs for all uses of the vreg as it becomes live/dead. For this changed style to work correctly we ignore uses of instructions that define the same register again: They won't affect register pressure.
• With lanemask tracking we remove all read-undef flags from sub register defs when building the graph and re-add them later when all vreg lanes have become dead.

[atrick]

How does this avoid creating multiple disjoint live subranges for a single virtual register? Will verifyLiveInterval choke on this?

RegisterPressureTracker now mutates the machine instructions which is conceptually wrong. If there is no other efficient way to handle setting the read-undef flags, then this needs to be very clearly documented in the tracker API and where the scheduler forces it to be used for subregisters.

lib/CodeGen/MachineScheduler.cpp

335–337

How are multiple disjoint live segments avoided in general for non-dead sub register definitions? What if the sub register definitions initially overlap and scheduling makes them disjoint? It looks like we place a read-undef flag on subsequent definitions creating multiple "connected components" in the live range.

[MatzeB]

In D14969#298725, @atrick wrote:

How does this avoid creating multiple disjoint live subranges for a single virtual register? Will verifyLiveInterval choke on this?

RegisterPressureTracker now mutates the machine instructions which is conceptually wrong. If there is no other efficient way to handle setting the read-undef flags, then this needs to be very clearly documented in the tracker API and where the scheduler forces it to be used for subregisters.

Yes the current code feels very wrong in this respect. To properly fix this I have to go into RegisterPressure and expose the RegisterOperands class publicly so the RegisterOperands can be built in the MachineScheduler and then be passed on to RegisterPressure and used to set the flags.
As that is a bigger refactoring I tried to avoid it in the context of this patch, but I can understand that you do not want these layering violations in the code, I'll look into the refactoring and propose separate patches for that.

lib/CodeGen/MachineScheduler.cpp

335–337

You are right, this cannot only happen for dead-subreg defs but may also happen for degenerated cases where we have completely independent def+use pairs for subregisters. I think except for completely dead subregister defs this won't happen in typical programs since the whole point of using subregisters is to have some defs or uses where they are joined.

But we certainly need MachineVerifier checking for this property. I'll work on that.

[MatzeB]

New versions which builds RegisterOperands in MachineScheduler instead of RegisterPressure. This has the effect that the machine instructions are modified outside of the RegisterPressureTracker now.

[atrick]

LGTM as long as you can verify that subreg lifetimes are not disjoint.

include/llvm/CodeGen/MachineScheduler.h

 class ScheduleDAGMI;
 
 /// Define a generic scheduling policy for targets that don't provide their own
 /// MachineSchedStrategy. This can be overriden for each scheduling region
 /// before building the DAG.
 struct MachineSchedPolicy {
   // Allow the scheduler to disable register pressure tracking.
   bool ShouldTrackPressure;
+  /// Track LaneMasks to allow reordering of independent subregister writes
+  /// of the same vreg. \sa MachineSchedStrategy::shouldTrackLaneMasks()
+  bool ShouldTrackLaneMasks;
 
   // Allow the scheduler to force top-down or bottom-up scheduling. If neither
   // is true, the scheduler runs in both directions and converges.
   bool OnlyTopDown;
   bool OnlyBottomUp;
 
   // Disable heuristic that tries to fetch nodes from long dependency chains
   // first.
   bool DisableLatencyHeuristic;
 
-  MachineSchedPolicy(): ShouldTrackPressure(false), OnlyTopDown(false),
-    OnlyBottomUp(false), DisableLatencyHeuristic(false) {}
+  MachineSchedPolicy(): ShouldTrackPressure(false), ShouldTrackLaneMasks(false),
+    OnlyTopDown(false), OnlyBottomUp(false), DisableLatencyHeuristic(false) {}
 };
 
 /// MachineSchedStrategy - Interface to the scheduling algorithm used by
 /// ScheduleDAGMI.
 ///
 /// Initialization sequence:
 ///   initPolicy -> shouldTrackPressure -> initialize(DAG) -> registerRoots
 class MachineSchedStrategy {
   virtual void anchor();
 public:
   virtual ~MachineSchedStrategy() {}
 
   /// Optionally override the per-region scheduling policy.
   virtual void initPolicy(MachineBasicBlock::iterator Begin,
                           MachineBasicBlock::iterator End,
                           unsigned NumRegionInstrs) {}
 
   virtual void dumpPolicy() {}
 
   /// Check if pressure tracking is needed before building the DAG and
   /// initializing this strategy. Called after initPolicy.
   virtual bool shouldTrackPressure() const { return true; }
 
+  /// Returns true if lanemasks should be tracked. LaneMask tracking is
+  /// necessary to reorder independent subregister defs for the same vreg.
+  /// This has to be enabled in combination with shouldTrackPressure().
+  virtual bool shouldTrackLaneMasks() const { return false; }
+
   /// Initialize the strategy after building the DAG for a new region.
   virtual void initialize(ScheduleDAGMI *DAG) = 0;
 
   /// Notify this strategy that all roots have been released (including those
   /// that depend on EntrySU or ExitSU).
   virtual void registerRoots() {}
 
   /// Pick the next node to schedule, or return NULL. Set IsTopNode to true to
protected:
 
   // Map each SU to its summary of pressure changes. This array is updated for
   // liveness during bottom-up scheduling. Top-down scheduling may proceed but
   // has no affect on the pressure diffs.
   PressureDiffs SUPressureDiffs;
 
   /// Register pressure in this region computed by initRegPressure.
   bool ShouldTrackPressure;
+  bool ShouldTrackLaneMasks;
   IntervalPressure RegPressure;
   RegPressureTracker RPTracker;
 
   /// List of pressure sets that exceed the target's pressure limit before
   /// scheduling, listed in increasing set ID order. Each pressure set is paired
   /// with its max pressure in the currently scheduled regions.
   std::vector<PressureChange> RegionCriticalPSets;
 
   /// The top of the unscheduled zone.
   IntervalPressure TopPressure;
   RegPressureTracker TopRPTracker;
 
   /// The bottom of the unscheduled zone.
   IntervalPressure BotPressure;
   RegPressureTracker BotRPTracker;
 
 public:
   ScheduleDAGMILive(MachineSchedContext *C,
                     std::unique_ptr<MachineSchedStrategy> S)
       : ScheduleDAGMI(C, std::move(S), /*RemoveKillFlags=*/false),
         RegClassInfo(C->RegClassInfo), DFSResult(nullptr),
-        ShouldTrackPressure(false), RPTracker(RegPressure),
-        TopRPTracker(TopPressure), BotRPTracker(BotPressure) {}
+        ShouldTrackPressure(false), ShouldTrackLaneMasks(false),
+        RPTracker(RegPressure), TopRPTracker(TopPressure),
+        BotRPTracker(BotPressure) {}
 
   ~ScheduleDAGMILive() override;
 
   /// Return true if this DAG supports VReg liveness and RegPressure.
   bool hasVRegLiveness() const override { return true; }
 
   /// \brief Return true if register pressure tracking is enabled.
   bool isTrackingPressure() const { return ShouldTrackPressure; }
void initPolicy(MachineBasicBlock::iterator Begin,
                   unsigned NumRegionInstrs) override;
 
   void dumpPolicy() override;
 
   bool shouldTrackPressure() const override {
     return RegionPolicy.ShouldTrackPressure;
   }
 
+  bool shouldTrackLaneMasks() const override {
+    return RegionPolicy.ShouldTrackLaneMasks;
+  }
+
   void initialize(ScheduleDAGMI *dag) override;
 
   SUnit *pickNode(bool &IsTopNode) override;
 
   void schedNode(SUnit *SU, bool IsTopNode) override;
 
   void releaseTopNode(SUnit *SU) override {
     Top.releaseTopNode(SU);

include/llvm/CodeGen/RegisterPressure.h

public:
   void setPos(MachineBasicBlock::const_iterator Pos) { CurrPos = Pos; }
 
   /// \brief Get the SlotIndex for the first nondebug instruction including or
   /// after the current position.
   SlotIndex getCurrSlot() const;
 
   /// Recede across the previous instruction.
   bool recede(SmallVectorImpl<RegisterMaskPair> *LiveUses = nullptr,
-              PressureDiff *PDiff = nullptr);
+              PressureDiff *PDiff = nullptr, bool AddReadUndef = false);
 
   /// Advance across the current instruction.
-  bool advance();
+  bool advance(bool AddReadUndef = false);
 
   /// Finalize the region boundaries and recored live ins and live outs.
   void closeRegion();
 
   /// Initialize the LiveThru pressure set based on the untied defs found in
   /// RPTracker.
   void initLiveThru(const RegPressureTracker &RPTracker);
 

 protected:
   const LiveRange *getLiveRange(unsigned Reg) const;
 
   void increaseRegPressure(unsigned RegUnit, LaneBitmask PreviousMask,
                            LaneBitmask NewMask);
   void decreaseRegPressure(unsigned RegUnit, LaneBitmask PreviousMask,
                            LaneBitmask NewMask);
 
-  void bumpDeadDefs(ArrayRef<RegisterMaskPair> DeadDefs);
+  void bumpDeadDefs(ArrayRef<RegisterMaskPair> DeadDefs,
+                    MachineInstr *AddReadUndefTo);
 
   void bumpUpwardPressure(const MachineInstr *MI);
   void bumpDownwardPressure(const MachineInstr *MI);
 
   void discoverLiveInOrOut(RegisterMaskPair Pair,
                            SmallVectorImpl<RegisterMaskPair> &LiveInOrOut);
 
   LaneBitmask getLastUsedLanes(unsigned RegUnit, SlotIndex Pos) const;

lib/CodeGen/MachineScheduler.cpp

bool MachineScheduler::runOnMachineFunction(MachineFunction &mf) {
   MF = &mf;
   MLI = &getAnalysis<MachineLoopInfo>();
   MDT = &getAnalysis<MachineDominatorTree>();
   PassConfig = &getAnalysis<TargetPassConfig>();
   AA = &getAnalysis<AAResultsWrapperPass>().getAAResults();
 
   LIS = &getAnalysis<LiveIntervals>();
 
+  if (mf.getRegInfo().subRegLivenessEnabled()) {
+    // Dead subregister defs have no users and therefore no dependencies,
+    // moving them around may cause liveintervals to degrade into multiple
+    // component. Change the dead subregister defs to have their own vreg.
+    LIS->renameDeadSubRegDefs();
+  }
+
   if (VerifyScheduling) {
     DEBUG(LIS->dump());
     MF->verify(this, "Before machine scheduling.");
   }
   RegClassInfo->runOnMachineFunction(*MF);
 
   // Instantiate the selected scheduler for this target, function, and
   // optimization level.
void ScheduleDAGMILive::enterRegion(MachineBasicBlock *bb,
   ScheduleDAGMI::enterRegion(bb, begin, end, regioninstrs);
 
   // For convenience remember the end of the liveness region.
   LiveRegionEnd = (RegionEnd == bb->end()) ? RegionEnd : std::next(RegionEnd);
 
   SUPressureDiffs.clear();
 
   ShouldTrackPressure = SchedImpl->shouldTrackPressure();
+  ShouldTrackLaneMasks = SchedImpl->shouldTrackLaneMasks();
+  if (ShouldTrackLaneMasks && !ShouldTrackPressure)
+    report_fatal_error("ShouldTrackLaneMasks requires ShouldTrackPressure");
 }
 
 // Setup the register pressure trackers for the top scheduled top and bottom
 // scheduled regions.
 void ScheduleDAGMILive::initRegPressure() {
-  TopRPTracker.init(&MF, RegClassInfo, LIS, BB, RegionBegin, false, false);
-  BotRPTracker.init(&MF, RegClassInfo, LIS, BB, LiveRegionEnd, false, false);
+  TopRPTracker.init(&MF, RegClassInfo, LIS, BB, RegionBegin,
+                    ShouldTrackLaneMasks, false);
+  BotRPTracker.init(&MF, RegClassInfo, LIS, BB, LiveRegionEnd,
+                    ShouldTrackLaneMasks, false);
 
   // Close the RPTracker to finalize live ins.
   RPTracker.closeRegion();
 
   DEBUG(RPTracker.dump());
 
   // Initialize the live ins and live outs.
   TopRPTracker.addLiveRegs(RPTracker.getPressure().LiveInRegs);
updateScheduledPressure(const SUnit *SU,
   }
 }
 
 /// Update the PressureDiff array for liveness after scheduling this
 /// instruction.
 void ScheduleDAGMILive::updatePressureDiffs(
     ArrayRef<RegisterMaskPair> LiveUses) {
   for (const RegisterMaskPair &P : LiveUses) {
-    /// FIXME: Currently assuming single-use physregs.
     unsigned Reg = P.RegUnit;
-    assert(P.LaneMask != 0);
-    DEBUG(dbgs() << "  LiveReg: " << PrintVRegOrUnit(Reg, TRI) << "\n");
+    /// FIXME: Currently assuming single-use physregs.
     if (!TRI->isVirtualRegister(Reg))
       continue;
 
+    if (ShouldTrackLaneMasks) {
+      // If the register has just become live then other uses won't change
+      // this fact anymore => decrement pressure.
+      // If the register has just become dead then other uses make it come
+      // back to life => increment pressure.
+      bool Decrement = P.LaneMask != 0;
+
+      for (const VReg2SUnit &V2SU
+           : make_range(VRegUses.find(Reg), VRegUses.end())) {
+        SUnit &SU = *V2SU.SU;
+        if (SU.isScheduled || &SU == &ExitSU)
+          continue;
+
+        PressureDiff &PDiff = getPressureDiff(&SU);
+        PDiff.addPressureChange(Reg, Decrement, &MRI);
+        DEBUG(
+          dbgs() << "  UpdateRegP: SU(" << SU.NodeNum << ") "
+                 << PrintReg(Reg, TRI) << ':' << PrintLaneMask(P.LaneMask)
+                 << ' ' << *SU.getInstr();
+          dbgs() << "              to ";
+          PDiff.dump(*TRI);
+        );
+      }
+    } else {
+      assert(P.LaneMask != 0);
+      DEBUG(dbgs() << "  LiveReg: " << PrintVRegOrUnit(Reg, TRI) << "\n");
       // This may be called before CurrentBottom has been initialized. However,
       // BotRPTracker must have a valid position. We want the value live into the
       // instruction or live out of the block, so ask for the previous
       // instruction's live-out.
       const LiveInterval &LI = LIS->getInterval(Reg);
       VNInfo *VNI;
       MachineBasicBlock::const_iterator I =
         nextIfDebug(BotRPTracker.getPos(), BB->end());
       if (I == BB->end())
         VNI = LI.getVNInfoBefore(LIS->getMBBEndIdx(BB));
       else {
         LiveQueryResult LRQ = LI.Query(LIS->getInstructionIndex(I));
         VNI = LRQ.valueIn();
       }
       // RegisterPressureTracker guarantees that readsReg is true for LiveUses.
       assert(VNI && "No live value at use.");
       for (const VReg2SUnit &V2SU
            : make_range(VRegUses.find(Reg), VRegUses.end())) {
         SUnit *SU = V2SU.SU;
-      // If this use comes before the reaching def, it cannot be a last use, so
-      // descrease its pressure change.
+        // If this use comes before the reaching def, it cannot be a last use,
+        // so descrease its pressure change.
         if (!SU->isScheduled && SU != &ExitSU) {
           LiveQueryResult LRQ
             = LI.Query(LIS->getInstructionIndex(SU->getInstr()));
           if (LRQ.valueIn() == VNI) {
             PressureDiff &PDiff = getPressureDiff(SU);
             PDiff.addPressureChange(Reg, true, &MRI);
             DEBUG(
               dbgs() << "  UpdateRegP: SU(" << SU->NodeNum << ") "
                      << *SU->getInstr();
               dbgs() << "              to ";
               PDiff.dump(*TRI);
             );
           }
         }
       }
     }
   }
+}
 
 /// schedule - Called back from MachineScheduler::runOnMachineFunction
 /// after setting up the current scheduling region. [RegionBegin, RegionEnd)
 /// only includes instructions that have DAG nodes, not scheduling boundaries.
 ///
 /// This is a skeletal driver, with all the functionality pushed into helpers,
 /// so that it can be easily extended by experimental schedulers. Generally,
 /// implementing MachineSchedStrategy should be sufficient to implement a new
if (!ShouldTrackPressure) {
     RPTracker.reset();
     RegionCriticalPSets.clear();
     buildSchedGraph(AA);
     return;
   }
 
   // Initialize the register pressure tracker used by buildSchedGraph.
   RPTracker.init(&MF, RegClassInfo, LIS, BB, LiveRegionEnd,
-                 false, /*TrackUntiedDefs=*/true);
+                 ShouldTrackLaneMasks, /*TrackUntiedDefs=*/true);
 
   // Account for liveness generate by the region boundary.
   if (LiveRegionEnd != RegionEnd)
     RPTracker.recede();
 
   // Build the DAG, and compute current register pressure.
-  buildSchedGraph(AA, &RPTracker, &SUPressureDiffs);
+  buildSchedGraph(AA, &RPTracker, &SUPressureDiffs, ShouldTrackLaneMasks);
 
   // Initialize top/bottom trackers after computing region pressure.
   initRegPressure();
 }
 
 void ScheduleDAGMILive::computeDFSResult() {
   if (!DFSResult)
     DFSResult = new SchedDFSResult(/*BottomU*/true, MinSubtreeSize);
if (&*CurrentTop == MI)
       CurrentTop = nextIfDebug(++CurrentTop, CurrentBottom);
     else {
       moveInstruction(MI, CurrentTop);
       TopRPTracker.setPos(MI);
     }
 
     if (ShouldTrackPressure) {
       // Update top scheduled pressure.
-      TopRPTracker.advance();
+      bool AddReadUndef = ShouldTrackLaneMasks;
+      TopRPTracker.advance(AddReadUndef);
       assert(TopRPTracker.getPos() == CurrentTop && "out of sync");
       DEBUG(
         dbgs() << "Top Pressure:\n";
         dumpRegSetPressure(TopRPTracker.getRegSetPressureAtPos(), TRI);
       );
 
       updateScheduledPressure(SU, TopRPTracker.getPressure().MaxSetPressure);
     }
else {
         TopRPTracker.setPos(CurrentTop);
       }
       moveInstruction(MI, CurrentBottom);
       CurrentBottom = MI;
     }
     if (ShouldTrackPressure) {
       // Update bottom scheduled pressure.
       SmallVector<RegisterMaskPair, 8> LiveUses;
-      BotRPTracker.recede(&LiveUses);
+      bool AddReadUndef = ShouldTrackLaneMasks;
+      BotRPTracker.recede(&LiveUses, nullptr, AddReadUndef);
       assert(BotRPTracker.getPos() == CurrentBottom && "out of sync");
       DEBUG(
         dbgs() << "Bottom Pressure:\n";
         dumpRegSetPressure(BotRPTracker.getRegSetPressureAtPos(), TRI);
       );
 
       updateScheduledPressure(SU, BotRPTracker.getPressure().MaxSetPressure);
       updatePressureDiffs(LiveUses);

lib/CodeGen/RegisterPressure.cpp

auto I = std::find_if(RegUnits.begin(), RegUnits.end(),
                         });
   if (I == RegUnits.end()) {
     RegUnits.push_back(Pair);
   } else {
     I->LaneMask |= Pair.LaneMask;
   }
 }
 
+static void setRegZero(SmallVectorImpl<RegisterMaskPair> &RegUnits,
+                       unsigned RegUnit) {
+  auto I = std::find_if(RegUnits.begin(), RegUnits.end(),
+                        [RegUnit](const RegisterMaskPair Other) {
+                          return Other.RegUnit == RegUnit;
+                        });
+  if (I == RegUnits.end()) {
+    RegUnits.push_back(RegisterMaskPair(RegUnit, 0));
+  } else {
+    I->LaneMask = 0;
+  }
+}
+
 static void removeRegLanes(SmallVectorImpl<RegisterMaskPair> &RegUnits,
                            RegisterMaskPair Pair) {
   unsigned RegUnit = Pair.RegUnit;
   assert(Pair.LaneMask != 0);
   auto I = std::find_if(RegUnits.begin(), RegUnits.end(),
                         [RegUnit](const RegisterMaskPair Other) {
                           return Other.RegUnit == RegUnit;
                         });
static void collectOperands(const MachineInstr *MI,
   // Remove redundant physreg dead defs.
   for (const RegisterMaskPair &P : RegOpers.Defs)
     removeRegLanes(RegOpers.DeadDefs, P);
 }
 
 static void adjustOperandsForLiveness(const LiveIntervals &LIS,
                                       const MachineRegisterInfo &MRI,
                                       RegisterOperands &RegOpers,
-                                      SlotIndex Pos) {
+                                      SlotIndex Pos, MachineInstr *AddFlagsMI) {
   for (auto I = RegOpers.Defs.begin(); I != RegOpers.Defs.end(); ) {
     LaneBitmask LiveAfter = getLiveLanesAt(LIS, MRI, I->RegUnit,
                                            Pos.getDeadSlot());
+    // If the the def is all that is live after the instruction, then in case
+    // of a subregister def we need a read-undef flag.
+    if (AddFlagsMI != nullptr && (LiveAfter & ~I->LaneMask) == 0)
+      AddFlagsMI->setRegisterDefReadUndef(I->RegUnit);
     unsigned LaneMask = I->LaneMask & LiveAfter;
-    if (LaneMask == 0)
+    if (LaneMask == 0) {
       I = RegOpers.Defs.erase(I);
-    else {
+      // Make sure the operand is properly marked as Dead.
+      if (AddFlagsMI != nullptr)
+        AddFlagsMI->addRegisterDead(I->RegUnit, MRI.getTargetRegisterInfo());
+    } else {
       I->LaneMask = LaneMask;
       ++I;
     }
   }
   for (auto I = RegOpers.Uses.begin(); I != RegOpers.Uses.end(); ) {
     LaneBitmask LiveBefore = getLiveLanesAt(LIS, MRI, I->RegUnit,
                                             Pos.getBaseIndex());
     unsigned LaneMask = I->LaneMask & LiveBefore;
if (NewUnitInc != 0) {
       if (J != E)
         *I = *J;
     }
   }
 }
 
 /// Record the pressure difference induced by the given operand list.
 static void collectPDiff(PressureDiff &PDiff, RegisterOperands &RegOpers,
-                         const MachineRegisterInfo *MRI) {
+                         const MachineRegisterInfo *MRI,
+                         bool TrackLaneMasks) {
   assert(!PDiff.begin()->isValid() && "stale PDiff");
 
   for (const RegisterMaskPair &P : RegOpers.Defs)
     PDiff.addPressureChange(P.RegUnit, true, MRI);
 
   for (const RegisterMaskPair &P : RegOpers.Uses)
     PDiff.addPressureChange(P.RegUnit, false, MRI);
 }

 void RegPressureTracker::discoverLiveIn(RegisterMaskPair Pair) {
   discoverLiveInOrOut(Pair, P.LiveInRegs);
 }
 
 void RegPressureTracker::discoverLiveOut(RegisterMaskPair Pair) {
   discoverLiveInOrOut(Pair, P.LiveOutRegs);
 }
 
-void RegPressureTracker::bumpDeadDefs(ArrayRef<RegisterMaskPair> DeadDefs) {
+void RegPressureTracker::bumpDeadDefs(ArrayRef<RegisterMaskPair> DeadDefs,
+                                      MachineInstr *AddReadUndefTo) {
   for (const RegisterMaskPair &P : DeadDefs) {
     unsigned Reg = P.RegUnit;
     LaneBitmask LiveMask = LiveRegs.contains(Reg);
     LaneBitmask BumpedMask = LiveMask | P.LaneMask;
     increaseRegPressure(Reg, LiveMask, BumpedMask);
+    if (AddReadUndefTo != nullptr && LiveMask == 0)
+      AddReadUndefTo->setRegisterDefReadUndef(Reg, true);
   }
   for (const RegisterMaskPair &P : DeadDefs) {
     unsigned Reg = P.RegUnit;
     LaneBitmask LiveMask = LiveRegs.contains(Reg);
     LaneBitmask BumpedMask = LiveMask | P.LaneMask;
     decreaseRegPressure(Reg, BumpedMask, LiveMask);
   }
 }
 
 /// Recede across the previous instruction. If LiveUses is provided, record any
 /// RegUnits that are made live by the current instruction's uses. This includes
 /// registers that are both defined and used by the instruction.  If a pressure
 /// difference pointer is provided record the changes is pressure caused by this
 /// instruction independent of liveness.
 bool RegPressureTracker::recede(SmallVectorImpl<RegisterMaskPair> *LiveUses,
-                                PressureDiff *PDiff) {
+                                PressureDiff *PDiff, bool AddReadUndef) {
   // Check for the top of the analyzable region.
   if (CurrPos == MBB->begin()) {
     closeRegion();
     return false;
   }
   if (!isBottomClosed())
     closeBottom();
 
bool RegPressureTracker::recede(SmallVectorImpl<RegisterMaskPair> *LiveUses,
   SlotIndex SlotIdx;
   if (RequireIntervals)
     SlotIdx = LIS->getInstructionIndex(CurrPos).getRegSlot();
 
   // Open the top of the region using slot indexes.
   if (RequireIntervals && isTopClosed())
     static_cast<IntervalPressure&>(P).openTop(SlotIdx);
 
+  const MachineInstr &MI = *CurrPos;
   RegisterOperands RegOpers(TRI, MRI, TrackLaneMasks, false);
-  collectOperands(CurrPos, RegOpers);
+  collectOperands(&MI, RegOpers);
   if (TrackLaneMasks)
-    adjustOperandsForLiveness(*LIS, *MRI, RegOpers, SlotIdx);
+    adjustOperandsForLiveness(*LIS, *MRI, RegOpers, SlotIdx,
+                              AddReadUndef ? const_cast<MachineInstr*>(&MI)
+                                           : nullptr);
 
   if (PDiff)
-    collectPDiff(*PDiff, RegOpers, MRI);
+    collectPDiff(*PDiff, RegOpers, MRI, TrackLaneMasks);
 
   // Boost pressure for all dead defs together.
-  bumpDeadDefs(RegOpers.DeadDefs);
+  bumpDeadDefs(RegOpers.DeadDefs,
+               AddReadUndef ? const_cast<MachineInstr*>(&MI) : nullptr);
 
   // Kill liveness at live defs.
   // TODO: consider earlyclobbers?
   for (const RegisterMaskPair &Def : RegOpers.Defs) {
     unsigned Reg = Def.RegUnit;
 
     LaneBitmask PreviousMask = LiveRegs.erase(Def);
     LaneBitmask NewMask = PreviousMask & ~Def.LaneMask;
if (PreviousMask == 0) {
           // not flagged as such. Since this register will not be recorded as
           // live-out, increase its PDiff value to avoid underflowing pressure.
           if (PDiff)
             PDiff->addPressureChange(Reg, false, MRI);
         }
       }
     }
 
+    if (NewMask == 0) {
+      if (AddReadUndef && PreviousMask != ~0u)
+        (const_cast<MachineInstr&>(MI)).setRegisterDefReadUndef(Def.RegUnit, true);
+      // Add a 0 entry to LiveUses as a marker that the complete vreg has become
+      // dead.
+      if (TrackLaneMasks && LiveUses != nullptr)
+        setRegZero(*LiveUses, Reg);
+    }
+
     decreaseRegPressure(Reg, PreviousMask, NewMask);
   }
 
   // Generate liveness for uses.
   for (const RegisterMaskPair &Use : RegOpers.Uses) {
     unsigned Reg = Use.RegUnit;
     assert(Use.LaneMask != 0);
     LaneBitmask PreviousMask = LiveRegs.insert(Use);
     LaneBitmask NewMask = PreviousMask | Use.LaneMask;
     if (NewMask == PreviousMask)
       continue;
 
     // Did the register just become live?
     if (PreviousMask == 0) {
       if (LiveUses != nullptr) {
-        unsigned NewLanes = NewMask & ~PreviousMask;
-        addRegLanes(*LiveUses, RegisterMaskPair(Reg, NewLanes));
+        if (!TrackLaneMasks) {
+          addRegLanes(*LiveUses, RegisterMaskPair(Reg, NewMask));
+        } else {
+          auto I = std::find_if(LiveUses->begin(), LiveUses->end(),
+                                [Reg](const RegisterMaskPair Other) {
+                                return Other.RegUnit == Reg;
+                                });
+          bool IsRedef = I != LiveUses->end();
+          if (IsRedef) {
+            // ignore re-defs here...
+            assert(I->LaneMask == 0);
+            removeRegLanes(*LiveUses, RegisterMaskPair(Reg, NewMask));
+          } else {
+            addRegLanes(*LiveUses, RegisterMaskPair(Reg, NewMask));
+          }
+        }
       }
 
       // Discover live outs if this may be the first occurance of this register.
       LaneBitmask LiveOut = getLiveThroughAt(Reg, SlotIdx);
       discoverLiveOut(RegisterMaskPair(Reg, LiveOut));
     }
 
     increaseRegPressure(Reg, PreviousMask, NewMask);
   }
   if (TrackUntiedDefs) {
     for (const RegisterMaskPair &Def : RegOpers.Defs) {
       unsigned RegUnit = Def.RegUnit;
       if (TargetRegisterInfo::isVirtualRegister(RegUnit) &&
           (LiveRegs.contains(RegUnit) & Def.LaneMask) == 0)
         UntiedDefs.insert(RegUnit);
     }
   }
   return true;
 }
 
 /// Advance across the current instruction.
-bool RegPressureTracker::advance() {
+bool RegPressureTracker::advance(bool AddReadUndef) {
   assert(!TrackUntiedDefs && "unsupported mode");
 
   // Check for the bottom of the analyzable region.
   if (CurrPos == MBB->end()) {
     closeRegion();
     return false;
   }
   if (!isTopClosed())
     closeTop();
 
   SlotIndex SlotIdx;
   if (RequireIntervals)
     SlotIdx = getCurrSlot();
 
   // Open the bottom of the region using slot indexes.
   if (isBottomClosed()) {
     if (RequireIntervals)
       static_cast<IntervalPressure&>(P).openBottom(SlotIdx);
     else
       static_cast<RegionPressure&>(P).openBottom(CurrPos);
   }
 
+  const MachineInstr &MI = *CurrPos;
   RegisterOperands RegOpers(TRI, MRI, TrackLaneMasks, false);
-  collectOperands(CurrPos, RegOpers);
+  collectOperands(&MI, RegOpers);
   if (TrackLaneMasks)
-    adjustOperandsForLiveness(*LIS, *MRI, RegOpers, SlotIdx);
+    adjustOperandsForLiveness(*LIS, *MRI, RegOpers, SlotIdx,
+                              AddReadUndef ? const_cast<MachineInstr*>(&MI)
+                                           : nullptr);
 
   for (const RegisterMaskPair &Use : RegOpers.Uses) {
     unsigned Reg = Use.RegUnit;
     LaneBitmask LiveMask = LiveRegs.contains(Reg);
     LaneBitmask LiveIn = Use.LaneMask & ~LiveMask;
     if (LiveIn != 0) {
       discoverLiveIn(RegisterMaskPair(Reg, LiveIn));
       increaseRegPressure(Reg, LiveMask, LiveMask | LiveIn);
       LiveRegs.insert(RegisterMaskPair(Reg, LiveIn));
     }
     // Kill liveness at last uses.
     LaneBitmask LastUseMask = getLastUsedLanes(Reg, SlotIdx);
     if (LastUseMask != 0) {
       LiveRegs.erase(RegisterMaskPair(Reg, LastUseMask));
       decreaseRegPressure(Reg, LiveMask, LiveMask & ~LastUseMask);
     }
   }
 
   // Generate liveness for defs.
   for (const RegisterMaskPair &Def : RegOpers.Defs) {
     LaneBitmask PreviousMask = LiveRegs.insert(Def);
     LaneBitmask NewMask = PreviousMask | Def.LaneMask;
     increaseRegPressure(Def.RegUnit, PreviousMask, NewMask);
+    if (AddReadUndef && PreviousMask == 0 && NewMask != ~0u) {
+      (const_cast<MachineInstr&>(MI)).setRegisterDefReadUndef(Def.RegUnit, true);
+    }
   }
 
   // Boost pressure for all dead defs together.
-  bumpDeadDefs(RegOpers.DeadDefs);
+  bumpDeadDefs(RegOpers.DeadDefs,
+               AddReadUndef ? const_cast<MachineInstr*>(&MI) : nullptr);
 
   // Find the next instruction.
   do
     ++CurrPos;
   while (CurrPos != MBB->end() && CurrPos->isDebugValue());
   return true;
 }
 
void RegPressureTracker::bumpUpwardPressure(const MachineInstr *MI) {
   SlotIndex SlotIdx;
   if (RequireIntervals)
     SlotIdx = LIS->getInstructionIndex(MI).getRegSlot();
 
   // Account for register pressure similar to RegPressureTracker::recede().
   RegisterOperands RegOpers(TRI, MRI, TrackLaneMasks, /*IgnoreDead=*/true);
   collectOperands(MI, RegOpers);
   if (TrackLaneMasks)
-    adjustOperandsForLiveness(*LIS, *MRI, RegOpers, SlotIdx);
+    adjustOperandsForLiveness(*LIS, *MRI, RegOpers, SlotIdx, nullptr);
 
   // Boost max pressure for all dead defs together.
   // Since CurrSetPressure and MaxSetPressure
-  bumpDeadDefs(RegOpers.DeadDefs);
+  bumpDeadDefs(RegOpers.DeadDefs, nullptr);
 
   // Kill liveness at live defs.
   for (const RegisterMaskPair &P : RegOpers.Defs) {
     unsigned Reg = P.RegUnit;
     unsigned LiveLanes = LiveRegs.contains(Reg);
     unsigned UseLanes = getRegLanes(RegOpers.Uses, Reg);
     unsigned DefLanes = P.LaneMask;
     unsigned LiveAfter = (LiveLanes & ~DefLanes) | UseLanes;
void RegPressureTracker::bumpDownwardPressure(const MachineInstr *MI) {
   SlotIndex MISlotIdx;
   if (RequireIntervals)
     MISlotIdx = LIS->getInstructionIndex(MI).getRegSlot();
 
   // Account for register pressure similar to RegPressureTracker::recede().
   RegisterOperands RegOpers(TRI, MRI, TrackLaneMasks, false);
   collectOperands(MI, RegOpers);
   if (TrackLaneMasks)
-    adjustOperandsForLiveness(*LIS, *MRI, RegOpers, MISlotIdx);
+    adjustOperandsForLiveness(*LIS, *MRI, RegOpers, MISlotIdx, nullptr);
 
   SlotIndex SlotIdx;
   if (RequireIntervals)
     SlotIdx = LIS->getInstructionIndex(CurrPos).getRegSlot();
 
   for (const RegisterMaskPair &Use : RegOpers.Uses) {
     unsigned Reg = Use.RegUnit;
     LaneBitmask LastUseMask = getLastUsedLanes(Reg, SlotIdx);
void RegPressureTracker::bumpDownwardPressure(const MachineInstr *MI) {
   for (const RegisterMaskPair &Def : RegOpers.Defs) {
     unsigned Reg = Def.RegUnit;
     LaneBitmask LiveMask = LiveRegs.contains(Reg);
     LaneBitmask NewMask = LiveMask | Def.LaneMask;
     increaseRegPressure(Reg, LiveMask, NewMask);
   }
 
   // Boost pressure for all dead defs together.
-  bumpDeadDefs(RegOpers.DeadDefs);
+  bumpDeadDefs(RegOpers.DeadDefs, nullptr);
 }
 
 /// Consider the pressure increase caused by traversing this instruction
 /// top-down. Find the register class with the most change in its pressure limit
 /// based on the tracker's current pressure, and return the number of excess
 /// register units of that pressure set introduced by this instruction.
 ///
 /// This assumes that the current LiveIn set is sufficient.

lib/CodeGen/ScheduleDAGInstrs.cpp

if (!MO.isReg())
       continue;
     if (!MO.isUse() && (MO.getSubReg() == 0 || !TrackLaneMasks))
       continue;
 
     unsigned Reg = MO.getReg();
     if (!TargetRegisterInfo::isVirtualRegister(Reg))
       continue;
 
+    // Ignore re-defs.
+    if (TrackLaneMasks) {
+      bool FoundDef = false;
+      for (const MachineOperand &MO2 : MI->operands()) {
+        if (MO2.isReg() && MO2.isDef() && MO2.getReg() == Reg && !MO2.isDead()) {
+          FoundDef = true;
+          break;
+        }
+      }
+      if (FoundDef)
+        continue;
+    }
+
     // Record this local VReg use.
     VReg2SUnitMultiMap::iterator UI = VRegUses.find(Reg);
     for (; UI != VRegUses.end(); ++UI) {
       if (UI->SU == SU)
         break;
     }
     if (UI == VRegUses.end())
       VRegUses.insert(VReg2SUnit(Reg, 0, SU));