MachineScheduler: Allow independent scheduling of sub register defs

[MatzeB]

Summary

If the TrackLaneMasks policy/strategy is enabled the MachineScheduler
will build a schedule graph where definitions of independent
subregisters are no longer serialised.

Implementation comments:

• Without lane mask tracking a sub register def also counts as a use (except for the first one with the read-undef flag set), with lane mask tracking enabled this is no longer the case.
• Pressure Diffs where previously maintained per definition of a vreg with the help of the SSA information contained in the LiveIntervals. With lanemask tracking enabled we cannot do this anymore and instead change the pressure diffs for all uses of the vreg as it becomes live/dead. For this changed style to work correctly we ignore uses of instructions that define the same register again: They won't affect register pressure.
• With lanemask tracking we remove all read-undef flags from sub register defs when building the graph and re-add them later when all vreg lanes have become dead.

[atrick]

How does this avoid creating multiple disjoint live subranges for a single virtual register? Will verifyLiveInterval choke on this?

RegisterPressureTracker now mutates the machine instructions which is conceptually wrong. If there is no other efficient way to handle setting the read-undef flags, then this needs to be very clearly documented in the tracker API and where the scheduler forces it to be used for subregisters.

lib/CodeGen/MachineScheduler.cpp

335–337 ↗

(On Diff #41086)

How are multiple disjoint live segments avoided in general for non-dead sub register definitions? What if the sub register definitions initially overlap and scheduling makes them disjoint? It looks like we place a read-undef flag on subsequent definitions creating multiple "connected components" in the live range.

[MatzeB]

In D14969#298725, @atrick wrote:

How does this avoid creating multiple disjoint live subranges for a single virtual register? Will verifyLiveInterval choke on this?

RegisterPressureTracker now mutates the machine instructions which is conceptually wrong. If there is no other efficient way to handle setting the read-undef flags, then this needs to be very clearly documented in the tracker API and where the scheduler forces it to be used for subregisters.

Yes the current code feels very wrong in this respect. To properly fix this I have to go into RegisterPressure and expose the RegisterOperands class publicly so the RegisterOperands can be built in the MachineScheduler and then be passed on to RegisterPressure and used to set the flags.
As that is a bigger refactoring I tried to avoid it in the context of this patch, but I can understand that you do not want these layering violations in the code, I'll look into the refactoring and propose separate patches for that.

lib/CodeGen/MachineScheduler.cpp

335–337 ↗

(On Diff #41086)

You are right, this cannot only happen for dead-subreg defs but may also happen for degenerated cases where we have completely independent def+use pairs for subregisters. I think except for completely dead subregister defs this won't happen in typical programs since the whole point of using subregisters is to have some defs or uses where they are joined.

But we certainly need MachineVerifier checking for this property. I'll work on that.

[MatzeB]

New versions which builds RegisterOperands in MachineScheduler instead of RegisterPressure. This has the effect that the machine instructions are modified outside of the RegisterPressureTracker now.

[atrick]

LGTM as long as you can verify that subreg lifetimes are not disjoint.

llvm/trunk/include/llvm/CodeGen/MachineScheduler.h

 class ScheduleDAGMI;
 
 /// Define a generic scheduling policy for targets that don't provide their own
 /// MachineSchedStrategy. This can be overriden for each scheduling region
 /// before building the DAG.
 struct MachineSchedPolicy {
   // Allow the scheduler to disable register pressure tracking.
   bool ShouldTrackPressure;
+  /// Track LaneMasks to allow reordering of independent subregister writes
+  /// of the same vreg. \sa MachineSchedStrategy::shouldTrackLaneMasks()
+  bool ShouldTrackLaneMasks;
 
   // Allow the scheduler to force top-down or bottom-up scheduling. If neither
   // is true, the scheduler runs in both directions and converges.
   bool OnlyTopDown;
   bool OnlyBottomUp;
 
   // Disable heuristic that tries to fetch nodes from long dependency chains
   // first.
   bool DisableLatencyHeuristic;
 
-  MachineSchedPolicy(): ShouldTrackPressure(false), OnlyTopDown(false),
-    OnlyBottomUp(false), DisableLatencyHeuristic(false) {}
+  MachineSchedPolicy(): ShouldTrackPressure(false), ShouldTrackLaneMasks(false),
+    OnlyTopDown(false), OnlyBottomUp(false), DisableLatencyHeuristic(false) {}
 };
 
 /// MachineSchedStrategy - Interface to the scheduling algorithm used by
 /// ScheduleDAGMI.
 ///
 /// Initialization sequence:
 ///   initPolicy -> shouldTrackPressure -> initialize(DAG) -> registerRoots
 class MachineSchedStrategy {
   virtual void anchor();
 public:
   virtual ~MachineSchedStrategy() {}
 
   /// Optionally override the per-region scheduling policy.
   virtual void initPolicy(MachineBasicBlock::iterator Begin,
                           MachineBasicBlock::iterator End,
                           unsigned NumRegionInstrs) {}
 
   virtual void dumpPolicy() {}
 
   /// Check if pressure tracking is needed before building the DAG and
   /// initializing this strategy. Called after initPolicy.
   virtual bool shouldTrackPressure() const { return true; }
 
+  /// Returns true if lanemasks should be tracked. LaneMask tracking is
+  /// necessary to reorder independent subregister defs for the same vreg.
+  /// This has to be enabled in combination with shouldTrackPressure().
+  virtual bool shouldTrackLaneMasks() const { return false; }
+
   /// Initialize the strategy after building the DAG for a new region.
   virtual void initialize(ScheduleDAGMI *DAG) = 0;
 
   /// Notify this strategy that all roots have been released (including those
   /// that depend on EntrySU or ExitSU).
   virtual void registerRoots() {}
 
   /// Pick the next node to schedule, or return NULL. Set IsTopNode to true to
protected:
 
   // Map each SU to its summary of pressure changes. This array is updated for
   // liveness during bottom-up scheduling. Top-down scheduling may proceed but
   // has no affect on the pressure diffs.
   PressureDiffs SUPressureDiffs;
 
   /// Register pressure in this region computed by initRegPressure.
   bool ShouldTrackPressure;
+  bool ShouldTrackLaneMasks;
   IntervalPressure RegPressure;
   RegPressureTracker RPTracker;
 
   /// List of pressure sets that exceed the target's pressure limit before
   /// scheduling, listed in increasing set ID order. Each pressure set is paired
   /// with its max pressure in the currently scheduled regions.
   std::vector<PressureChange> RegionCriticalPSets;
 
   /// The top of the unscheduled zone.
   IntervalPressure TopPressure;
   RegPressureTracker TopRPTracker;
 
   /// The bottom of the unscheduled zone.
   IntervalPressure BotPressure;
   RegPressureTracker BotRPTracker;
 
+  /// True if disconnected subregister components are already renamed.
+  /// The renaming is only done on demand if lane masks are tracked.
+  bool DisconnectedComponentsRenamed;
+
 public:
   ScheduleDAGMILive(MachineSchedContext *C,
                     std::unique_ptr<MachineSchedStrategy> S)
       : ScheduleDAGMI(C, std::move(S), /*RemoveKillFlags=*/false),
         RegClassInfo(C->RegClassInfo), DFSResult(nullptr),
-        ShouldTrackPressure(false), RPTracker(RegPressure),
-        TopRPTracker(TopPressure), BotRPTracker(BotPressure) {}
+        ShouldTrackPressure(false), ShouldTrackLaneMasks(false),
+        RPTracker(RegPressure), TopRPTracker(TopPressure),
+        BotRPTracker(BotPressure), DisconnectedComponentsRenamed(false) {}
 
   ~ScheduleDAGMILive() override;
 
   /// Return true if this DAG supports VReg liveness and RegPressure.
   bool hasVRegLiveness() const override { return true; }
 
   /// \brief Return true if register pressure tracking is enabled.
   bool isTrackingPressure() const { return ShouldTrackPressure; }
void initPolicy(MachineBasicBlock::iterator Begin,
                   unsigned NumRegionInstrs) override;
 
   void dumpPolicy() override;
 
   bool shouldTrackPressure() const override {
     return RegionPolicy.ShouldTrackPressure;
   }
 
+  bool shouldTrackLaneMasks() const override {
+    return RegionPolicy.ShouldTrackLaneMasks;
+  }
+
   void initialize(ScheduleDAGMI *dag) override;
 
   SUnit *pickNode(bool &IsTopNode) override;
 
   void schedNode(SUnit *SU, bool IsTopNode) override;
 
   void releaseTopNode(SUnit *SU) override {
     Top.releaseTopNode(SU);

llvm/trunk/include/llvm/CodeGen/RegisterPressure.h

void collect(const MachineInstr &MI, const TargetRegisterInfo &TRI,
                bool IgnoreDead);
 
   /// Use liveness information to find dead defs not marked with a dead flag
   /// and move them to the DeadDefs vector.
   void detectDeadDefs(const MachineInstr &MI, const LiveIntervals &LIS);
 
   /// Use liveness information to find out which uses/defs are partially
   /// undefined/dead and adjust the RegisterMaskPairs accordingly.
+  /// If \p AddFlagsMI is given then missing read-undef and dead flags will be
+  /// added to the instruction.
   void adjustLaneLiveness(const LiveIntervals &LIS,
-                          const MachineRegisterInfo &MRI, SlotIndex Pos);
-
+                          const MachineRegisterInfo &MRI, SlotIndex Pos,
+                          MachineInstr *AddFlagsMI = nullptr);
 };
 
 /// Array of PressureDiffs.
 class PressureDiffs {
   PressureDiff *PDiffArray;
   unsigned Size;
   unsigned Max;
 public:
void recede(const RegisterOperands &RegOpers,
               SmallVectorImpl<RegisterMaskPair> *LiveUses = nullptr);
 
   /// Recede until we find an instruction which is not a DebugValue.
   void recedeSkipDebugValues();
 
   /// Advance across the current instruction.
   void advance();
 
+  /// Advance across the current instruction.
+  /// This is a "low-level" variant of advance() which takes precomputed
+  /// RegisterOperands of the instruction.
+  void advance(const RegisterOperands &RegOpers);
+
   /// Finalize the region boundaries and recored live ins and live outs.
   void closeRegion();
 
   /// Initialize the LiveThru pressure set based on the untied defs found in
   /// RPTracker.
   void initLiveThru(const RegPressureTracker &RPTracker);
 
   /// Copy an existing live thru pressure result.

llvm/trunk/include/llvm/CodeGen/ScheduleDAGInstrs.h

public:
     /// Notify that the scheduler has finished scheduling the current region.
     virtual void exitRegion();
 
     /// buildSchedGraph - Build SUnits from the MachineBasicBlock that we are
     /// input.
     void buildSchedGraph(AliasAnalysis *AA,
                          RegPressureTracker *RPTracker = nullptr,
                          PressureDiffs *PDiffs = nullptr,
+                         LiveIntervals *LIS = nullptr,
                          bool TrackLaneMasks = false);
 
     /// addSchedBarrierDeps - Add dependencies from instructions in the current
     /// list of instructions being scheduled to scheduling barrier. We want to
     /// make sure instructions which define registers that are either used by
     /// the terminator or are live-out are properly scheduled. This is
     /// especially important when the definition latency of the return value(s)
     /// are too high to be hidden by the branch or when the liveout registers

llvm/trunk/lib/CodeGen/MachineScheduler.cpp

void ScheduleDAGMILive::enterRegion(MachineBasicBlock *bb,
   ScheduleDAGMI::enterRegion(bb, begin, end, regioninstrs);
 
   // For convenience remember the end of the liveness region.
   LiveRegionEnd = (RegionEnd == bb->end()) ? RegionEnd : std::next(RegionEnd);
 
   SUPressureDiffs.clear();
 
   ShouldTrackPressure = SchedImpl->shouldTrackPressure();
+  ShouldTrackLaneMasks = SchedImpl->shouldTrackLaneMasks();
+
+  if (ShouldTrackLaneMasks) {
+    if (!ShouldTrackPressure)
+      report_fatal_error("ShouldTrackLaneMasks requires ShouldTrackPressure");
+    // Dead subregister defs have no users and therefore no dependencies,
+    // moving them around may cause liveintervals to degrade into multiple
+    // components. Change independent components to have their own vreg to avoid
+    // this.
+    if (!DisconnectedComponentsRenamed)
+      LIS->renameDisconnectedComponents();
+  }
 }
 
 // Setup the register pressure trackers for the top scheduled top and bottom
 // scheduled regions.
 void ScheduleDAGMILive::initRegPressure() {
-  TopRPTracker.init(&MF, RegClassInfo, LIS, BB, RegionBegin, false, false);
-  BotRPTracker.init(&MF, RegClassInfo, LIS, BB, LiveRegionEnd, false, false);
+  TopRPTracker.init(&MF, RegClassInfo, LIS, BB, RegionBegin,
+                    ShouldTrackLaneMasks, false);
+  BotRPTracker.init(&MF, RegClassInfo, LIS, BB, LiveRegionEnd,
+                    ShouldTrackLaneMasks, false);
 
   // Close the RPTracker to finalize live ins.
   RPTracker.closeRegion();
 
   DEBUG(RPTracker.dump());
 
   // Initialize the live ins and live outs.
   TopRPTracker.addLiveRegs(RPTracker.getPressure().LiveInRegs);
updateScheduledPressure(const SUnit *SU,
   }
 }
 
 /// Update the PressureDiff array for liveness after scheduling this
 /// instruction.
 void ScheduleDAGMILive::updatePressureDiffs(
     ArrayRef<RegisterMaskPair> LiveUses) {
   for (const RegisterMaskPair &P : LiveUses) {
-    /// FIXME: Currently assuming single-use physregs.
     unsigned Reg = P.RegUnit;
-    assert(P.LaneMask != 0);
-    DEBUG(dbgs() << "  LiveReg: " << PrintVRegOrUnit(Reg, TRI) << "\n");
+    /// FIXME: Currently assuming single-use physregs.
     if (!TRI->isVirtualRegister(Reg))
       continue;
 
+    if (ShouldTrackLaneMasks) {
+      // If the register has just become live then other uses won't change
+      // this fact anymore => decrement pressure.
+      // If the register has just become dead then other uses make it come
+      // back to life => increment pressure.
+      bool Decrement = P.LaneMask != 0;
+
+      for (const VReg2SUnit &V2SU
+           : make_range(VRegUses.find(Reg), VRegUses.end())) {
+        SUnit &SU = *V2SU.SU;
+        if (SU.isScheduled || &SU == &ExitSU)
+          continue;
+
+        PressureDiff &PDiff = getPressureDiff(&SU);
+        PDiff.addPressureChange(Reg, Decrement, &MRI);
+        DEBUG(
+          dbgs() << "  UpdateRegP: SU(" << SU.NodeNum << ") "
+                 << PrintReg(Reg, TRI) << ':' << PrintLaneMask(P.LaneMask)
+                 << ' ' << *SU.getInstr();
+          dbgs() << "              to ";
+          PDiff.dump(*TRI);
+        );
+      }
+    } else {
+      assert(P.LaneMask != 0);
+      DEBUG(dbgs() << "  LiveReg: " << PrintVRegOrUnit(Reg, TRI) << "\n");
       // This may be called before CurrentBottom has been initialized. However,
       // BotRPTracker must have a valid position. We want the value live into the
       // instruction or live out of the block, so ask for the previous
       // instruction's live-out.
       const LiveInterval &LI = LIS->getInterval(Reg);
       VNInfo *VNI;
       MachineBasicBlock::const_iterator I =
         nextIfDebug(BotRPTracker.getPos(), BB->end());
       if (I == BB->end())
         VNI = LI.getVNInfoBefore(LIS->getMBBEndIdx(BB));
       else {
         LiveQueryResult LRQ = LI.Query(LIS->getInstructionIndex(I));
         VNI = LRQ.valueIn();
       }
       // RegisterPressureTracker guarantees that readsReg is true for LiveUses.
       assert(VNI && "No live value at use.");
       for (const VReg2SUnit &V2SU
            : make_range(VRegUses.find(Reg), VRegUses.end())) {
         SUnit *SU = V2SU.SU;
-      // If this use comes before the reaching def, it cannot be a last use, so
-      // descrease its pressure change.
+        // If this use comes before the reaching def, it cannot be a last use,
+        // so decrease its pressure change.
         if (!SU->isScheduled && SU != &ExitSU) {
           LiveQueryResult LRQ
             = LI.Query(LIS->getInstructionIndex(SU->getInstr()));
           if (LRQ.valueIn() == VNI) {
             PressureDiff &PDiff = getPressureDiff(SU);
             PDiff.addPressureChange(Reg, true, &MRI);
             DEBUG(
               dbgs() << "  UpdateRegP: SU(" << SU->NodeNum << ") "
                      << *SU->getInstr();
               dbgs() << "              to ";
               PDiff.dump(*TRI);
             );
           }
         }
       }
     }
   }
+}
 
 /// schedule - Called back from MachineScheduler::runOnMachineFunction
 /// after setting up the current scheduling region. [RegionBegin, RegionEnd)
 /// only includes instructions that have DAG nodes, not scheduling boundaries.
 ///
 /// This is a skeletal driver, with all the functionality pushed into helpers,
 /// so that it can be easily extended by experimental schedulers. Generally,
 /// implementing MachineSchedStrategy should be sufficient to implement a new
if (!ShouldTrackPressure) {
     RPTracker.reset();
     RegionCriticalPSets.clear();
     buildSchedGraph(AA);
     return;
   }
 
   // Initialize the register pressure tracker used by buildSchedGraph.
   RPTracker.init(&MF, RegClassInfo, LIS, BB, LiveRegionEnd,
-                 false, /*TrackUntiedDefs=*/true);
+                 ShouldTrackLaneMasks, /*TrackUntiedDefs=*/true);
 
   // Account for liveness generate by the region boundary.
   if (LiveRegionEnd != RegionEnd)
     RPTracker.recede();
 
   // Build the DAG, and compute current register pressure.
-  buildSchedGraph(AA, &RPTracker, &SUPressureDiffs);
+  buildSchedGraph(AA, &RPTracker, &SUPressureDiffs, LIS, ShouldTrackLaneMasks);
 
   // Initialize top/bottom trackers after computing region pressure.
   initRegPressure();
 }
 
 void ScheduleDAGMILive::computeDFSResult() {
   if (!DFSResult)
     DFSResult = new SchedDFSResult(/*BottomU*/true, MinSubtreeSize);
if (&*CurrentTop == MI)
       CurrentTop = nextIfDebug(++CurrentTop, CurrentBottom);
     else {
       moveInstruction(MI, CurrentTop);
       TopRPTracker.setPos(MI);
     }
 
     if (ShouldTrackPressure) {
       // Update top scheduled pressure.
-      TopRPTracker.advance();
+      RegisterOperands RegOpers;
+      RegOpers.collect(*MI, *TRI, MRI, ShouldTrackLaneMasks, false);
+      if (ShouldTrackLaneMasks) {
+        // Adjust liveness and add missing dead+read-undef flags.
+        SlotIndex SlotIdx = LIS->getInstructionIndex(MI).getRegSlot();
+        RegOpers.adjustLaneLiveness(*LIS, MRI, SlotIdx, MI);
+      } else {
+        // Adjust for missing dead-def flags.
+        RegOpers.detectDeadDefs(*MI, *LIS);
+      }
+
+      TopRPTracker.advance(RegOpers);
       assert(TopRPTracker.getPos() == CurrentTop && "out of sync");
       DEBUG(
         dbgs() << "Top Pressure:\n";
         dumpRegSetPressure(TopRPTracker.getRegSetPressureAtPos(), TRI);
       );
 
       updateScheduledPressure(SU, TopRPTracker.getPressure().MaxSetPressure);
     }
   }
   else {
     assert(SU->isBottomReady() && "node still has unscheduled dependencies");
     MachineBasicBlock::iterator priorII =
       priorNonDebug(CurrentBottom, CurrentTop);
     if (&*priorII == MI)
       CurrentBottom = priorII;
     else {
       if (&*CurrentTop == MI) {
         CurrentTop = nextIfDebug(++CurrentTop, priorII);
         TopRPTracker.setPos(CurrentTop);
       }
       moveInstruction(MI, CurrentBottom);
       CurrentBottom = MI;
     }
     if (ShouldTrackPressure) {
-      // Update bottom scheduled pressure.
+      RegisterOperands RegOpers;
+      RegOpers.collect(*MI, *TRI, MRI, ShouldTrackLaneMasks, false);
+      if (ShouldTrackLaneMasks) {
+        // Adjust liveness and add missing dead+read-undef flags.
+        SlotIndex SlotIdx = LIS->getInstructionIndex(MI).getRegSlot();
+        RegOpers.adjustLaneLiveness(*LIS, MRI, SlotIdx, MI);
+      } else {
+        // Adjust for missing dead-def flags.
+        RegOpers.detectDeadDefs(*MI, *LIS);
+      }
+
+      BotRPTracker.recedeSkipDebugValues();
       SmallVector<RegisterMaskPair, 8> LiveUses;
-      BotRPTracker.recede(&LiveUses);
+      BotRPTracker.recede(RegOpers, &LiveUses);
       assert(BotRPTracker.getPos() == CurrentBottom && "out of sync");
       DEBUG(
         dbgs() << "Bottom Pressure:\n";
         dumpRegSetPressure(BotRPTracker.getRegSetPressureAtPos(), TRI);
       );
 
       updateScheduledPressure(SU, BotRPTracker.getPressure().MaxSetPressure);
       updatePressureDiffs(LiveUses);

llvm/trunk/lib/CodeGen/RegisterPressure.cpp

auto I = std::find_if(RegUnits.begin(), RegUnits.end(),
                         });
   if (I == RegUnits.end()) {
     RegUnits.push_back(Pair);
   } else {
     I->LaneMask |= Pair.LaneMask;
   }
 }
 
+static void setRegZero(SmallVectorImpl<RegisterMaskPair> &RegUnits,
+                       unsigned RegUnit) {
+  auto I = std::find_if(RegUnits.begin(), RegUnits.end(),
+                        [RegUnit](const RegisterMaskPair Other) {
+                          return Other.RegUnit == RegUnit;
+                        });
+  if (I == RegUnits.end()) {
+    RegUnits.push_back(RegisterMaskPair(RegUnit, 0));
+  } else {
+    I->LaneMask = 0;
+  }
+}
+
 static void removeRegLanes(SmallVectorImpl<RegisterMaskPair> &RegUnits,
                            RegisterMaskPair Pair) {
   unsigned RegUnit = Pair.RegUnit;
   assert(Pair.LaneMask != 0);
   auto I = std::find_if(RegUnits.begin(), RegUnits.end(),
                         [RegUnit](const RegisterMaskPair Other) {
                           return Other.RegUnit == RegUnit;
                         });
if (LR != nullptr) {
       }
     }
     ++RI;
   }
 }
 
 void RegisterOperands::adjustLaneLiveness(const LiveIntervals &LIS,
                                           const MachineRegisterInfo &MRI,
-                                          SlotIndex Pos) {
+                                          SlotIndex Pos,
+                                          MachineInstr *AddFlagsMI) {
   for (auto I = Defs.begin(); I != Defs.end(); ) {
     LaneBitmask LiveAfter = getLiveLanesAt(LIS, MRI, true, I->RegUnit,
                                            Pos.getDeadSlot());
 #if 0
     unsigned DeadDef = I->LaneMask & ~LiveAfter;
     if (DeadDef != 0)
       addRegLanes(DeadDefs, RegisterMaskPair(I->RegUnit, DeadDef));
 #endif
+    // If the the def is all that is live after the instruction, then in case
+    // of a subregister def we need a read-undef flag.
+    unsigned RegUnit = I->RegUnit;
+    if (TargetRegisterInfo::isVirtualRegister(RegUnit) &&
+        AddFlagsMI != nullptr && (LiveAfter & ~I->LaneMask) == 0)
+      AddFlagsMI->setRegisterDefReadUndef(RegUnit);
+
     unsigned LaneMask = I->LaneMask & LiveAfter;
-    if (LaneMask == 0)
+    if (LaneMask == 0) {
       I = Defs.erase(I);
-    else {
+      // Make sure the operand is properly marked as Dead.
+      if (AddFlagsMI != nullptr)
+        AddFlagsMI->addRegisterDead(RegUnit, MRI.getTargetRegisterInfo());
+    } else {
       I->LaneMask = LaneMask;
       ++I;
     }
   }
   for (auto I = Uses.begin(); I != Uses.end(); ) {
     LaneBitmask LiveBefore = getLiveLanesAt(LIS, MRI, true, I->RegUnit,
                                             Pos.getBaseIndex());
     unsigned LaneMask = I->LaneMask & LiveBefore;
     if (LaneMask == 0) {
       I = Uses.erase(I);
     } else {
       I->LaneMask = LaneMask;
       ++I;
     }
   }
+  if (AddFlagsMI != nullptr) {
+    for (const RegisterMaskPair &P : DeadDefs) {
+      unsigned RegUnit = P.RegUnit;
+      LaneBitmask LiveAfter = getLiveLanesAt(LIS, MRI, true, RegUnit,
+                                             Pos.getDeadSlot());
+      if (LiveAfter == 0)
+        AddFlagsMI->setRegisterDefReadUndef(RegUnit);
+    }
+  }
 }
 
 /// Initialize an array of N PressureDiffs.
 void PressureDiffs::init(unsigned N) {
   Size = N;
   if (N <= Max) {
     memset(PDiffArray, 0, N * sizeof(PressureDiff));
     return;
for (const RegisterMaskPair &Def : RegOpers.Defs) {
     LaneBitmask LiveOut = Def.LaneMask & ~PreviousMask;
     if (LiveOut != 0) {
       discoverLiveOut(RegisterMaskPair(Reg, LiveOut));
       // Retroactively model effects on pressure of the live out lanes.
       increaseSetPressure(CurrSetPressure, *MRI, Reg, 0, LiveOut);
       PreviousMask = LiveOut;
     }
 
+    if (NewMask == 0) {
+      // Add a 0 entry to LiveUses as a marker that the complete vreg has become
+      // dead.
+      if (TrackLaneMasks && LiveUses != nullptr)
+        setRegZero(*LiveUses, Reg);
+    }
+
     decreaseRegPressure(Reg, PreviousMask, NewMask);
   }
 
   SlotIndex SlotIdx;
   if (RequireIntervals)
     SlotIdx = LIS->getInstructionIndex(CurrPos).getRegSlot();
 
   // Generate liveness for uses.
   for (const RegisterMaskPair &Use : RegOpers.Uses) {
     unsigned Reg = Use.RegUnit;
     assert(Use.LaneMask != 0);
     LaneBitmask PreviousMask = LiveRegs.insert(Use);
     LaneBitmask NewMask = PreviousMask | Use.LaneMask;
     if (NewMask == PreviousMask)
       continue;
 
     // Did the register just become live?
     if (PreviousMask == 0) {
       if (LiveUses != nullptr) {
-        unsigned NewLanes = NewMask & ~PreviousMask;
-        addRegLanes(*LiveUses, RegisterMaskPair(Reg, NewLanes));
+        if (!TrackLaneMasks) {
+          addRegLanes(*LiveUses, RegisterMaskPair(Reg, NewMask));
+        } else {
+          auto I = std::find_if(LiveUses->begin(), LiveUses->end(),
+                                [Reg](const RegisterMaskPair Other) {
+                                return Other.RegUnit == Reg;
+                                });
+          bool IsRedef = I != LiveUses->end();
+          if (IsRedef) {
+            // ignore re-defs here...
+            assert(I->LaneMask == 0);
+            removeRegLanes(*LiveUses, RegisterMaskPair(Reg, NewMask));
+          } else {
+            addRegLanes(*LiveUses, RegisterMaskPair(Reg, NewMask));
+          }
+        }
       }
 
       // Discover live outs if this may be the first occurance of this register.
       LaneBitmask LiveOut = getLiveThroughAt(Reg, SlotIdx);
       discoverLiveOut(RegisterMaskPair(Reg, LiveOut));
     }
 
     increaseRegPressure(Reg, PreviousMask, NewMask);
void RegPressureTracker::recede(SmallVectorImpl<RegisterMaskPair> *LiveUses) {
   } else if (RequireIntervals) {
     RegOpers.detectDeadDefs(MI, *LIS);
   }
 
   recede(RegOpers, LiveUses);
 }
 
 /// Advance across the current instruction.
-void RegPressureTracker::advance() {
+void RegPressureTracker::advance(const RegisterOperands &RegOpers) {
   assert(!TrackUntiedDefs && "unsupported mode");
-
   assert(CurrPos != MBB->end());
   if (!isTopClosed())
     closeTop();
 
   SlotIndex SlotIdx;
   if (RequireIntervals)
     SlotIdx = getCurrSlot();
 
   // Open the bottom of the region using slot indexes.
   if (isBottomClosed()) {
     if (RequireIntervals)
       static_cast<IntervalPressure&>(P).openBottom(SlotIdx);
     else
       static_cast<RegionPressure&>(P).openBottom(CurrPos);
   }
 
-  RegisterOperands RegOpers;
-  RegOpers.collect(*CurrPos, *TRI, *MRI, TrackLaneMasks, false);
-  if (TrackLaneMasks)
-    RegOpers.adjustLaneLiveness(*LIS, *MRI, SlotIdx);
-
   for (const RegisterMaskPair &Use : RegOpers.Uses) {
     unsigned Reg = Use.RegUnit;
     LaneBitmask LiveMask = LiveRegs.contains(Reg);
     LaneBitmask LiveIn = Use.LaneMask & ~LiveMask;
     if (LiveIn != 0) {
       discoverLiveIn(RegisterMaskPair(Reg, LiveIn));
       increaseRegPressure(Reg, LiveMask, LiveMask | LiveIn);
       LiveRegs.insert(RegisterMaskPair(Reg, LiveIn));
void RegPressureTracker::advance(const RegisterOperands &RegOpers) {
   bumpDeadDefs(RegOpers.DeadDefs);
 
   // Find the next instruction.
   do
     ++CurrPos;
   while (CurrPos != MBB->end() && CurrPos->isDebugValue());
 }
 
+void RegPressureTracker::advance() {
+  const MachineInstr &MI = *CurrPos;
+  RegisterOperands RegOpers;
+  RegOpers.collect(MI, *TRI, *MRI, TrackLaneMasks, false);
+  if (TrackLaneMasks) {
+    SlotIndex SlotIdx = getCurrSlot();
+    RegOpers.adjustLaneLiveness(*LIS, *MRI, SlotIdx);
+  }
+  advance(RegOpers);
+}
+
 /// Find the max change in excess pressure across all sets.
 static void computeExcessPressureDelta(ArrayRef<unsigned> OldPressureVec,
                                        ArrayRef<unsigned> NewPressureVec,
                                        RegPressureDelta &Delta,
                                        const RegisterClassInfo *RCI,
                                        ArrayRef<unsigned> LiveThruPressureVec) {
   Delta.Excess = PressureChange();
   for (unsigned i = 0, e = OldPressureVec.size(); i < e; ++i) {

llvm/trunk/lib/CodeGen/ScheduleDAGInstrs.cpp

 
 #include "llvm/CodeGen/ScheduleDAGInstrs.h"
 #include "llvm/ADT/IntEqClasses.h"
 #include "llvm/ADT/MapVector.h"
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/Analysis/AliasAnalysis.h"
 #include "llvm/Analysis/ValueTracking.h"
+#include "llvm/CodeGen/LiveIntervalAnalysis.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
 #include "llvm/CodeGen/MachineMemOperand.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/PseudoSourceValue.h"
 #include "llvm/CodeGen/RegisterPressure.h"
 #include "llvm/CodeGen/ScheduleDFS.h"
if (!MO.readsReg())
       continue;
     if (TrackLaneMasks && !MO.isUse())
       continue;
 
     unsigned Reg = MO.getReg();
     if (!TargetRegisterInfo::isVirtualRegister(Reg))
       continue;
 
+    // Ignore re-defs.
+    if (TrackLaneMasks) {
+      bool FoundDef = false;
+      for (const MachineOperand &MO2 : MI->operands()) {
+        if (MO2.isReg() && MO2.isDef() && MO2.getReg() == Reg && !MO2.isDead()) {
+          FoundDef = true;
+          break;
+        }
+      }
+      if (FoundDef)
+        continue;
+    }
+
     // Record this local VReg use.
     VReg2SUnitMultiMap::iterator UI = VRegUses.find(Reg);
     for (; UI != VRegUses.end(); ++UI) {
       if (UI->SU == SU)
         break;
     }
     if (UI == VRegUses.end())
       VRegUses.insert(VReg2SUnit(Reg, 0, SU));
   }
 }
 
 /// If RegPressure is non-null, compute register pressure as a side effect. The
 /// DAG builder is an efficient place to do it because it already visits
 /// operands.
 void ScheduleDAGInstrs::buildSchedGraph(AliasAnalysis *AA,
                                         RegPressureTracker *RPTracker,
                                         PressureDiffs *PDiffs,
+                                        LiveIntervals *LIS,
                                         bool TrackLaneMasks) {
   const TargetSubtargetInfo &ST = MF.getSubtarget();
   bool UseAA = EnableAASchedMI.getNumOccurrences() > 0 ? EnableAASchedMI
                                                        : ST.useAA();
   AliasAnalysis *AAForDep = UseAA ? AA : nullptr;
 
   this->TrackLaneMasks = TrackLaneMasks;
   MISUnitMap.clear();
for (MachineBasicBlock::iterator MII = RegionEnd, MIE = RegionBegin;
     SUnit *SU = MISUnitMap[MI];
     assert(SU && "No SUnit mapped to this MI");
 
     if (RPTracker) {
       collectVRegUses(SU);
 
       RegisterOperands RegOpers;
       RegOpers.collect(*MI, *TRI, MRI, TrackLaneMasks, false);
+      if (TrackLaneMasks) {
+        SlotIndex SlotIdx = LIS->getInstructionIndex(MI);
+        RegOpers.adjustLaneLiveness(*LIS, MRI, SlotIdx);
+      }
       if (PDiffs != nullptr)
         PDiffs->addInstruction(SU->NodeNum, RegOpers, MRI);
 
       RPTracker->recedeSkipDebugValues();
       assert(&*RPTracker->getPos() == MI && "RPTracker in sync");
       RPTracker->recede(RegOpers);
     }