[CodeGen] Fix incorrect usage of MCPhysReg for diff list elements

[barannikov88]

Summary

The lists contain differences between register numbers, not the register
numbers themselves. Since a difference can also be negative, this also
changes its type to signed.

Changing the type to signed exposed a "bug". For AMDGPU, which has many
registers, the first element of a sequence could be as big as ~45k.
The value does not fit into int16_t, but fits into uint16_t. The bug
didn't show up because of unsigned wrapping and truncation of the Val
field in the advance() method.

To fix the issue, I changed the way regunit difflists are encoded. The
4-bit 'scale' field of MCRegisterDesc::RegUnit was replaced by 12-bit
number of the first regunit, and the first element of each of the lists
was removed. The higher 20 bits of RegUnit field contain the initial
offset into DiffLists array.
AMDGPU has 1'409 regunits (2^12 = 4'096), and the biggest offset is
80'041 (2^20 = 1'048'576). That is, there is enough room.

Changing the encoding method also resulted in a smaller array size, the
numbers are below (I omitted targets with less than 100 elements).

AMDGPU   | 80052 | 78741 |  -1,6%
RISCV    |  6498 |  6297 |  -3,1%
ARM      |  4181 |  3966 |  -5,1%
AArch64  |  2770 |  2592 |  -6,4%
PPC      |  1578 |  1441 |  -8,7%
Hexagon  |   994 |   740 | -25,6%
R600     |   508 |   398 | -21,7%
VE       |   471 |   459 |  -2,5%
Sparc    |   381 |   363 |  -4,7%
X86      |   326 |   208 | -36,2%
Mips     |   253 |   200 | -20,9%
SystemZ  |   186 |   162 | -12,9%

[barannikov88]

clang-format

[foad]

Nice!

llvm/include/llvm/MC/MCRegisterInfo.h

699

Maybe inline advance into its one remaining caller.

[barannikov88]

llvm/include/llvm/MC/MCRegisterInfo.h

699

Changed prior to commit.

[barannikov88]

This causes buildbot failures:
https://lab.llvm.org/buildbot/#/builders/38/builds/12037
https://lab.llvm.org/buildbot/#/builders/5/builds/33885
https://lab.llvm.org/buildbot/#/builders/236/builds/4189
Need some time to investigate, reverted for now.

[barannikov88]

The issue with the first buildbot breakage is that needsStackFrame in HexagonFrameLowering.cpp calls subregs_inclusive for the zero register.
https://github.com/llvm/llvm-project/blob/14bc3748109c73adde7350da60366149ae0d4c3a/llvm/lib/Target/Hexagon/HexagonFrameLowering.cpp#L314
It used to work because MCRegisterDesc for the zero register referred to the beginning of the DiffList array, where it contained two consecutive zeros.
The effect was as if the zero register didn't have any sub- or super-registers.
After this patch, these two zero entries are no longer emitted.

It feels like subregs_inclusive and related methods (and maybe even MCRegisterInfo::get) should not allow zero register input,
but it appears that many places rely on this behavior (probably unintended). Fixing all this places is out of scope of this patch,
so I'll just restore the original behavior by special-casing zero register input with a fixme.

[barannikov88]

Add a couple of assertions and fix detected bugs

[barannikov88]

I added a couple of assertions.
Although there were many crashes, they narrowed down to just a few places, so I thought I might be able to fix them is this patch.

llvm/lib/Target/AMDGPU/SIFrameLowering.cpp

568

@arsenm @foad Please take a look. Is this a correct change?
MFI->getGITPtrLoReg(MF) above returns null register for !ST.isAmdPalOS().
Also, is it ok for this change to be part of this patch or should it be fixed separately?

[barannikov88]

Fix a couple more places

[barannikov88]

Fix another occurrence in bolt

[barannikov88]

Rebase on top of D151285

[barannikov88]

Rebase

[barannikov88]

Rebase on top of D151289

[barannikov88]

Rebase

[barannikov88]

Drop Hexagon changes (extracted into separate review)

[barannikov88]

Rebase

llvm/include/llvm/MC/MCRegisterInfo.h

struct MCRegisterDesc {
   uint32_t Name;      // Printable name for the reg (for debugging)
   uint32_t SubRegs;   // Sub-register set, described above
   uint32_t SuperRegs; // Super-register set, described above
 
   // Offset into MCRI::SubRegIndices of a list of sub-register indices for each
   // sub-register in SubRegs.
   uint32_t SubRegIndices;
 
-  // RegUnits - Points to the list of register units. The low 4 bits holds the
-  // Scale, the high bits hold an offset into DiffLists. See MCRegUnitIterator.
+  // Points to the list of register units. The low bits hold the first regunit
+  // number, the high bits hold an offset into DiffLists. See MCRegUnitIterator.
   uint32_t RegUnits;
 
   /// Index into list with lane mask sequences. The sequence contains a lanemask
   /// for every register unit.
   uint16_t RegUnitLaneMasks;
 };
 
 /// MCRegisterInfo base class - We assume that the target defines a static
private:
   const MCRegisterDesc *Desc;                 // Pointer to the descriptor array
   unsigned NumRegs;                           // Number of entries in the array
   MCRegister RAReg;                           // Return address register
   MCRegister PCReg;                           // Program counter register
   const MCRegisterClass *Classes;             // Pointer to the regclass array
   unsigned NumClasses;                        // Number of entries in the array
   unsigned NumRegUnits;                       // Number of regunits.
   const MCPhysReg (*RegUnitRoots)[2];         // Pointer to regunit root table.
-  const MCPhysReg *DiffLists;                 // Pointer to the difflists array
+  const int16_t *DiffLists;                   // Pointer to the difflists array
   const LaneBitmask *RegUnitMaskSequences;    // Pointer to lane mask sequences
                                               // for register units.
   const char *RegStrings;                     // Pointer to the string table.
   const char *RegClassStrings;                // Pointer to the class strings.
   const uint16_t *SubRegIndices;              // Pointer to the subreg lookup
                                               // array.
   const SubRegCoveredBits *SubRegIdxRanges;   // Pointer to the subreg covered
                                               // bit ranges array.
public:
   // Forward declaration to become a friend class of DiffListIterator.
   template <class SubT> class mc_difflist_iterator;
 
   /// DiffListIterator - Base iterator class that can traverse the
   /// differentially encoded register and regunit lists in DiffLists.
   /// Don't use this class directly, use one of the specialized sub-classes
   /// defined below.
   class DiffListIterator {
-    uint16_t Val = 0;
-    const MCPhysReg *List = nullptr;
+    unsigned Val = 0;
+    const int16_t *List = nullptr;
 
   protected:
     /// Create an invalid iterator. Call init() to point to something useful.
     DiffListIterator() = default;
 
-    /// init - Point the iterator to InitVal, decoding subsequent values from
-    /// DiffList. The iterator will initially point to InitVal, sub-classes are
-    /// responsible for skipping the seed value if it is not part of the list.
-    void init(MCPhysReg InitVal, const MCPhysReg *DiffList) {
+    /// Point the iterator to InitVal, decoding subsequent values from DiffList.
+    void init(unsigned InitVal, const int16_t *DiffList) {
       Val = InitVal;
       List = DiffList;
     }
 
-    /// advance - Move to the next list position, return the applied
-    /// differential. This function does not detect the end of the list, that
-    /// is the caller's responsibility (by checking for a 0 return value).
-    MCRegister advance() {
-      assert(isValid() && "Cannot move off the end of the list.");
-      MCPhysReg D = *List++;
-      Val += D;
-      return D;
-    }
-
   public:
     /// isValid - returns true if this iterator is not yet at the end.
     bool isValid() const { return List; }
 
     /// Dereference the iterator to get the value at the current position.
     MCRegister operator*() const { return Val; }
 
     /// Pre-increment to move to the next position.
     void operator++() {
+      assert(isValid() && "Cannot move off the end of the list.");
+      int16_t D = *List++;
+      Val += D;
       // The end of the list is encoded as a 0 differential.
-      if (!advance())
+      if (!D)
         List = nullptr;
     }
 
     template <class SubT> friend class MCRegisterInfo::mc_difflist_iterator;
   };
 
   /// Forward iterator using DiffListIterator.
   template <class SubT>
   class mc_difflist_iterator
       : public iterator_facade_base<mc_difflist_iterator<SubT>,
                                     std::forward_iterator_tag, MCPhysReg> {
     MCRegisterInfo::DiffListIterator Iter;
     /// Current value as MCPhysReg, so we can return a reference to it.
     MCPhysReg Val = 0;
 
   protected:
     mc_difflist_iterator(MCRegisterInfo::DiffListIterator Iter) : Iter(Iter) {}
 
-    // Allow conversion between instantiations where valid.
-    mc_difflist_iterator(MCRegister Reg, const MCPhysReg *DiffList) {
-      Iter.init(Reg, DiffList);
+    /// Point the iterator to InitVal, decoding subsequent values from DiffList.
+    void init(unsigned InitVal, const int16_t *DiffList) {
+      Iter.init(InitVal, DiffList);
       Val = *Iter;
     }
 
   public:
     // Allow default construction to build variables, but this doesn't build
     // a useful iterator.
     mc_difflist_iterator() = default;
 
public:
 
   /// Forward iterator over all sub-registers.
   /// TODO: Replace remaining uses of MCSubRegIterator.
   class mc_subreg_iterator : public mc_difflist_iterator<mc_subreg_iterator> {
   public:
     mc_subreg_iterator(MCRegisterInfo::DiffListIterator Iter)
         : mc_difflist_iterator(Iter) {}
     mc_subreg_iterator() = default;
-    mc_subreg_iterator(MCRegister Reg, const MCRegisterInfo *MCRI)
-        : mc_difflist_iterator(Reg, MCRI->DiffLists + MCRI->get(Reg).SubRegs) {}
+
+    mc_subreg_iterator(MCRegister Reg, const MCRegisterInfo *MCRI) {
+      assert(MCRegister::isPhysicalRegister(Reg.id()));
+      init(Reg.id(), MCRI->DiffLists + MCRI->get(Reg).SubRegs);
+    }
   };
 
   /// Forward iterator over all super-registers.
   /// TODO: Replace remaining uses of MCSuperRegIterator.
   class mc_superreg_iterator
       : public mc_difflist_iterator<mc_superreg_iterator> {
   public:
     mc_superreg_iterator(MCRegisterInfo::DiffListIterator Iter)
         : mc_difflist_iterator(Iter) {}
     mc_superreg_iterator() = default;
-    mc_superreg_iterator(MCRegister Reg, const MCRegisterInfo *MCRI)
-        : mc_difflist_iterator(Reg,
-                               MCRI->DiffLists + MCRI->get(Reg).SuperRegs) {}
+
+    mc_superreg_iterator(MCRegister Reg, const MCRegisterInfo *MCRI) {
+      assert(MCRegister::isPhysicalRegister(Reg.id()));
+      init(Reg.id(), MCRI->DiffLists + MCRI->get(Reg).SuperRegs);
+    }
   };
 
   /// Return an iterator range over all sub-registers of \p Reg, excluding \p
   /// Reg.
   iterator_range<mc_subreg_iterator> subregs(MCRegister Reg) const {
     return make_range(std::next(mc_subreg_iterator(Reg, this)),
                       mc_subreg_iterator::end());
   }
public:
   friend class MCSuperRegIterator;
   friend class MCRegUnitIterator;
   friend class MCRegUnitMaskIterator;
   friend class MCRegUnitRootIterator;
 
   /// Initialize MCRegisterInfo, called by TableGen
   /// auto-generated routines. *DO NOT USE*.
   void InitMCRegisterInfo(const MCRegisterDesc *D, unsigned NR, unsigned RA,
-                          unsigned PC,
-                          const MCRegisterClass *C, unsigned NC,
-                          const MCPhysReg (*RURoots)[2],
-                          unsigned NRU,
-                          const MCPhysReg *DL,
-                          const LaneBitmask *RUMS,
-                          const char *Strings,
-                          const char *ClassStrings,
-                          const uint16_t *SubIndices,
-                          unsigned NumIndices,
+                          unsigned PC, const MCRegisterClass *C, unsigned NC,
+                          const MCPhysReg (*RURoots)[2], unsigned NRU,
+                          const int16_t *DL, const LaneBitmask *RUMS,
+                          const char *Strings, const char *ClassStrings,
+                          const uint16_t *SubIndices, unsigned NumIndices,
                           const SubRegCoveredBits *SubIdxRanges,
                           const uint16_t *RET) {
     Desc = D;
     NumRegs = NR;
     RAReg = RA;
     PCReg = PC;
     Classes = C;
     DiffLists = DL;

 // aliasing registers. Use these iterator classes to traverse the lists.
 
 /// MCSubRegIterator enumerates all sub-registers of Reg.
 /// If IncludeSelf is set, Reg itself is included in the list.
 class MCSubRegIterator : public MCRegisterInfo::DiffListIterator {
 public:
   MCSubRegIterator(MCRegister Reg, const MCRegisterInfo *MCRI,
                    bool IncludeSelf = false) {
-    init(Reg, MCRI->DiffLists + MCRI->get(Reg).SubRegs);
+    assert(MCRegister::isPhysicalRegister(Reg.id()));
+    init(Reg.id(), MCRI->DiffLists + MCRI->get(Reg).SubRegs);
     // Initially, the iterator points to Reg itself.
     if (!IncludeSelf)
       ++*this;
   }
 };
 
 /// Iterator that enumerates the sub-registers of a Reg and the associated
 /// sub-register indices.

 /// MCSuperRegIterator enumerates all super-registers of Reg.
 /// If IncludeSelf is set, Reg itself is included in the list.
 class MCSuperRegIterator : public MCRegisterInfo::DiffListIterator {
 public:
   MCSuperRegIterator() = default;
 
   MCSuperRegIterator(MCRegister Reg, const MCRegisterInfo *MCRI,
                      bool IncludeSelf = false) {
-    init(Reg, MCRI->DiffLists + MCRI->get(Reg).SuperRegs);
+    assert(MCRegister::isPhysicalRegister(Reg.id()));
+    init(Reg.id(), MCRI->DiffLists + MCRI->get(Reg).SuperRegs);
     // Initially, the iterator points to Reg itself.
     if (!IncludeSelf)
       ++*this;
   }
 };
 
 // Definition for isSuperRegister. Put it down here since it needs the
 // iterator defined above in addition to the MCRegisterInfo class itself.

 //
 // A target with a complicated sub-register structure will typically have many
 // fewer register units than actual registers. MCRI::getNumRegUnits() returns
 // the number of register units in the target.
 
 // MCRegUnitIterator enumerates a list of register units for Reg. The list is
 // in ascending numerical order.
 class MCRegUnitIterator : public MCRegisterInfo::DiffListIterator {
+  // The value must be kept in sync with RegisterInfoEmitter.cpp.
+  static constexpr unsigned RegUnitBits = 12;
+
 public:
   /// MCRegUnitIterator - Create an iterator that traverses the register units
   /// in Reg.
   MCRegUnitIterator() = default;
 
   MCRegUnitIterator(MCRegister Reg, const MCRegisterInfo *MCRI) {
     assert(Reg && "Null register has no regunits");
     assert(MCRegister::isPhysicalRegister(Reg.id()));
     // Decode the RegUnits MCRegisterDesc field.
     unsigned RU = MCRI->get(Reg).RegUnits;
-    unsigned Scale = RU & 15;
-    unsigned Offset = RU >> 4;
-
-    // Initialize the iterator to Reg * Scale, and the List pointer to
-    // DiffLists + Offset.
-    init(Reg * Scale, MCRI->DiffLists + Offset);
-
-    // That may not be a valid unit, we need to advance by one to get the real
-    // unit number. The first differential can be 0 which would normally
-    // terminate the list, but since we know every register has at least one
-    // unit, we can allow a 0 differential here.
-    advance();
+    unsigned FirstRU = RU & ((1u << RegUnitBits) - 1);
+    unsigned Offset = RU >> RegUnitBits;
+    init(FirstRU, MCRI->DiffLists + Offset);
   }
 
   MCRegUnitIterator &operator++() {
     MCRegisterInfo::DiffListIterator::operator++();
     return *this;
   }
 };
 

llvm/lib/CodeGen/CriticalAntiDepBreaker.cpp

for (unsigned i = 0, e = MI.getNumOperands(); i != e; ++i) {
       MachineOperand &MO = MI.getOperand(i);
 
       if (MO.isRegMask()) {
         auto ClobbersPhysRegAndSubRegs = [&](unsigned PhysReg) {
           return all_of(TRI->subregs_inclusive(PhysReg),
                         [&](MCPhysReg SR) { return MO.clobbersPhysReg(SR); });
         };
 
-        for (unsigned i = 0, e = TRI->getNumRegs(); i != e; ++i) {
+        for (unsigned i = 1, e = TRI->getNumRegs(); i != e; ++i) {
           if (ClobbersPhysRegAndSubRegs(i)) {
             DefIndices[i] = Count;
             KillIndices[i] = ~0u;
             KeepRegs.reset(i);
             Classes[i] = nullptr;
             RegRefs.erase(i);
           }
         }

llvm/lib/CodeGen/StackMaps.cpp

StackMaps::parseRegisterLiveOutMask(const uint32_t *Mask) const {
   for (auto I = LiveOuts.begin(), E = LiveOuts.end(); I != E; ++I) {
     for (auto *II = std::next(I); II != E; ++II) {
       if (I->DwarfRegNum != II->DwarfRegNum) {
         // Skip all the now invalid entries.
         I = --II;
         break;
       }
       I->Size = std::max(I->Size, II->Size);
-      if (TRI->isSuperRegister(I->Reg, II->Reg))
+      if (I->Reg && TRI->isSuperRegister(I->Reg, II->Reg))
         I->Reg = II->Reg;
       II->Reg = 0; // mark for deletion.
     }
   }
 
   llvm::erase_if(LiveOuts, [](const LiveOutReg &LO) { return LO.Reg == 0; });
 
   return LiveOuts;

llvm/lib/Target/AMDGPU/SIFrameLowering.cpp

Register SIFrameLowering::getEntryFunctionReservedScratchRsrcReg(
   // Skip the last N reserved elements because they should have already been
   // reserved for VCC etc.
   Register GITPtrLoReg = MFI->getGITPtrLoReg(MF);
   for (MCPhysReg Reg : AllSGPR128s) {
     // Pick the first unallocated one. Make sure we don't clobber the other
     // reserved input we needed. Also for PAL, make sure we don't clobber
     // the GIT pointer passed in SGPR0 or SGPR8.
     if (!MRI.isPhysRegUsed(Reg) && MRI.isAllocatable(Reg) &&
-        !TRI->isSubRegisterEq(Reg, GITPtrLoReg)) {
+        !(GITPtrLoReg && TRI->isSubRegisterEq(Reg, GITPtrLoReg))) {
       MRI.replaceRegWith(ScratchRsrcReg, Reg);
       MFI->setScratchRSrcReg(Reg);
       return Reg;
     }
   }
 
   return ScratchRsrcReg;
 }

llvm/lib/Target/Hexagon/HexagonFrameLowering.cpp

for (const MachineInstr &MI : MBB) {
       for (const MachineOperand &MO : MI.operands()) {
         // While the presence of a frame index does not prove that a stack
         // frame will be required, all frame indexes should be within alloc-
         // frame/deallocframe. Otherwise, the code that translates a frame
         // index into an offset would have to be aware of the placement of
         // the frame creation/destruction instructions.
         if (MO.isFI())
           return true;
-        if (MO.isReg()) {
+        if (MO.isReg() && MO.getReg()) {
           Register R = MO.getReg();
           // Virtual registers will need scavenging, which then may require
           // a stack slot.
           if (R.isVirtual())
             return true;
           for (MCPhysReg S : HRI.subregs_inclusive(R))
             if (CSR[S])
               return true;

llvm/lib/Target/Mips/AsmParser/MipsAsmParser.cpp

if (ABI.ArePtrs64bit() && isGP64bit()) {
     // source register.
 
     const MipsMCExpr *HighestExpr =
         MipsMCExpr::create(MipsMCExpr::MEK_HIGHEST, SymExpr, getContext());
     const MipsMCExpr *HigherExpr =
         MipsMCExpr::create(MipsMCExpr::MEK_HIGHER, SymExpr, getContext());
 
     bool RdRegIsRsReg =
+        SrcReg &&
         getContext().getRegisterInfo()->isSuperOrSubRegisterEq(DstReg, SrcReg);
 
     if (canUseATReg() && UseSrcReg && RdRegIsRsReg) {
       unsigned ATReg = getATReg(IDLoc);
 
       // If $rs is the same as $rd:
       // (d)la $rd, sym($rd) => lui    $at, %highest(sym)
       //                        daddiu $at, $at, %higher(sym)

llvm/utils/TableGen/RegisterInfoEmitter.cpp

 // differential list as:
 //
 //   out[0] = InitVal;
 //   out[n+1] = out[n] + diff[n]; // n = 0, 1, ...
 //
 // The initial value depends on the specific list. The list is terminated by a
 // 0 differential which means we can't encode repeated elements.
 
-typedef SmallVector<uint16_t, 4> DiffVec;
+typedef SmallVector<int16_t, 4> DiffVec;
 typedef SmallVector<LaneBitmask, 4> MaskVec;
 
-// Differentially encode a sequence of numbers into V. The starting value and
-// terminating 0 are not added to V, so it will have the same size as List.
-static
-DiffVec &diffEncode(DiffVec &V, unsigned InitVal, SparseBitVector<> List) {
+// Fills V with differentials between every two consecutive elements of List.
+static DiffVec &diffEncode(DiffVec &V, SparseBitVector<> List) {
   assert(V.empty() && "Clear DiffVec before diffEncode.");
-  uint16_t Val = uint16_t(InitVal);
-
-  for (uint16_t Cur : List) {
+  SparseBitVector<>::iterator I = List.begin(), E = List.end();
+  unsigned Val = *I;
+  while (++I != E) {
+    unsigned Cur = *I;
     V.push_back(Cur - Val);
     Val = Cur;
   }
   return V;
 }
 
 template<typename Iter>
 static
 DiffVec &diffEncode(DiffVec &V, unsigned InitVal, Iter Begin, Iter End) {
   assert(V.empty() && "Clear DiffVec before diffEncode.");
-  uint16_t Val = uint16_t(InitVal);
+  unsigned Val = InitVal;
   for (Iter I = Begin; I != End; ++I) {
-    uint16_t Cur = (*I)->EnumValue;
+    unsigned Cur = (*I)->EnumValue;
     V.push_back(Cur - Val);
     Val = Cur;
   }
   return V;
 }
 
-static void printDiff16(raw_ostream &OS, uint16_t Val) {
-  OS << Val;
-}
+static void printDiff16(raw_ostream &OS, int16_t Val) { OS << Val; }
 
 static void printMask(raw_ostream &OS, LaneBitmask Val) {
   OS << "LaneBitmask(0x" << PrintLaneMask(Val) << ')';
 }
 
 // Try to combine Idx's compose map into Vec if it is compatible.
 // Return false if it's not possible.
 static bool combine(const CodeGenSubRegIndex *Idx,
RegisterInfoEmitter::runMCDesc(raw_ostream &OS, CodeGenTarget &Target,
   // allows us to share suffixes.
   typedef std::vector<const CodeGenRegister*> RegVec;
 
   // Differentially encoded lists.
   SequenceToOffsetTable<DiffVec> DiffSeqs;
   SmallVector<DiffVec, 4> SubRegLists(Regs.size());
   SmallVector<DiffVec, 4> SuperRegLists(Regs.size());
   SmallVector<DiffVec, 4> RegUnitLists(Regs.size());
-  SmallVector<unsigned, 4> RegUnitInitScale(Regs.size());
 
   // List of lane masks accompanying register unit sequences.
   SequenceToOffsetTable<MaskVec> LaneMaskSeqs;
   SmallVector<MaskVec, 4> RegUnitLaneMasks(Regs.size());
 
   // Keep track of sub-register names as well. These are not differentially
   // encoded.
   typedef SmallVector<const CodeGenSubRegIndex*, 4> SubRegIdxVec;
for (auto I = Regs.begin(), E = Regs.end(); I != E; ++I, ++i) {
     SubRegIdxSeqs.add(SRIs);
 
     // Super-registers are already computed.
     const RegVec &SuperRegList = Reg.getSuperRegs();
     diffEncode(SuperRegLists[i], Reg.EnumValue, SuperRegList.begin(),
                SuperRegList.end());
     DiffSeqs.add(SuperRegLists[i]);
 
-    // Differentially encode the register unit list, seeded by register number.
-    // First compute a scale factor that allows more diff-lists to be reused:
-    //
-    //   D0 -> (S0, S1)
-    //   D1 -> (S2, S3)
-    //
-    // A scale factor of 2 allows D0 and D1 to share a diff-list. The initial
-    // value for the differential decoder is the register number multiplied by
-    // the scale.
-    //
-    // Check the neighboring registers for arithmetic progressions.
-    unsigned ScaleA = ~0u, ScaleB = ~0u;
-    SparseBitVector<> RUs = Reg.getNativeRegUnits();
-    if (I != Regs.begin() &&
-        std::prev(I)->getNativeRegUnits().count() == RUs.count())
-      ScaleB = *RUs.begin() - *std::prev(I)->getNativeRegUnits().begin();
-    if (std::next(I) != Regs.end() &&
-        std::next(I)->getNativeRegUnits().count() == RUs.count())
-      ScaleA = *std::next(I)->getNativeRegUnits().begin() - *RUs.begin();
-    unsigned Scale = std::min(ScaleB, ScaleA);
-    // Default the scale to 0 if it can't be encoded in 4 bits.
-    if (Scale >= 16)
-      Scale = 0;
-    RegUnitInitScale[i] = Scale;
-    DiffSeqs.add(diffEncode(RegUnitLists[i], Scale * Reg.EnumValue, RUs));
+    const SparseBitVector<> &RUs = Reg.getNativeRegUnits();
+    DiffSeqs.add(diffEncode(RegUnitLists[i], RUs));
 
     const auto &RUMasks = Reg.getRegUnitLaneMasks();
     MaskVec &LaneMaskVec = RegUnitLaneMasks[i];
     assert(LaneMaskVec.empty());
     llvm::append_range(LaneMaskVec, RUMasks);
     // Terminator mask should not be used inside of the list.
 #ifndef NDEBUG
     for (LaneBitmask M : LaneMaskVec) {
       assert(!M.all() && "terminator mask should not be part of the list");
     }
 #endif
     LaneMaskSeqs.add(LaneMaskVec);
   }
 
   // Compute the final layout of the sequence table.
   DiffSeqs.layout();
   LaneMaskSeqs.layout();
   SubRegIdxSeqs.layout();
 
   OS << "namespace llvm {\n\n";
 
   const std::string &TargetName = std::string(Target.getName());
 
   // Emit the shared table of differential lists.
-  OS << "extern const MCPhysReg " << TargetName << "RegDiffLists[] = {\n";
+  OS << "extern const int16_t " << TargetName << "RegDiffLists[] = {\n";
   DiffSeqs.emit(OS, printDiff16);
   OS << "};\n\n";
 
   // Emit the shared table of regunit lane mask sequences.
   OS << "extern const LaneBitmask " << TargetName << "LaneMaskLists[] = {\n";
   LaneMaskSeqs.emit(OS, printMask, "LaneBitmask::getAll()");
   OS << "};\n\n";
 
#endif
 
   OS << "extern const MCRegisterDesc " << TargetName
      << "RegDesc[] = { // Descriptors\n";
   OS << "  { " << RegStrings.get("") << ", 0, 0, 0, 0, 0 },\n";
 
   // Emit the register descriptors now.
   i = 0;
   for (const auto &Reg : Regs) {
+    unsigned FirstRU = Reg.getNativeRegUnits().find_first();
+    unsigned Offset = DiffSeqs.get(RegUnitLists[i]);
+    // The value must be kept in sync with MCRegisterInfo.h.
+    constexpr unsigned RegUnitBits = 12;
+    assert(isUInt<RegUnitBits>(FirstRU) && "Too many regunits");
+    assert(isUInt<32 - RegUnitBits>(Offset) && "Offset is too big");
     OS << "  { " << RegStrings.get(std::string(Reg.getName())) << ", "
        << DiffSeqs.get(SubRegLists[i]) << ", " << DiffSeqs.get(SuperRegLists[i])
        << ", " << SubRegIdxSeqs.get(SubRegIdxLists[i]) << ", "
-       << (DiffSeqs.get(RegUnitLists[i]) * 16 + RegUnitInitScale[i]) << ", "
+       << (Offset << RegUnitBits | FirstRU) << ", "
        << LaneMaskSeqs.get(RegUnitLaneMasks[i]) << " },\n";
     ++i;
   }
   OS << "};\n\n";      // End of register descriptors...
 
   // Emit the table of register unit roots. Each regunit has one or two root
   // registers.
   OS << "extern const MCPhysReg " << TargetName << "RegUnitRoots[][2] = {\n";
if (!BaseClasses.empty()) {
       OS << "  assert(Reg < sizeof(Mapping));\n";
       OS << "  return BaseClasses[Mapping[Reg]];\n";
       OS << "}\n";
     }
   }
 
   // Emit the constructor of the class...
   OS << "extern const MCRegisterDesc " << TargetName << "RegDesc[];\n";
-  OS << "extern const MCPhysReg " << TargetName << "RegDiffLists[];\n";
+  OS << "extern const int16_t " << TargetName << "RegDiffLists[];\n";
   OS << "extern const LaneBitmask " << TargetName << "LaneMaskLists[];\n";
   OS << "extern const char " << TargetName << "RegStrings[];\n";
   OS << "extern const char " << TargetName << "RegClassStrings[];\n";
   OS << "extern const MCPhysReg " << TargetName << "RegUnitRoots[][2];\n";
   OS << "extern const uint16_t " << TargetName << "SubRegIdxLists[];\n";
   OS << "extern const MCRegisterInfo::SubRegCoveredBits "
      << TargetName << "SubRegIdxRanges[];\n";
   OS << "extern const uint16_t " << TargetName << "RegEncodingTable[];\n";