[RISCV] Avoid Splitting MBB in RISCVExpandPseudo

[lenary]

Summary

Since the RISCVExpandPseudo pass has been split from
RISCVExpandAtomicPseudo pass, it would be nice to run the former as
early as possible (The latter has to be run as late as possible to
ensure correctness). Running earlier means we can reschedule these pairs
as we see fit.

Running earlier in the machine pass pipeline is good, but would mean
teaching many more passes about hasLabelMustBeEmitted. Splitting the
basic blocks also pessimises possible optimisations because some
optimisations are MBB-local, and others are disabled if the block has
its address taken (which is notionally what hasLabelMustBeEmitted
means).

This patch uses a new approach of setting the pre-instruction symbol on
the AUIPC instruction to a temporary symbol and referencing that. This
avoids splitting the basic block, but allows us to reference exactly the
instruction that we need to. Notionally, this approach seems more
correct because we do actually want to address a specific instruction.

This then allows the pass to be moved much earlier in the pass pipeline,
before both scheduling and register allocation. However, to do so we must
leave the MIR in SSA form (by not redefining registers), and so use a
virtual register for the intermediate value. By using this virtual register,
this pass now has to come before register allocation.

[luismarques]

This is looking quite good. As Sam says, not splitting BBs is preferable, so while the D79635 issues could be resolved in other ways, if we can make it work like this then that's even better.
I suppose that this patch could also remove everything related to LabelMustBeEmitted.
If anyone wants to provide about the status of setPreInstrSymbol that would be helpful. Currently it's marked as FIXME: This is not fully implemented yet., although Sam indicates that it's being used by x86, so that may be just a holdover.

[rogfer01]

Thanks a lot @lenary, this seems a much better approach.

I am also in favour of removing LabelMustBeEmitted machinery which now is redundant (as only RISC-V was using it).

[lenary]

• Remove LabelMustBeEmitted API.

I have a question out to llvm-dev about the stability of PreInstrSymbol, but
given it's used by the retpoline pass, I am reasonably confident the
functionality is complete enough for our uses.

[lenary]

llvm/lib/Target/RISCV/RISCVExpandPseudoInsts.cpp

74–76

Oh this loop can be simplifed - though I'm not sure I should be incrementing MBBI if we've inserted new instructions using BuildMI, which I think also increments the iterator automatically. Guidance here would be helpful.

[luismarques]

LGTM. Nice!

llvm/lib/Target/RISCV/RISCVExpandPseudoInsts.cpp

74–76

Maybe I misunderstood your issue, but I think the increment is correct:

1. It is what AArch64 does.
2. If you have two consecutive pseudo-instructions this code expands them both, so you aren't skipping over the second by performing the explicit increment.
3. You don't want to iterate over the expanded instructions, to recursively expand them, since we don't emit other pseudo-instructions in the expansions.

BTW, what simplification were you considering for this loop?

[asb]

This is a nice simplification, thanks. My only request before committing is to split out the RISCVTargetMachine to a separate pass, as that is logically distinct.

[lenary]

The RISCVTargetMachine changes are not distinct. You can only use virtual registers (ScratchReg) if you are before register allocation, as I understand it.

llvm/lib/Target/RISCV/RISCVExpandPseudoInsts.cpp

74–76

Turning it back into a for loop like:

for (MachineBasicBlock::iterator MBBI = MBB.begin(), E = MBB.end(); MBBI != E; MBBI++) {
  Modified |= expandMI(MBB, MBBI)
}

[asb]

Got it, thanks. In that case LGTM and please tweak the commit message (the last paragraph specifically) so it's clear that the two changes are interlinked.

[rogfer01]

Hi @lenary,

I'm sorry to report that this approach is still problematic. In some of the testcases of the LLVM test-suite the pass when built with -fPIC -O3, pass Branch Probability Basic Block Placement decides to copy the AUIPC / LD pair but the copies carry around the same symbol. This fails with:

fatal error: error in backend: invalid symbol redefinition

I've seen other testcases in which we trigger

fatal error: error in backend: could not find corresponding %pcrel_hi

bb.5.for.end55:
; predecessors: %bb.4

  renamable $x10 = AUIPC target-flags(riscv-got-hi) @perSCC, pre-instr-symbol <mcsymbol .Ltmp1>
  renamable $x10 = LD killed renamable $x10, target-flags(riscv-pcrel-lo) <mcsymbol .Ltmp1>
  renamable $x12 = LD killed renamable $x10, 0 :: (dereferenceable load 8 from @perSCC)
  PseudoCALL target-flags(riscv-plt) @SCCofVCG, <regmask $x1 $x3 $x4 $x8 $x9 $x18 $x19 $x20 $x21 $x22 $x23 $x24 $x25 $x26 $x27 $f8_d $f9_d $f18_d $f19_d $f20_d $f21_d $f22_d $f23_d $f24_d $f25_d $f26_d $f27_d $f8_f $f9_f $f18_f $f19_f $f20_f $f21_f and 6 more...>, implicit-def dead $x1, implicit undef $x10, implicit undef $x11, implicit $x12, implicit-def $x2

bb.6.if.then47:
; predecessors: %bb.3
  liveins: $x11
  SD $x0, killed renamable $x11, 0 :: (store 8 into %ir.4)
  renamable $x10 = AUIPC target-flags(riscv-got-hi) @perSCC, pre-instr-symbol <mcsymbol .Ltmp1>
  renamable $x10 = LD killed renamable $x10, target-flags(riscv-pcrel-lo) <mcsymbol .Ltmp1>
  renamable $x12 = LD killed renamable $x10, 0 :: (dereferenceable load 8 from @perSCC)
  PseudoCALL target-flags(riscv-plt) @SCCofVCG, <regmask $x1 $x3 $x4 $x8 $x9 $x18 $x19 $x20 $x21 $x22 $x23 $x24 $x25 $x26 $x27 $f8_d $f9_d $f18_d $f19_d $f20_d $f21_d $f22_d $f23_d $f24_d $f25_d $f26_d $f27_d $f8_f $f9_f $f18_f $f19_f $f20_f $f21_f and 6 more...>, implicit-def dead $x1, implicit undef $x10, implicit undef $x11, implicit $x12, implicit-def $x2

I don't have many ideas here. I understand the root cause is that we can't really copy the instruction (unless we're somehow able to create a new symbol for the copied AUIPC and map the reference of the copied LD to the new one). A specific pass that fixes these cases doesn't seem very appealing either.

Kind regards,

[rogfer01]

Would it make sense to still expand these constructs late (e.g. renaming RISCVExpandAtomicPseudoInsts.cpp into RISCVLateExpandPseudoInsts.cpp) still using PreInstrSymbol?

[efriedma]

Instructions have an isNotDuplicable property which can be used to stop various code duplication passes.

That said, it's not clear you want to stop the code duplication early, so you might want to move the expansion later anyway.

[lenary]

We're close to the branch date. I'm going to revert rG97106f9d80f6 and look at this again. Sound good?

[lenary]

Reverted in rG1d15bbb9d91

[nickdesaulniers]

fatal error: error in backend: could not find corresponding %pcrel_hi

We're also seeing this in our RISC-V Linux kernel builds: https://github.com/ClangBuiltLinux/linux/issues/1089

[lenary]

@rogfer01 @efriedma I would love this pass to happen early in the pipeline, so that auipc/l{d,w} can be rescheduled or shared. Maybe the easy solution is to mark the auipc as not duplicatable (as that's the thing you want to share, anyway), so the l{d,w} can be duplicated if necessary?

The other solution is to work out what's going on with duplicating symbols and setPreInstrSymbol and try something there, but the issue is if a MBB has been duplicated, I cannot RAUW the symbol, because some of the instructions should refer to the old symbol, and some to the new. I think this is what Roger is alluding to with the "fixup pass", but I think this would have to be done at duplication-time, not after.

I doubt I have time to investigate why this is going wrong any time soon, and especially not before the branch tomorrow.

[lenary]

@luismarques @asb @mundaym Please can one of you take over this patch if lowRISC intends to keep working on it? I do realise that we weren't sure what the scheduling/performance payoff would be from this work, and it was showing some pretty obscure bugs.

llvm/include/llvm/CodeGen/MachineBasicBlock.h

private:
 
   /// Indicate that this basic block is entered via an exception handler.
   bool IsEHPad = false;
 
   /// Indicate that this basic block is potentially the target of an indirect
   /// branch.
   bool AddressTaken = false;
 
-  /// Indicate that this basic block needs its symbol be emitted regardless of
-  /// whether the flow just falls-through to it.
-  bool LabelMustBeEmitted = false;
-
   /// Indicate that this basic block is the entry block of an EH scope, i.e.,
   /// the block that used to have a catchpad or cleanuppad instruction in the
   /// LLVM IR.
   bool IsEHScopeEntry = false;
 
   /// Indicate that this basic block is the entry block of an EH funclet.
   bool IsEHFuncletEntry = false;
 
public:
 
   /// Test whether this block is potentially the target of an indirect branch.
   bool hasAddressTaken() const { return AddressTaken; }
 
   /// Set this block to reflect that it potentially is the target of an indirect
   /// branch.
   void setHasAddressTaken() { AddressTaken = true; }
 
-  /// Test whether this block must have its label emitted.
-  bool hasLabelMustBeEmitted() const { return LabelMustBeEmitted; }
-
-  /// Set this block to reflect that, regardless how we flow to it, we need
-  /// its label be emitted.
-  void setLabelMustBeEmitted() { LabelMustBeEmitted = true; }
-
   /// Return the MachineFunction containing this basic block.
   const MachineFunction *getParent() const { return xParent; }
   MachineFunction *getParent() { return xParent; }
 
   using instr_iterator = Instructions::iterator;
   using const_instr_iterator = Instructions::const_iterator;
   using reverse_instr_iterator = Instructions::reverse_iterator;
   using const_reverse_instr_iterator = Instructions::const_reverse_iterator;

llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp

if (isVerbose()) {
     }
 
     assert(MLI != nullptr && "MachineLoopInfo should has been computed");
     emitBasicBlockLoopComments(MBB, MLI, *this);
   }
 
   if (MBB.pred_empty() ||
       (!MF->hasBBLabels() && isBlockOnlyReachableByFallthrough(&MBB) &&
-       !MBB.isEHFuncletEntry() && !MBB.hasLabelMustBeEmitted())) {
+       !MBB.isEHFuncletEntry())) {
     if (isVerbose()) {
       // NOTE: Want this comment at start of line, don't emit with AddComment.
       OutStreamer->emitRawComment(" %bb." + Twine(MBB.getNumber()) + ":",
                                   false);
     }
   } else {
-    if (isVerbose() && MBB.hasLabelMustBeEmitted()) {
-      OutStreamer->AddComment("Label of block must be emitted");
-    }
     // Switch to a new section if this basic block must begin a section.
     if (MBB.isBeginSection()) {
       OutStreamer->SwitchSection(
           getObjFileLowering().getSectionForMachineBasicBlock(MF->getFunction(),
                                                               MBB, TM));
       CurrentSectionBeginSym = MBB.getSymbol();
     }
     OutStreamer->emitLabel(MBB.getSymbol());

llvm/lib/Target/RISCV/RISCVExpandPseudoInsts.cpp

 
 #include "RISCV.h"
 #include "RISCVInstrInfo.h"
 #include "RISCVTargetMachine.h"
 
 #include "llvm/CodeGen/LivePhysRegs.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
+#include "llvm/MC/MCContext.h"
 
 using namespace llvm;
 
 #define RISCV_EXPAND_PSEUDO_NAME "RISCV pseudo instruction expansion pass"
 
 namespace {
 
 class RISCVExpandPseudo : public MachineFunctionPass {
 public:
   const RISCVInstrInfo *TII;
   static char ID;
 
   RISCVExpandPseudo() : MachineFunctionPass(ID) {
     initializeRISCVExpandPseudoPass(*PassRegistry::getPassRegistry());
   }
 
   bool runOnMachineFunction(MachineFunction &MF) override;
 
   StringRef getPassName() const override { return RISCV_EXPAND_PSEUDO_NAME; }
 
 private:
   bool expandMBB(MachineBasicBlock &MBB);
-  bool expandMI(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
-                MachineBasicBlock::iterator &NextMBBI);
+  bool expandMI(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI);
   bool expandAuipcInstPair(MachineBasicBlock &MBB,
                            MachineBasicBlock::iterator MBBI,
-                           MachineBasicBlock::iterator &NextMBBI,
                            unsigned FlagsHi, unsigned SecondOpcode);
   bool expandLoadLocalAddress(MachineBasicBlock &MBB,
-                              MachineBasicBlock::iterator MBBI,
-                              MachineBasicBlock::iterator &NextMBBI);
+                              MachineBasicBlock::iterator MBBI);
   bool expandLoadAddress(MachineBasicBlock &MBB,
-                         MachineBasicBlock::iterator MBBI,
-                         MachineBasicBlock::iterator &NextMBBI);
+                         MachineBasicBlock::iterator MBBI);
   bool expandLoadTLSIEAddress(MachineBasicBlock &MBB,
-                              MachineBasicBlock::iterator MBBI,
-                              MachineBasicBlock::iterator &NextMBBI);
+                              MachineBasicBlock::iterator MBBI);
   bool expandLoadTLSGDAddress(MachineBasicBlock &MBB,
-                              MachineBasicBlock::iterator MBBI,
-                              MachineBasicBlock::iterator &NextMBBI);
+                              MachineBasicBlock::iterator MBBI);
 };
 
 char RISCVExpandPseudo::ID = 0;
 
 bool RISCVExpandPseudo::runOnMachineFunction(MachineFunction &MF) {
   TII = static_cast<const RISCVInstrInfo *>(MF.getSubtarget().getInstrInfo());
   bool Modified = false;
   for (auto &MBB : MF)
     Modified |= expandMBB(MBB);
   return Modified;
 }
 
 bool RISCVExpandPseudo::expandMBB(MachineBasicBlock &MBB) {
   bool Modified = false;
 
   MachineBasicBlock::iterator MBBI = MBB.begin(), E = MBB.end();
   while (MBBI != E) {
     MachineBasicBlock::iterator NMBBI = std::next(MBBI);
-    Modified |= expandMI(MBB, MBBI, NMBBI);
+    Modified |= expandMI(MBB, MBBI);
     MBBI = NMBBI;
   }
 
   return Modified;
 }
 
 bool RISCVExpandPseudo::expandMI(MachineBasicBlock &MBB,
-                                 MachineBasicBlock::iterator MBBI,
-                                 MachineBasicBlock::iterator &NextMBBI) {
+                                 MachineBasicBlock::iterator MBBI) {
   switch (MBBI->getOpcode()) {
   case RISCV::PseudoLLA:
-    return expandLoadLocalAddress(MBB, MBBI, NextMBBI);
+    return expandLoadLocalAddress(MBB, MBBI);
   case RISCV::PseudoLA:
-    return expandLoadAddress(MBB, MBBI, NextMBBI);
+    return expandLoadAddress(MBB, MBBI);
   case RISCV::PseudoLA_TLS_IE:
-    return expandLoadTLSIEAddress(MBB, MBBI, NextMBBI);
+    return expandLoadTLSIEAddress(MBB, MBBI);
   case RISCV::PseudoLA_TLS_GD:
-    return expandLoadTLSGDAddress(MBB, MBBI, NextMBBI);
+    return expandLoadTLSGDAddress(MBB, MBBI);
   }
 
   return false;
 }
 
-bool RISCVExpandPseudo::expandAuipcInstPair(
-    MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
-    MachineBasicBlock::iterator &NextMBBI, unsigned FlagsHi,
+bool RISCVExpandPseudo::expandAuipcInstPair(MachineBasicBlock &MBB,
+                                            MachineBasicBlock::iterator MBBI,
+                                            unsigned FlagsHi,
                                             unsigned SecondOpcode) {
   MachineFunction *MF = MBB.getParent();
   MachineInstr &MI = *MBBI;
   DebugLoc DL = MI.getDebugLoc();
 
   Register DestReg = MI.getOperand(0).getReg();
-  const MachineOperand &Symbol = MI.getOperand(1);
-
-  MachineBasicBlock *NewMBB = MF->CreateMachineBasicBlock(MBB.getBasicBlock());
+  Register ScratchReg =
+      MF->getRegInfo().createVirtualRegister(&RISCV::GPRRegClass);
 
-  // Tell AsmPrinter that we unconditionally want the symbol of this label to be
-  // emitted.
-  NewMBB->setLabelMustBeEmitted();
-
-  MF->insert(++MBB.getIterator(), NewMBB);
-
-  BuildMI(NewMBB, DL, TII->get(RISCV::AUIPC), DestReg)
-      .addDisp(Symbol, 0, FlagsHi);
-  BuildMI(NewMBB, DL, TII->get(SecondOpcode), DestReg)
-      .addReg(DestReg)
-      .addMBB(NewMBB, RISCVII::MO_PCREL_LO);
-
-  // Move all the rest of the instructions to NewMBB.
-  NewMBB->splice(NewMBB->end(), &MBB, std::next(MBBI), MBB.end());
-  // Update machine-CFG edges.
-  NewMBB->transferSuccessorsAndUpdatePHIs(&MBB);
-  // Make the original basic block fall-through to the new.
-  MBB.addSuccessor(NewMBB);
-
-  // Make sure live-ins are correctly attached to this new basic block.
-  LivePhysRegs LiveRegs;
-  computeAndAddLiveIns(LiveRegs, *NewMBB);
+  MachineOperand &Symbol = MI.getOperand(1);
+  Symbol.setTargetFlags(FlagsHi);
+  MCSymbol *AUIPCSymbol = MF->getContext().createTempSymbol(false);
+
+  MachineInstr *MIAUIPC =
+      BuildMI(MBB, MBBI, DL, TII->get(RISCV::AUIPC), ScratchReg).add(Symbol);
+  MIAUIPC->setPreInstrSymbol(*MF, AUIPCSymbol);
+
+  BuildMI(MBB, MBBI, DL, TII->get(SecondOpcode), DestReg)
+      .addReg(ScratchReg)
+      .addSym(AUIPCSymbol, RISCVII::MO_PCREL_LO);
 
-  NextMBBI = MBB.end();
   MI.eraseFromParent();
   return true;
 }
 
 bool RISCVExpandPseudo::expandLoadLocalAddress(
-    MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
-    MachineBasicBlock::iterator &NextMBBI) {
-  return expandAuipcInstPair(MBB, MBBI, NextMBBI, RISCVII::MO_PCREL_HI,
-                             RISCV::ADDI);
+    MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI) {
+  return expandAuipcInstPair(MBB, MBBI, RISCVII::MO_PCREL_HI, RISCV::ADDI);
 }
 
-bool RISCVExpandPseudo::expandLoadAddress(
-    MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
-    MachineBasicBlock::iterator &NextMBBI) {
+bool RISCVExpandPseudo::expandLoadAddress(MachineBasicBlock &MBB,
+                                          MachineBasicBlock::iterator MBBI) {
   MachineFunction *MF = MBB.getParent();
 
   unsigned SecondOpcode;
   unsigned FlagsHi;
   if (MF->getTarget().isPositionIndependent()) {
     const auto &STI = MF->getSubtarget<RISCVSubtarget>();
     SecondOpcode = STI.is64Bit() ? RISCV::LD : RISCV::LW;
     FlagsHi = RISCVII::MO_GOT_HI;
   } else {
     SecondOpcode = RISCV::ADDI;
     FlagsHi = RISCVII::MO_PCREL_HI;
   }
-  return expandAuipcInstPair(MBB, MBBI, NextMBBI, FlagsHi, SecondOpcode);
+  return expandAuipcInstPair(MBB, MBBI, FlagsHi, SecondOpcode);
 }
 
 bool RISCVExpandPseudo::expandLoadTLSIEAddress(
-    MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
-    MachineBasicBlock::iterator &NextMBBI) {
+    MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI) {
   MachineFunction *MF = MBB.getParent();
 
   const auto &STI = MF->getSubtarget<RISCVSubtarget>();
   unsigned SecondOpcode = STI.is64Bit() ? RISCV::LD : RISCV::LW;
-  return expandAuipcInstPair(MBB, MBBI, NextMBBI, RISCVII::MO_TLS_GOT_HI,
-                             SecondOpcode);
+  return expandAuipcInstPair(MBB, MBBI, RISCVII::MO_TLS_GOT_HI, SecondOpcode);
 }
 
 bool RISCVExpandPseudo::expandLoadTLSGDAddress(
-    MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
-    MachineBasicBlock::iterator &NextMBBI) {
-  return expandAuipcInstPair(MBB, MBBI, NextMBBI, RISCVII::MO_TLS_GD_HI,
-                             RISCV::ADDI);
+    MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI) {
+  return expandAuipcInstPair(MBB, MBBI, RISCVII::MO_TLS_GD_HI, RISCV::ADDI);
 }
 
 } // end of anonymous namespace
 
 INITIALIZE_PASS(RISCVExpandPseudo, "riscv-expand-pseudo",
                 RISCV_EXPAND_PSEUDO_NAME, false, false)
 namespace llvm {
 
 FunctionPass *createRISCVExpandPseudoPass() { return new RISCVExpandPseudo(); }
 
 } // end of namespace llvm

llvm/lib/Target/RISCV/RISCVMCInstLower.cpp

case MachineOperand::MO_BlockAddress:
     break;
   case MachineOperand::MO_ExternalSymbol:
     MCOp = lowerSymbolOperand(
         MO, AP.GetExternalSymbolSymbol(MO.getSymbolName()), AP);
     break;
   case MachineOperand::MO_ConstantPoolIndex:
     MCOp = lowerSymbolOperand(MO, AP.GetCPISymbol(MO.getIndex()), AP);
     break;
+  case MachineOperand::MO_MCSymbol:
+    MCOp = lowerSymbolOperand(MO, MO.getMCSymbol(), AP);
+    break;
   }
   return true;
 }
 
 void llvm::LowerRISCVMachineInstrToMCInst(const MachineInstr *MI, MCInst &OutMI,
                                           const AsmPrinter &AP) {
   OutMI.setOpcode(MI->getOpcode());
 
   for (const MachineOperand &MO : MI->operands()) {
     MCOperand MCOp;
     if (LowerRISCVMachineOperandToMCOperand(MO, MCOp, AP))
       OutMI.addOperand(MCOp);
   }
 }

llvm/lib/Target/RISCV/RISCVTargetMachine.cpp

bool RISCVPassConfig::addGlobalInstructionSelect() {
   return false;
 }
 
 void RISCVPassConfig::addPreSched2() {}
 
 void RISCVPassConfig::addPreEmitPass() { addPass(&BranchRelaxationPassID); }
 
 void RISCVPassConfig::addPreEmitPass2() {
-  addPass(createRISCVExpandPseudoPass());
   // Schedule the expansion of AMOs at the last possible moment, avoiding the
   // possibility for other passes to break the requirements for forward
   // progress in the LR/SC block.
   addPass(createRISCVExpandAtomicPseudoPass());
 }
 
 void RISCVPassConfig::addPreRegAlloc() {
+  addPass(createRISCVExpandPseudoPass());
   addPass(createRISCVMergeBaseOffsetOptPass());
 }

llvm/test/CodeGen/RISCV/codemodel-lowering.ll

 ; RV32I-SMALL-LABEL: lower_global:
 ; RV32I-SMALL:       # %bb.0:
 ; RV32I-SMALL-NEXT:    lui a0, %hi(G)
 ; RV32I-SMALL-NEXT:    lw a0, %lo(G)(a0)
 ; RV32I-SMALL-NEXT:    ret
 ;
 ; RV32I-MEDIUM-LABEL: lower_global:
 ; RV32I-MEDIUM:       # %bb.0:
-; RV32I-MEDIUM-NEXT:  .LBB0_1: # Label of block must be emitted
+; RV32I-MEDIUM-NEXT:  .Ltmp0:
 ; RV32I-MEDIUM-NEXT:    auipc a0, %pcrel_hi(G)
-; RV32I-MEDIUM-NEXT:    addi a0, a0, %pcrel_lo(.LBB0_1)
+; RV32I-MEDIUM-NEXT:    addi a0, a0, %pcrel_lo(.Ltmp0)
 ; RV32I-MEDIUM-NEXT:    lw a0, 0(a0)
 ; RV32I-MEDIUM-NEXT:    ret
   %1 = load volatile i32, i32* @G
   ret i32 %1
 }
 
 ; Check lowering of blockaddresses
 
 @addr = global i8* null
 
 define void @lower_blockaddress() nounwind {
 ; RV32I-SMALL-LABEL: lower_blockaddress:
 ; RV32I-SMALL:       # %bb.0:
 ; RV32I-SMALL-NEXT:    lui a0, %hi(addr)
 ; RV32I-SMALL-NEXT:    addi a1, zero, 1
 ; RV32I-SMALL-NEXT:    sw a1, %lo(addr)(a0)
 ; RV32I-SMALL-NEXT:    ret
 ;
 ; RV32I-MEDIUM-LABEL: lower_blockaddress:
 ; RV32I-MEDIUM:       # %bb.0:
-; RV32I-MEDIUM-NEXT:  .LBB1_1: # Label of block must be emitted
+; RV32I-MEDIUM-NEXT:  .Ltmp1:
 ; RV32I-MEDIUM-NEXT:    auipc a0, %pcrel_hi(addr)
-; RV32I-MEDIUM-NEXT:    addi a0, a0, %pcrel_lo(.LBB1_1)
+; RV32I-MEDIUM-NEXT:    addi a0, a0, %pcrel_lo(.Ltmp1)
 ; RV32I-MEDIUM-NEXT:    addi a1, zero, 1
 ; RV32I-MEDIUM-NEXT:    sw a1, 0(a0)
 ; RV32I-MEDIUM-NEXT:    ret
   store volatile i8* blockaddress(@lower_blockaddress, %block), i8** @addr
   ret void
 
 block:
   unreachable

 ; RV32I-SMALL-NEXT:    lw ra, 12(sp)
 ; RV32I-SMALL-NEXT:    addi sp, sp, 16
 ; RV32I-SMALL-NEXT:    ret
 ;
 ; RV32I-MEDIUM-LABEL: lower_blockaddress_displ:
 ; RV32I-MEDIUM:       # %bb.0: # %entry
 ; RV32I-MEDIUM-NEXT:    addi sp, sp, -16
 ; RV32I-MEDIUM-NEXT:    sw ra, 12(sp)
-; RV32I-MEDIUM-NEXT:  .LBB2_5: # %entry
-; RV32I-MEDIUM-NEXT:    # Label of block must be emitted
-; RV32I-MEDIUM-NEXT:    auipc a1, %pcrel_hi(.Ltmp0)
-; RV32I-MEDIUM-NEXT:    addi a1, a1, %pcrel_lo(.LBB2_5)
+; RV32I-MEDIUM-NEXT:  .Ltmp2:
+; RV32I-MEDIUM-NEXT:    auipc a1, %pcrel_hi(.Ltmp3)
+; RV32I-MEDIUM-NEXT:    addi a1, a1, %pcrel_lo(.Ltmp2)
 ; RV32I-MEDIUM-NEXT:    addi a2, zero, 101
 ; RV32I-MEDIUM-NEXT:    sw a1, 8(sp)
 ; RV32I-MEDIUM-NEXT:    blt a0, a2, .LBB2_3
 ; RV32I-MEDIUM-NEXT:  # %bb.1: # %if.then
 ; RV32I-MEDIUM-NEXT:    lw a0, 8(sp)
 ; RV32I-MEDIUM-NEXT:    jr a0
-; RV32I-MEDIUM-NEXT:  .Ltmp0: # Block address taken
+; RV32I-MEDIUM-NEXT:  .Ltmp3: # Block address taken
 ; RV32I-MEDIUM-NEXT:  .LBB2_2: # %return
 ; RV32I-MEDIUM-NEXT:    addi a0, zero, 4
 ; RV32I-MEDIUM-NEXT:    j .LBB2_4
 ; RV32I-MEDIUM-NEXT:  .LBB2_3: # %return.clone
 ; RV32I-MEDIUM-NEXT:    addi a0, zero, 3
 ; RV32I-MEDIUM-NEXT:  .LBB2_4: # %.split
 ; RV32I-MEDIUM-NEXT:    lw ra, 12(sp)
 ; RV32I-MEDIUM-NEXT:    addi sp, sp, 16

 ; RV32I-SMALL-NEXT:    flw ft0, %lo(.LCPI3_0)(a1)
 ; RV32I-SMALL-NEXT:    fmv.w.x ft1, a0
 ; RV32I-SMALL-NEXT:    fadd.s ft0, ft1, ft0
 ; RV32I-SMALL-NEXT:    fmv.x.w a0, ft0
 ; RV32I-SMALL-NEXT:    ret
 ;
 ; RV32I-MEDIUM-LABEL: lower_constantpool:
 ; RV32I-MEDIUM:       # %bb.0:
-; RV32I-MEDIUM-NEXT:  .LBB3_1: # Label of block must be emitted
+; RV32I-MEDIUM-NEXT:  .Ltmp4:
 ; RV32I-MEDIUM-NEXT:    auipc a1, %pcrel_hi(.LCPI3_0)
-; RV32I-MEDIUM-NEXT:    addi a1, a1, %pcrel_lo(.LBB3_1)
+; RV32I-MEDIUM-NEXT:    addi a1, a1, %pcrel_lo(.Ltmp4)
 ; RV32I-MEDIUM-NEXT:    flw ft0, 0(a1)
 ; RV32I-MEDIUM-NEXT:    fmv.w.x ft1, a0
 ; RV32I-MEDIUM-NEXT:    fadd.s ft0, ft1, ft0
 ; RV32I-MEDIUM-NEXT:    fmv.x.w a0, ft0
 ; RV32I-MEDIUM-NEXT:    ret
   %1 = fadd float %a, 1.0
   ret float %1
 }

llvm/test/CodeGen/RISCV/mir-target-flags.ll

 
 define i32 @caller(i32 %a) nounwind {
 ; RV32-SMALL-LABEL: name: caller
 ; RV32-SMALL:      target-flags(riscv-hi) @g_e
 ; RV32-SMALL-NEXT: target-flags(riscv-lo) @g_e
 ; RV32-SMALL-NEXT: target-flags(riscv-hi) @g_i
 ; RV32-SMALL-NEXT: target-flags(riscv-lo) @g_i
 ; RV32-SMALL:      target-flags(riscv-tls-got-hi) @t_un
-; RV32-SMALL-NEXT: target-flags(riscv-pcrel-lo) %bb.1
+; RV32-SMALL-NEXT: target-flags(riscv-pcrel-lo)
 ; RV32-SMALL:      target-flags(riscv-tls-got-hi) @t_ld
-; RV32-SMALL-NEXT: target-flags(riscv-pcrel-lo) %bb.2
+; RV32-SMALL-NEXT: target-flags(riscv-pcrel-lo)
 ; RV32-SMALL:      target-flags(riscv-tls-got-hi) @t_ie
-; RV32-SMALL-NEXT: target-flags(riscv-pcrel-lo) %bb.3
+; RV32-SMALL-NEXT: target-flags(riscv-pcrel-lo)
 ; RV32-SMALL:      target-flags(riscv-tprel-hi) @t_le
 ; RV32-SMALL-NEXT: target-flags(riscv-tprel-add) @t_le
 ; RV32-SMALL-NEXT: target-flags(riscv-tprel-lo) @t_le
 ; RV32-SMALL:      target-flags(riscv-call) @callee
 ;
 ; RV32-MED-LABEL: name: caller
 ; RV32-MED:      target-flags(riscv-got-hi) @g_e
-; RV32-MED-NEXT: target-flags(riscv-pcrel-lo) %bb.1
+; RV32-MED-NEXT: target-flags(riscv-pcrel-lo)
 ; RV32-MED:      target-flags(riscv-pcrel-hi) @g_i
-; RV32-MED-NEXT: target-flags(riscv-pcrel-lo) %bb.2
+; RV32-MED-NEXT: target-flags(riscv-pcrel-lo)
 ; RV32-MED:      target-flags(riscv-tls-gd-hi) @t_un
-; RV32-MED-NEXT: target-flags(riscv-pcrel-lo) %bb.3
-; RV32-MED-NEXT: target-flags(riscv-plt) &__tls_get_addr
+; RV32-MED-NEXT: target-flags(riscv-pcrel-lo)
+; RV32-MED:      target-flags(riscv-plt) &__tls_get_addr
 ; RV32-MED:      target-flags(riscv-tls-gd-hi) @t_ld
-; RV32-MED-NEXT: target-flags(riscv-pcrel-lo) %bb.4
-; RV32-MED-NEXT: target-flags(riscv-plt) &__tls_get_addr
+; RV32-MED-NEXT: target-flags(riscv-pcrel-lo)
+; RV32-MED:      target-flags(riscv-plt) &__tls_get_addr
 ; RV32-MED:      target-flags(riscv-tls-got-hi) @t_ie
-; RV32-MED-NEXT: target-flags(riscv-pcrel-lo) %bb.5
+; RV32-MED-NEXT: target-flags(riscv-pcrel-lo)
 ; RV32-MED:      target-flags(riscv-tprel-hi) @t_le
 ; RV32-MED-NEXT: target-flags(riscv-tprel-add) @t_le
 ; RV32-MED-NEXT: target-flags(riscv-tprel-lo) @t_le
 ; RV32-MED:      target-flags(riscv-plt) @callee
 ;
   %b = load i32, i32* @g_e
   %c = load i32, i32* @g_i
   %d = load i32, i32* @t_un

llvm/test/CodeGen/RISCV/pic-models.ll

 ; RV32-STATIC-LABEL: f1:
 ; RV32-STATIC:       # %bb.0: # %entry
 ; RV32-STATIC-NEXT:    lui a0, %hi(external_var)
 ; RV32-STATIC-NEXT:    addi a0, a0, %lo(external_var)
 ; RV32-STATIC-NEXT:    ret
 ;
 ; RV32-PIC-LABEL: f1:
 ; RV32-PIC:       # %bb.0: # %entry
-; RV32-PIC-NEXT:  .LBB0_1: # %entry
-; RV32-PIC-NEXT:    # Label of block must be emitted
+; RV32-PIC-NEXT:  .Ltmp0:
 ; RV32-PIC-NEXT:    auipc a0, %got_pcrel_hi(external_var)
-; RV32-PIC-NEXT:    lw a0, %pcrel_lo(.LBB0_1)(a0)
+; RV32-PIC-NEXT:    lw a0, %pcrel_lo(.Ltmp0)(a0)
 ; RV32-PIC-NEXT:    ret
 ;
 ; RV64-STATIC-LABEL: f1:
 ; RV64-STATIC:       # %bb.0: # %entry
 ; RV64-STATIC-NEXT:    lui a0, %hi(external_var)
 ; RV64-STATIC-NEXT:    addi a0, a0, %lo(external_var)
 ; RV64-STATIC-NEXT:    ret
 ;
 ; RV64-PIC-LABEL: f1:
 ; RV64-PIC:       # %bb.0: # %entry
-; RV64-PIC-NEXT:  .LBB0_1: # %entry
-; RV64-PIC-NEXT:    # Label of block must be emitted
+; RV64-PIC-NEXT:  .Ltmp0:
 ; RV64-PIC-NEXT:    auipc a0, %got_pcrel_hi(external_var)
-; RV64-PIC-NEXT:    ld a0, %pcrel_lo(.LBB0_1)(a0)
+; RV64-PIC-NEXT:    ld a0, %pcrel_lo(.Ltmp0)(a0)
 ; RV64-PIC-NEXT:    ret
 entry:
   ret i32* @external_var
 }
 
 
 ; internal address
 
 define i32* @f2() nounwind {
 ; RV32-STATIC-LABEL: f2:
 ; RV32-STATIC:       # %bb.0: # %entry
 ; RV32-STATIC-NEXT:    lui a0, %hi(internal_var)
 ; RV32-STATIC-NEXT:    addi a0, a0, %lo(internal_var)
 ; RV32-STATIC-NEXT:    ret
 ;
 ; RV32-PIC-LABEL: f2:
 ; RV32-PIC:       # %bb.0: # %entry
-; RV32-PIC-NEXT:  .LBB1_1: # %entry
-; RV32-PIC-NEXT:    # Label of block must be emitted
+; RV32-PIC-NEXT:  .Ltmp1:
 ; RV32-PIC-NEXT:    auipc a0, %pcrel_hi(internal_var)
-; RV32-PIC-NEXT:    addi a0, a0, %pcrel_lo(.LBB1_1)
+; RV32-PIC-NEXT:    addi a0, a0, %pcrel_lo(.Ltmp1)
 ; RV32-PIC-NEXT:    ret
 ;
 ; RV64-STATIC-LABEL: f2:
 ; RV64-STATIC:       # %bb.0: # %entry
 ; RV64-STATIC-NEXT:    lui a0, %hi(internal_var)
 ; RV64-STATIC-NEXT:    addi a0, a0, %lo(internal_var)
 ; RV64-STATIC-NEXT:    ret
 ;
 ; RV64-PIC-LABEL: f2:
 ; RV64-PIC:       # %bb.0: # %entry
-; RV64-PIC-NEXT:  .LBB1_1: # %entry
-; RV64-PIC-NEXT:    # Label of block must be emitted
+; RV64-PIC-NEXT:  .Ltmp1:
 ; RV64-PIC-NEXT:    auipc a0, %pcrel_hi(internal_var)
-; RV64-PIC-NEXT:    addi a0, a0, %pcrel_lo(.LBB1_1)
+; RV64-PIC-NEXT:    addi a0, a0, %pcrel_lo(.Ltmp1)
 ; RV64-PIC-NEXT:    ret
 entry:
   ret i32* @internal_var
 }

llvm/test/CodeGen/RISCV/tls-models.ll

 
 ; No model specified
 
 define i32* @f1() nounwind {
 ; RV32-PIC-LABEL: f1:
 ; RV32-PIC:       # %bb.0: # %entry
 ; RV32-PIC-NEXT:    addi sp, sp, -16
 ; RV32-PIC-NEXT:    sw ra, 12(sp)
-; RV32-PIC-NEXT:  .LBB0_1: # %entry
-; RV32-PIC-NEXT:    # Label of block must be emitted
+; RV32-PIC-NEXT:  .Ltmp0:
 ; RV32-PIC-NEXT:    auipc a0, %tls_gd_pcrel_hi(unspecified)
-; RV32-PIC-NEXT:    addi a0, a0, %pcrel_lo(.LBB0_1)
+; RV32-PIC-NEXT:    addi a0, a0, %pcrel_lo(.Ltmp0)
 ; RV32-PIC-NEXT:    call __tls_get_addr@plt
 ; RV32-PIC-NEXT:    lw ra, 12(sp)
 ; RV32-PIC-NEXT:    addi sp, sp, 16
 ; RV32-PIC-NEXT:    ret
 ;
 ; RV64-PIC-LABEL: f1:
 ; RV64-PIC:       # %bb.0: # %entry
 ; RV64-PIC-NEXT:    addi sp, sp, -16
 ; RV64-PIC-NEXT:    sd ra, 8(sp)
-; RV64-PIC-NEXT:  .LBB0_1: # %entry
-; RV64-PIC-NEXT:    # Label of block must be emitted
+; RV64-PIC-NEXT:  .Ltmp0:
 ; RV64-PIC-NEXT:    auipc a0, %tls_gd_pcrel_hi(unspecified)
-; RV64-PIC-NEXT:    addi a0, a0, %pcrel_lo(.LBB0_1)
+; RV64-PIC-NEXT:    addi a0, a0, %pcrel_lo(.Ltmp0)
 ; RV64-PIC-NEXT:    call __tls_get_addr@plt
 ; RV64-PIC-NEXT:    ld ra, 8(sp)
 ; RV64-PIC-NEXT:    addi sp, sp, 16
 ; RV64-PIC-NEXT:    ret
 ;
 ; RV32-NOPIC-LABEL: f1:
 ; RV32-NOPIC:       # %bb.0: # %entry
-; RV32-NOPIC-NEXT:  .LBB0_1: # %entry
-; RV32-NOPIC-NEXT:    # Label of block must be emitted
+; RV32-NOPIC-NEXT:  .Ltmp0:
 ; RV32-NOPIC-NEXT:    auipc a0, %tls_ie_pcrel_hi(unspecified)
-; RV32-NOPIC-NEXT:    lw a0, %pcrel_lo(.LBB0_1)(a0)
+; RV32-NOPIC-NEXT:    lw a0, %pcrel_lo(.Ltmp0)(a0)
 ; RV32-NOPIC-NEXT:    add a0, a0, tp
 ; RV32-NOPIC-NEXT:    ret
 ;
 ; RV64-NOPIC-LABEL: f1:
 ; RV64-NOPIC:       # %bb.0: # %entry
-; RV64-NOPIC-NEXT:  .LBB0_1: # %entry
-; RV64-NOPIC-NEXT:    # Label of block must be emitted
+; RV64-NOPIC-NEXT:  .Ltmp0:
 ; RV64-NOPIC-NEXT:    auipc a0, %tls_ie_pcrel_hi(unspecified)
-; RV64-NOPIC-NEXT:    ld a0, %pcrel_lo(.LBB0_1)(a0)
+; RV64-NOPIC-NEXT:    ld a0, %pcrel_lo(.Ltmp0)(a0)
 ; RV64-NOPIC-NEXT:    add a0, a0, tp
 ; RV64-NOPIC-NEXT:    ret
 entry:
   ret i32* @unspecified
 }
 
 
 ; localdynamic specified
 
 define i32* @f2() nounwind {
 ; RV32-PIC-LABEL: f2:
 ; RV32-PIC:       # %bb.0: # %entry
 ; RV32-PIC-NEXT:    addi sp, sp, -16
 ; RV32-PIC-NEXT:    sw ra, 12(sp)
-; RV32-PIC-NEXT:  .LBB1_1: # %entry
-; RV32-PIC-NEXT:    # Label of block must be emitted
+; RV32-PIC-NEXT:  .Ltmp1:
 ; RV32-PIC-NEXT:    auipc a0, %tls_gd_pcrel_hi(ld)
-; RV32-PIC-NEXT:    addi a0, a0, %pcrel_lo(.LBB1_1)
+; RV32-PIC-NEXT:    addi a0, a0, %pcrel_lo(.Ltmp1)
 ; RV32-PIC-NEXT:    call __tls_get_addr@plt
 ; RV32-PIC-NEXT:    lw ra, 12(sp)
 ; RV32-PIC-NEXT:    addi sp, sp, 16
 ; RV32-PIC-NEXT:    ret
 ;
 ; RV64-PIC-LABEL: f2:
 ; RV64-PIC:       # %bb.0: # %entry
 ; RV64-PIC-NEXT:    addi sp, sp, -16
 ; RV64-PIC-NEXT:    sd ra, 8(sp)
-; RV64-PIC-NEXT:  .LBB1_1: # %entry
-; RV64-PIC-NEXT:    # Label of block must be emitted
+; RV64-PIC-NEXT:  .Ltmp1:
 ; RV64-PIC-NEXT:    auipc a0, %tls_gd_pcrel_hi(ld)
-; RV64-PIC-NEXT:    addi a0, a0, %pcrel_lo(.LBB1_1)
+; RV64-PIC-NEXT:    addi a0, a0, %pcrel_lo(.Ltmp1)
 ; RV64-PIC-NEXT:    call __tls_get_addr@plt
 ; RV64-PIC-NEXT:    ld ra, 8(sp)
 ; RV64-PIC-NEXT:    addi sp, sp, 16
 ; RV64-PIC-NEXT:    ret
 ;
 ; RV32-NOPIC-LABEL: f2:
 ; RV32-NOPIC:       # %bb.0: # %entry
-; RV32-NOPIC-NEXT:  .LBB1_1: # %entry
-; RV32-NOPIC-NEXT:    # Label of block must be emitted
+; RV32-NOPIC-NEXT:  .Ltmp1:
 ; RV32-NOPIC-NEXT:    auipc a0, %tls_ie_pcrel_hi(ld)
-; RV32-NOPIC-NEXT:    lw a0, %pcrel_lo(.LBB1_1)(a0)
+; RV32-NOPIC-NEXT:    lw a0, %pcrel_lo(.Ltmp1)(a0)
 ; RV32-NOPIC-NEXT:    add a0, a0, tp
 ; RV32-NOPIC-NEXT:    ret
 ;
 ; RV64-NOPIC-LABEL: f2:
 ; RV64-NOPIC:       # %bb.0: # %entry
-; RV64-NOPIC-NEXT:  .LBB1_1: # %entry
-; RV64-NOPIC-NEXT:    # Label of block must be emitted
+; RV64-NOPIC-NEXT:  .Ltmp1:
 ; RV64-NOPIC-NEXT:    auipc a0, %tls_ie_pcrel_hi(ld)
-; RV64-NOPIC-NEXT:    ld a0, %pcrel_lo(.LBB1_1)(a0)
+; RV64-NOPIC-NEXT:    ld a0, %pcrel_lo(.Ltmp1)(a0)
 ; RV64-NOPIC-NEXT:    add a0, a0, tp
 ; RV64-NOPIC-NEXT:    ret
 entry:
   ret i32* @ld
 }
 
 
 ; initialexec specified
 
 define i32* @f3() nounwind {
 ; RV32-PIC-LABEL: f3:
 ; RV32-PIC:       # %bb.0: # %entry
-; RV32-PIC-NEXT:  .LBB2_1: # %entry
-; RV32-PIC-NEXT:    # Label of block must be emitted
+; RV32-PIC-NEXT:  .Ltmp2:
 ; RV32-PIC-NEXT:    auipc a0, %tls_ie_pcrel_hi(ie)
-; RV32-PIC-NEXT:    lw a0, %pcrel_lo(.LBB2_1)(a0)
+; RV32-PIC-NEXT:    lw a0, %pcrel_lo(.Ltmp2)(a0)
 ; RV32-PIC-NEXT:    add a0, a0, tp
 ; RV32-PIC-NEXT:    ret
 ;
 ; RV64-PIC-LABEL: f3:
 ; RV64-PIC:       # %bb.0: # %entry
-; RV64-PIC-NEXT:  .LBB2_1: # %entry
-; RV64-PIC-NEXT:    # Label of block must be emitted
+; RV64-PIC-NEXT:  .Ltmp2:
 ; RV64-PIC-NEXT:    auipc a0, %tls_ie_pcrel_hi(ie)
-; RV64-PIC-NEXT:    ld a0, %pcrel_lo(.LBB2_1)(a0)
+; RV64-PIC-NEXT:    ld a0, %pcrel_lo(.Ltmp2)(a0)
 ; RV64-PIC-NEXT:    add a0, a0, tp
 ; RV64-PIC-NEXT:    ret
 ;
 ; RV32-NOPIC-LABEL: f3:
 ; RV32-NOPIC:       # %bb.0: # %entry
-; RV32-NOPIC-NEXT:  .LBB2_1: # %entry
-; RV32-NOPIC-NEXT:    # Label of block must be emitted
+; RV32-NOPIC-NEXT:  .Ltmp2:
 ; RV32-NOPIC-NEXT:    auipc a0, %tls_ie_pcrel_hi(ie)
-; RV32-NOPIC-NEXT:    lw a0, %pcrel_lo(.LBB2_1)(a0)
+; RV32-NOPIC-NEXT:    lw a0, %pcrel_lo(.Ltmp2)(a0)
 ; RV32-NOPIC-NEXT:    add a0, a0, tp
 ; RV32-NOPIC-NEXT:    ret
 ;
 ; RV64-NOPIC-LABEL: f3:
 ; RV64-NOPIC:       # %bb.0: # %entry
-; RV64-NOPIC-NEXT:  .LBB2_1: # %entry
-; RV64-NOPIC-NEXT:    # Label of block must be emitted
+; RV64-NOPIC-NEXT:  .Ltmp2:
 ; RV64-NOPIC-NEXT:    auipc a0, %tls_ie_pcrel_hi(ie)
-; RV64-NOPIC-NEXT:    ld a0, %pcrel_lo(.LBB2_1)(a0)
+; RV64-NOPIC-NEXT:    ld a0, %pcrel_lo(.Ltmp2)(a0)
 ; RV64-NOPIC-NEXT:    add a0, a0, tp
 ; RV64-NOPIC-NEXT:    ret
 entry:
   ret i32* @ie
 }
 
 
 ; localexec specified